RFC 1950

okay bestie, this one is RFC 1950 🎀 and it's called ZLIB Compressed Data Format Specification version 3.3 ✨. It came out of the Network Working Group in May 1996 and it's classed as Informational 🌸. The authors are P. Deutsch from Aladdin Enterprises and J-L. Gailly from Info-ZIP 💖. It doesn't list anything it obsoletes or updates, so no drama there 🦋.

Status of This Memo 🌸

this memo gives information to the Internet community 💖. it does not specify an Internet standard of any kind 🙅 and distribution of it is unlimited 🌈.

IESG Note 🙈

the IESG takes no position on whether any Intellectual Property Rights statements in this document are valid 🫶.

Notices 📜

Copyright (c) 1996 L. Peter Deutsch and Jean-Loup Gailly 💅

you're allowed to copy and distribute this document for any purpose and without charge, including translating it into other languages and putting it into compilations 🎁. the only conditions are that the copyright notice and this notice stay intact, and that any substantive changes or deletions from the original are clearly marked ✍️.

a pointer to the latest version of this doc and related stuff in HTML format lives at the URL <ftp://ftp.uu.net/graphics/png/documents/zlib/zdoc-index.html> 🔗.

Abstract ✨

okay so this spec defines a lossless compressed data format 🔥. the data can be produced or consumed even for an arbitrarily long, sequentially presented input stream, and you only need an a priori bounded amount of intermediate storage to do it 🧠. right now it uses the DEFLATE compression method, but it can easily be extended to use other compression methods 🧩. it can also be implemented readily in a way that isn't covered by patents 🕊️.

this spec also defines the ADLER-32 checksum, which is an extension and improvement of the Fletcher checksum 🦋. it's used for detecting data corruption, and the spec gives an algorithm for computing it 💜.

Table of Contents 📑

   1. Introduction ................................................... 2
      1.1. Purpose ................................................... 2
      1.2. Intended audience ......................................... 3
      1.3. Scope ..................................................... 3
      1.4. Compliance ................................................ 3
      1.5.  Definitions of terms and conventions used ................ 3
      1.6. Changes from previous versions ............................ 3
   2. Detailed specification ......................................... 3
      2.1. Overall conventions ....................................... 3
      2.2. Data format ............................................... 4
      2.3. Compliance ................................................ 7
   3. References ..................................................... 7
   4. Source code .................................................... 8
   5. Security Considerations ........................................ 8
   6. Acknowledgements ............................................... 8
   7. Authors' Addresses ............................................. 8
   8. Appendix: Rationale ............................................ 9
   9. Appendix: Sample code ..........................................10

1. Introduction ✨

1.1. Purpose 🎯

the whole point of this spec is to define a lossless compressed data format that:

• is independent of CPU type, operating system, file system, and character set, so it can be used for interchange 🔁;
• can be produced or consumed even for an arbitrarily long sequentially presented input data stream, using only an a priori bounded amount of intermediate storage, so it can be used in data communications or similar structures such as Unix filters 📡;
• can use a number of different compression methods 🧰;
• can be implemented readily in a way not covered by patents, so it can be practiced freely 🆓.

also, the data format defined here does not try to allow random access to compressed data 🙅‍♀️.

1.2. Intended audience 🧑‍💻

this spec is for implementors of software that compresses data into zlib format and/or decompresses data from zlib format 💻. the text assumes you've got a basic background in programming at the level of bits and other primitive data representations 🧠.

1.3. Scope 🔭

the spec describes a compressed data format that can be used for in-memory compression of a sequence of arbitrary bytes 💾.

1.4. Compliance 💅

unless otherwise indicated below, a compliant decompressor must be able to accept and decompress any data set that conforms to all the specifications presented here 📥. a compliant compressor must produce data sets that conform to all the specifications presented here 📤. no skipping steps, bestie 😤.

1.5. Definitions of terms and conventions used 📖

byte: 8 bits stored or transmitted as a unit (same as an octet) 🧩. for this spec, a byte is exactly 8 bits, even on machines that store a character on a number of bits different from 8 🤓. the numbering of bits within a byte is explained below 👀.

1.6. Changes from previous versions 🕰️

the glow-up timeline 💖: version 3.1 was the first public release of this spec 🌱. in version 3.2, some terminology got changed and the Adler-32 sample code was rewritten for clarity ✨. then in version 3.3, support for a preset dictionary was introduced, and the spec got converted to RFC style 🎀 iconic.

2. Detailed specification ✨

2.1. Overall conventions 🎀

Okay bestie, the diagrams in this section look like a lot, but they're actually simple. Here's the deal 💅

A box like this is one byte:

   +---+
   |   | <-- the vertical bars might be missing
   +---+

And a box like this is a variable number of bytes:

   +==============+
   |              |
   +==============+

Bytes stored inside a computer don't have a "bit order" 🦋, since they're always treated as a unit. BUT, a byte treated as an integer between 0 and 255 does have a most- and least-significant bit. We write numbers with the most-significant digit on the left, so we write bytes with the most-significant bit on the left too. In the diagrams below, bit 0 is the least-significant bit, so the bits are numbered like this:

   +--------+
   |76543210|
   +--------+

A number can take up multiple bytes inside a computer. All multi-byte numbers in this format are stored with the MOST-significant byte first (at the lower memory address). For example, the decimal number 520 is stored like this 🌸:

             0     1
         +--------+--------+
         |00000010|00001000|
         +--------+--------+
          ^        ^
          |        |
          |        + less significant byte = 8
          + more significant byte = 2 x 256

2.2. Data format 💖

A zlib stream has the following structure ✨:

           0   1
         +---+---+
         |CMF|FLG|   (more-->)
         +---+---+

      (if FLG.FDICT set)

           0   1   2   3
         +---+---+---+---+
         |     DICTID    |   (more-->)
         +---+---+---+---+

         +=====================+---+---+---+---+
         |...compressed data...|    ADLER32    |
         +=====================+---+---+---+---+

Anything that shows up after ADLER32 is NOT part of the zlib stream 🙅‍♀️. Nothing after it counts.

CMF (Compression Method and flags): this byte is split into a 4-bit compression method and a 4-bit information field, and what the info field means depends on the compression method 🧚

bits 0 to 3  CM     Compression method
bits 4 to 7  CINFO  Compression info

CM (Compression method): this tells you which compression method was used in the file. CM = 8 means the "deflate" compression method with a window size up to 32K. This is the method gzip and PNG use (check references [1] and [2] in Chapter 3, below, for the reference documents). CM = 15 is reserved 🔒. It might get used in a future version of this spec to say there's an extra field before the compressed data.

CINFO (Compression info): for CM = 8, CINFO is the base-2 logarithm of the LZ77 window size, minus eight. CINFO=7 means a 32K window. CINFO values above 7 are not allowed in this version of the spec. CINFO isn't defined in this spec for any CM that isn't 8 🙈.

FLG (FLaGs): this flag byte breaks down like this 🎀:

bits 0 to 4  FCHECK  (check bits for CMF and FLG)
bit  5       FDICT   (preset dictionary)
bits 6 to 7  FLEVEL  (compression level)

The FCHECK value has to be set so that CMF and FLG, read together as a 16-bit unsigned integer stored in MSB order (CMF*256 + FLG), is a multiple of 31 🔢.

FDICT (Preset dictionary): if FDICT is set, a DICT dictionary identifier shows up right after the FLG byte. The dictionary is a sequence of bytes that gets fed to the compressor first, without making any compressed output. DICT is the Adler-32 checksum of that byte sequence (see the ADLER32 definition below). The decompressor can use this identifier to figure out which dictionary the compressor used 🕵️‍♀️.

FLEVEL (Compression level): these flags are there for specific compression methods to use. The "deflate" method (CM = 8) sets them like this:

0 - compressor used fastest algorithm
1 - compressor used fast algorithm
2 - compressor used default algorithm
3 - compressor used maximum compression, slowest algorithm

FLEVEL isn't needed for decompression at all. It's just there to hint whether recompressing might be worth it 💅.

compressed data: for compression method 8, the compressed data is stored in the deflate compressed data format, as described in the document "DEFLATE Compressed Data Format Specification" by L. Peter Deutsch (see reference [3] in Chapter 3, below). Other compressed data formats aren't specified in this version of the zlib spec 🚫.

ADLER32 (Adler-32 checksum): this holds a checksum value of the uncompressed data (excluding any dictionary data), computed with the Adler-32 algorithm. Adler-32 is a 32-bit extension and improvement of the Fletcher algorithm, which is used in the ITU-T X.224 / ISO 8073 standard. (See references [4] and [5] in Chapter 3, below.) 🦋

Adler-32 is made of two sums that get accumulated per byte. s1 is the sum of all the bytes, and s2 is the sum of all the s1 values. Both sums are done modulo 65521. s1 starts at 1, and s2 starts at zero. The Adler-32 checksum gets stored as s2*65536 + s1, most-significant-byte first (network) order 🌈.

2.3. Compliance 🫶

A compliant compressor must produce streams with correct CMF, FLG and ADLER32 values, but it doesn't need to support preset dictionaries. When the zlib data format is used as part of another standard data format, the compressor may use only the preset dictionaries that this other data format specifies. If that other format doesn't use the preset dictionary feature, the compressor must not set the FDICT flag ⚠️.

A compliant decompressor must check CMF, FLG and ADLER32, and it must give an error indication if any of them have incorrect values 🚨. A compliant decompressor must also give an error indication if CM isn't one of the values this spec defines (only the value 8 is allowed in this version). The reason is that another value could mean new features are present, and then the following data would be read wrong 😱. A compliant decompressor must give an error indication if FDICT is set and DICTID isn't the identifier of a known preset dictionary. A decompressor may ignore FLEVEL and still be compliant. When the zlib data format is used as part of another standard format, a compliant decompressor must support all the preset dictionaries that the other format specifies. When the other format doesn't use the preset dictionary feature, a compliant decompressor must reject any stream where the FDICT flag is set 🙅‍♀️.

3. References 📚

[1] Deutsch, L.P.,"GZIP Compressed Data Format Specification", available in ftp://ftp.uu.net/pub/archiving/zip/doc/

[2] Thomas Boutell, "PNG (Portable Network Graphics) specification", available in ftp://ftp.uu.net/graphics/png/documents/

[3] Deutsch, L.P.,"DEFLATE Compressed Data Format Specification", available in ftp://ftp.uu.net/pub/archiving/zip/doc/

[4] Fletcher, J. G., "An Arithmetic Checksum for Serial Transmissions," IEEE Transactions on Communications, Vol. COM-30, No. 1, January 1982, pp. 247-252.

[5] ITU-T Recommendation X.224, Annex D, "Checksum Algorithms," November, 1993, pp. 144, 145. (Available from gopher://info.itu.ch). ITU-T X.244 is also the same as ISO 8073.

4. Source code 💻

Source code for a C language implementation of a "zlib" compliant library is available at ftp://ftp.uu.net/pub/archiving/zip/zlib/ 🎀.

5. Security Considerations 💀

A decoder that fails to check the ADLER32 checksum value may be subject to undetected data corruption. This is serious, bestie 😭🚨. Skipping that check means corrupted data can slip right through without anyone noticing 🙈.

6. Acknowledgements 🫶

Trademarks cited in this document are the property of their respective owners 💜.

Jean-Loup Gailly and Mark Adler designed the zlib format and wrote the related software described in this specification 👑. Glenn Randers-Pehrson converted this document to RFC and HTML format 🌸.

7. Authors' Addresses ✉️

L. Peter Deutsch
Aladdin Enterprises
203 Santa Margarita Ave.
Menlo Park, CA 94025

Phone: (415) 322-0103 (AM only)
FAX:   (415) 322-1734
EMail: <ghost@aladdin.com>


Jean-Loup Gailly

EMail: <gzip@prep.ai.mit.edu>

Questions about the technical content of this specification can be sent by email to Jean-Loup Gailly <gzip@prep.ai.mit.edu> and Mark Adler <madler@alumni.caltech.edu> 🧠.

Editorial comments on this specification can be sent by email to L. Peter Deutsch <ghost@aladdin.com> and Glenn Randers-Pehrson <randeg@alumni.rpi.edu> 📝.

8. Appendix: Rationale 🧚

8.1. Preset dictionaries 📖

A preset dictionary is especially useful for compressing short input sequences 🍓. The compressor can use the dictionary context to encode the input in a more compact way. The decompressor can be set up with the right context by virtually decompressing a compressed version of the dictionary, without producing any output. For certain compression algorithms, such as deflate, this can be done without actually performing any decompression 🤯.

The compressor and the decompressor must use exactly the same dictionary. The dictionary may be fixed, or it may be picked from a certain number of predefined dictionaries, depending on the kind of input data. The decompressor can figure out which dictionary the compressor picked by checking the dictionary identifier. This document doesn't specify the contents of predefined dictionaries, since the best dictionaries are application specific. Standard data formats that use this feature of the zlib spec must precisely define the allowed dictionaries 💜.

8.2. The Adler-32 algorithm 🦋

The Adler-32 algorithm is much faster than the CRC32 algorithm, and it still gives an extremely low probability of undetected errors 🔥.

The modulo on unsigned long accumulators can be delayed for 5552 bytes, so the modulo operation time is negligible 🌈. If the bytes are a, b, c, the second sum is 3a + 2b + c + 3. That makes it sensitive to position and order, unlike the first sum, which is just a checksum 👀. That 65521 is prime matters, because it avoids a possible large class of two-byte errors that would leave the check unchanged 🧐. (The Fletcher checksum uses 255, which isn't prime, and which also makes the Fletcher check insensitive to single byte changes 0 <-> 255 🙃.)

The sum s1 starts at 1 instead of zero, so the length of the sequence becomes part of s2. That way the length doesn't have to be checked separately 💅. (Any sequence of zeroes has a Fletcher checksum of zero 💀.)

9. Appendix: Sample code ✨

okay so this is the sample code 🎀 It computes the Adler-32 checksum of a data buffer. It's written for clarity, not speed, so it's not trying to be fast, it's trying to be readable 🧚 It's ANSI C. If you're not a C girlie, here are some hints 💅

• & is the bitwise AND operator.
• >> is the bitwise right shift operator. When it's applied to an unsigned quantity, like here, right shift inserts zero bit(s) at the left.
• << is the bitwise left shift operator. Left shift inserts zero bit(s) at the right.
• ++ means n++ increments the variable n.
• % is the modulo operator. a % b is the remainder of a divided by b.

---

#define BASE 65521 /* largest prime smaller than 65536 */

/*
   Update a running Adler-32 checksum with the bytes buf[0..len-1]
 and return the updated checksum. The Adler-32 checksum should be
 initialized to 1.

 Usage example:

   unsigned long adler = 1L;

   while (read_buffer(buffer, length) != EOF) {
     adler = update_adler32(adler, buffer, length);
   }
   if (adler != original_adler) error();
*/
unsigned long update_adler32(unsigned long adler,
   unsigned char *buf, int len)
{
  unsigned long s1 = adler & 0xffff;
  unsigned long s2 = (adler >> 16) & 0xffff;
  int n;

  for (n = 0; n < len; n++) {
    s1 = (s1 + buf[n]) % BASE;
    s2 = (s2 + s1)     % BASE;
  }
  return (s2 << 16) + s1;
}

/* Return the adler32 of the bytes buf[0..len-1] */
unsigned long adler32(unsigned char *buf, int len)
{
  return update_adler32(1L, buf, len);
}