RFC 9841

Shared Brotli Compressed Data Format

📅 published September 2025, from ietf

RFC 9841: Shared Brotli Compressed Data Format ✨💖

  • Authors: J. Alakuijala, T. Duong, E. Kliuchnikov, Z. Szabadka, L. Vandevenne (Ed.), all at Google, Inc. 🫶
  • Category: Informational 🌸
  • Date: September 2025 📅
  • ISSN: 2070-1721
  • Updates: RFC 7932 (brotli) 🔄
  • Series: Internet Engineering Task Force (IETF) 🦋

Abstract

okay so this spec defines a data format for shared brotli compression 🎀 It adds shared dictionaries, large window support, and a container format on top of brotli (RFC 7932). Shared dictionaries and large window support can give you way bigger compression gains than regular brotli 🔥 This document is an extension to the method defined in RFC 7932, so it's giving "brotli, but make it level up" 💅

Status of This Memo

not gonna lie, this one is not an Internet Standards Track spec 🥺 It's published for informational purposes 📚

It's a product of the IETF 💜 It represents the consensus of the IETF community and got public review, then the Internet Engineering Steering Group (IESG) approved it for publication ✅ Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you're curious 🤓

For the latest status, errata, and how to send feedback, look at https://www.rfc-editor.org/info/rfc9841 🌈

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. ©️✨

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read these, bestie, because they describe your rights and restrictions for this document 🙈 Code Components pulled out of this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Revised BSD License ⚠️

Table of Contents

1.  Introduction
  1.1.  Purpose
  1.2.  Intended Audience
  1.3.  Scope
  1.4.  Compliance
  1.5.  Definitions of Terms and Conventions Used
    1.5.1.  Packing into Bytes
2.  Shared Brotli Overview
3.  Shared Dictionaries
  3.1.  Custom Static Dictionaries
    3.1.1.  Transform Operations
  3.2.  LZ77 Dictionaries
4.  Varint Encoding
5.  Shared Dictionary Stream
6.  Large Window Brotli Compressed Data Stream
7.  Shared Brotli Compressed Data Stream
8.  Shared Brotli Framing Format Stream
  8.1.  Main Format
  8.2.  Chunk Format
  8.3.  Metadata Format
  8.4.  Chunk Specifications
    8.4.1.  Padding Chunk (Type 0)
    8.4.2.  Metadata Chunk (Type 1)
    8.4.3.  Data Chunk (Type 2)
    8.4.4.  First Partial Data Chunk (Type 3)
    8.4.5.  Middle Partial Data Chunk (Type 4)
    8.4.6.  Last Partial Data Chunk (Type 5)
    8.4.7.  Footer Metadata Chunk (Type 6)
    8.4.8.  Global Metadata Chunk (Type 7)
    8.4.9.  Repeat Metadata Chunk (Type 8)
    8.4.10. Central Directory Chunk (Type 9)
    8.4.11. Final Footer Chunk (Type 10)
    8.4.12. Chunk Ordering
9.  Security Considerations
10. IANA Considerations
11. References
  11.1.  Normative References
  11.2.  Informative References
Acknowledgments
Authors' Addresses

1. Introduction ✨

1.1. Purpose

okay the whole point of this spec is to extend the brotli compressed data format [RFC7932] with new abilities that get you even more compression gains 🚀

  • Shared dictionaries give the encoder and decoder a static shared context, for seriously big compression gains 💅
  • Large window brotli lets back reference distances go way further, which helps with files over 16 MiB 🐘
  • The framing format is a container format that stores multiple resources and can reference dictionaries 🎁

This document is the authoritative spec for shared brotli data formats, plus the backwards compatible changes to brotli. It also defines the following:

  • The data format of serialized shared dictionaries 📖
  • The data format of the framing format 🖼️
  • The encoding of window bits and distances for large window brotli in the brotli data format 🪟
  • The encoding of shared dictionary references in the brotli data format 🔗

1.2. Intended Audience

this spec is for software implementers who want to compress data into, and/or decompress data from, the shared brotli dictionary format 👩‍💻

The text assumes you have a basic background in programming at the level of bits and other primitive data representations 🧠 Knowing the LZ77 coding technique [LZ77] helps, but it's not required 🙌

1.3. Scope

this spec defines a data format for shared brotli compression. It adds support for dictionaries and extended features to brotli [RFC7932] 🎀

1.4. Compliance

unless stated otherwise below, a compliant decompressor must be able to accept and decompress any data set that conforms to all the specifications presented here 🫧 Also, a compliant compressor must produce data sets that conform to all the specifications presented here 💯

1.5. Definitions of Terms and Conventions Used

Byte: 8 bits stored or transmitted as a unit (same as an octet) 🍓 For this spec, a byte is exactly 8 bits, even on machines that store a character on a different number of bits, so no sneaky exceptions 🙅‍♀️ Scroll down for how bits are numbered inside a byte 👇

String: a sequence of arbitrary bytes 🧵

Bytes stored inside a computer don't have a "bit order", since they're always treated as one unit 🤷‍♀️ But a byte viewed as an integer between 0 and 255 does have a most significant bit (MSB) and a least significant bit (LSB) 🌟 We write numbers with the most significant digit on the left, so we write bytes with the MSB on the left too 👈 In the diagrams below, bit 0 is the LSB, so the bits are numbered like this:

   +--------+
   |76543210|
   +--------+

A number can take up multiple bytes inside a computer 💫 All multi-byte numbers in this format are unsigned and stored with the least significant byte first (at the lower memory address) 🧩 For example, the decimal 16-bit number 520 is stored like this:

   0        1
   +--------+--------+
   |00001000|00000010|
   +--------+--------+
   ^        ^
   |        |
   |        + more significant byte = 2 x 256
   + less significant byte = 8

1.5.1. Packing into Bytes

this document doesn't cover the order in which the bits of a byte get sent over a bit-sequential medium, since the final data format is byte-oriented, not bit-oriented 🛰️ But the compressed block format is described as a sequence of data elements with different bit lengths, not as a sequence of bytes 😵‍💫 So we have to say how to pack those elements into bytes to make the final compressed byte sequence 🎁

  • Data elements get packed into bytes in order of increasing bit number within the byte, so starting with the LSB of the byte 🔢
  • Data elements other than prefix codes are packed starting with the LSB of the data element. These are called integer values and they're treated as unsigned 💜
  • Prefix codes are packed starting with the MSB of the code 🪄

Put another way: if you printed the compressed data as a sequence of bytes, starting with the first byte at the right margin and moving to the left, with each byte's MSB on the left as usual, you could parse it from right to left. Fixed-width elements would be in the correct MSB-to-LSB order, and prefix codes would be in bit-reversed order (the first bit of the code sits in the relative LSB position) 🔄

Here's an example. Pack these data elements into a sequence of 3 bytes: 3-bit integer value 6, 4-bit integer value 2, 3-bit prefix code b'110, 2-bit prefix code b'10, and 12-bit integer value 3628 🍰

     byte 2   byte 1   byte 0
   +--------+--------+--------+
   |11100010|11000101|10010110|
   +--------+--------+--------+
    ^            ^ ^   ^   ^
    |            | |   |   |
    |            | |   |   +------ integer value 6
    |            | |   +---------- integer value 2
    |            | +-------------- prefix code 110
    |            +---------------- prefix code 10
    +----------------------------- integer value 3628

2. Shared Brotli Overview 🌷

shared brotli extends brotli [RFC7932] with support for shared dictionaries, a larger LZ77 window, and a framing format 🪟✨

3. Shared Dictionaries 📚

a shared dictionary is a piece of data that's shared by a compressor and a decompressor 🤝 The compressor can use the dictionary context to encode the input more compactly 📦 The compressor and the decompressor must use exactly the same dictionary. Shared dictionaries are especially useful for compressing short input sequences 🎯

Wait, I need to correct myself: in the spec text, "must" is lowercase, so the force stays the same as written. The line above is the spec's own wording, so "must" stays as the original's lowercase "must". 💅

A shared brotli dictionary can use two methods of sharing context:

  • LZ77 dictionary: the encoder and decoder can refer to a given sequence of bytes. Multiple LZ77 dictionaries can be set 🔁
  • Custom static dictionary: a word list with transforms 🧩 The encoder and decoder replace the static dictionary data with the data in the shared dictionary. The original static dictionary is described in Section 8 in [RFC7932]. The original data from Appendices A and B of [RFC7932] gets replaced. It's also possible to switch this dictionary dynamically based on the data compression context, and/or to include a reference to the original dictionary inside the custom dictionary 🌈

If no shared dictionary is set, the decoder behaves the same as in [RFC7932] on a brotli stream 🫶

If a shared dictionary is set, it can set LZ77 dictionaries, override static dictionary words, and/or override transforms 👑

Part 1 of 5 is done, bestie, and more parts are coming 🦋💖

3.1. Custom Static Dictionaries ✨

okay so, if a custom word list is set, the decoder from [RFC7932] gets a glow-up and some of its defaults are overridden 💅:

  • Instead of the Static Dictionary Data from Appendix A of [RFC7932], one or more word lists from the custom static dictionary data are used.
  • Instead of NDBITS at the end of Appendix A of [RFC7932], a custom SIZE_BITS_BY_LENGTH per custom word list is used.
  • The copy length for a static dictionary reference must be between 4 and 31 and may not be a value for which SIZE_BITS_BY_LENGTH of this dictionary is 0.

If a custom transforms list is set without context dependency, the following also gets overridden 🦋:

  • The "List of Word Transformations" from Appendix B of [RFC7932] is overridden by one or more lists of custom prefixes, suffixes, and transform operations.
  • The transform_id must be smaller than the number of transforms given in the custom transforms list.

If the dictionary is context dependent, it includes a lookup table of 64 word list and transform list combinations. When resolving a static dictionary word, the decoder computes the literal Context ID as described in Section 7.1 of [RFC7932]. That literal Context ID is used as the index in the lookup tables to select the word list and transforms to use. If the dictionary is not context dependent, this ID is implicitly 0 instead. 🧚

If a distance goes beyond the dictionary for the current ID and multiple word/transform list combinations are defined, the next dictionary gets picked in this order 👑:

  • If context dependent:
    • use the index matching the current context first, and then
    • use the same order as defined in the shared dictionary (excluding the current context) next.
  • If not context dependent:
    • use the same order as defined in the shared dictionary.

3.1.1. Transform Operations

A shared dictionary may include custom word transformations to replace those specified in Section 8 and Appendix B of [RFC7932]. A transform consists of a possible prefix, a transform operation, a parameter (for some operations), and a possible suffix. In the shared dictionary format, the transform operation is represented by a numerical ID, which is listed in the table below. 🌸

                    +====+===========================+
                    | ID | Operation                 |
                    +====+===========================+
                    | 0  | Identity                  |
                    +----+---------------------------+
                    | 1  | OmitLast1                 |
                    +----+---------------------------+
                    | 2  | OmitLast2                 |
                    +----+---------------------------+
                    | 3  | OmitLast3                 |
                    +----+---------------------------+
                    | 4  | OmitLast4                 |
                    +----+---------------------------+
                    | 5  | OmitLast5                 |
                    +----+---------------------------+
                    | 6  | OmitLast6                 |
                    +----+---------------------------+
                    | 7  | OmitLast7                 |
                    +----+---------------------------+
                    | 8  | OmitLast8                 |
                    +----+---------------------------+
                    | 9  | OmitLast9                 |
                    +----+---------------------------+
                    | 10 | FermentFirst              |
                    +----+---------------------------+
                    | 11 | FermentAll                |
                    +----+---------------------------+
                    | 12 | OmitFirst1                |
                    +----+---------------------------+
                    | 13 | OmitFirst2                |
                    +----+---------------------------+
                    | 14 | OmitFirst3                |
                    +----+---------------------------+
                    | 15 | OmitFirst4                |
                    +----+---------------------------+
                    | 16 | OmitFirst5                |
                    +----+---------------------------+
                    | 17 | OmitFirst6                |
                    +----+---------------------------+
                    | 18 | OmitFirst7                |
                    +----+---------------------------+
                    | 19 | OmitFirst8                |
                    +----+---------------------------+
                    | 20 | OmitFirst9                |
                    +----+---------------------------+
                    | 21 | ShiftFirst (by PARAMETER) |
                    +----+---------------------------+
                    | 22 | ShiftAll (by PARAMETER)   |
                    +----+---------------------------+

                                 Table 1

Operations 0 to 20 are specified in Section 8 of [RFC7932]. ShiftFirst and ShiftAll are the ones that transform specifically encoded SCALARs, so let's talk about those 🔥.

A SCALAR is a 7-, 11-, 16-, or 21-bit unsigned integer, encoded with 1, 2, 3, or 4 bytes respectively. Here's what the bit layouts look like 🍓:

7-bit SCALAR:
+--------+
|0sssssss|
+--------+

11-bit SCALAR:
+--------+--------+
|110sssss|XXssssss|
+--------+--------+

16-bit SCALAR:
+--------+--------+--------+
|1110ssss|XXssssss|XXssssss|
+--------+--------+--------+

21-bit SCALAR:
+--------+--------+--------+--------+
|11110sss|XXssssss|XXssssss|XXssssss|
+--------+--------+--------+--------+

Given the input bytes that match the SCALAR encoding pattern, the SCALAR value is obtained by concatenating the "s" bits, with the MSBs coming from the earliest byte. The "X" bits could have any arbitrary value. 💜

An ADDEND is defined as the result of limited sign extension of a 16-bit unsigned PARAMETER. Here's the vibe check 🌈:

  • At first, the PARAMETER is zero-extended to 32 bits. After this, 0xFF0000 is added if the resulting value is greater or equal than 0x8000.

ShiftAll starts at the beginning of the word and repetitively applies the following transformation until the whole word is transformed ✨:

  • If the next untransformed byte matches the first byte of the 7-, 11-, 16-, or 21-bit SCALAR pattern, then:
    • If the untransformed part of the word is not long enough to match the whole SCALAR pattern, then the whole word is marked as transformed.
    • Otherwise, let SHIFTED be the sum of the ADDEND and the encoded SCALAR. The lowest bits from SHIFTED are written back into the corresponding "s" bits. The "0", "1", and "X" bits remain unchanged. Next, 1, 2, 3, or 4 untransformed bytes are marked as transformed according to the SCALAR pattern length.
  • Otherwise, the next untransformed byte is marked as transformed.

ShiftFirst applies the same transformation as ShiftAll, but it does not iterate. One and done 🎀.

3.2. LZ77 Dictionaries

If an LZ77 dictionary is set, the decoder treats it as a regular LZ77 copy, but behaves as if the bytes of this dictionary are accessible as the uncompressed bytes outside of the regular LZ77 window for backwards references. It's giving hidden extra storage 🙈.

Let LZ77_DICTIONARY_LENGTH be the length of the LZ77 dictionary. Then word_id, described in Section 8 of [RFC7932], is redefined as:

word_id = distance - (max allowed distance + 1 +
LZ77_DICTIONARY_LENGTH)

For the case when LZ77_DICTIONARY_LENGTH is 0, word_id matches the [RFC7932] definition. 🦋

Let dictionary_address be:

LZ77_DICTIONARY_LENGTH + max allowed distance - distance

Then distance values of <length, distance> pairs [RFC7932] in range (max allowed distance + 1)..(LZ77_DICTIONARY_LENGTH + max allowed distance) are interpreted as references starting in the LZ77 dictionary at the byte at dictionary_address. If length is longer than (LZ77_DICTIONARY_LENGTH - dictionary_address), then the reference continues to copy (length - LZ77_DICTIONARY_LENGTH + dictionary_address) bytes from the regular LZ77 window starting at the beginning. Main character energy, honestly 👑.

4. Varint Encoding

A varint is encoded in base 128, in one or more bytes, as follows 💖:

+--------+--------+             +--------+
|1xxxxxxx|1xxxxxxx| {0-8 times} |0xxxxxxx|
+--------+--------+             +--------+

Where the "x" bits of the first byte are the LSBs of the value, and the "x" bits of the last byte are the MSBs of the value. The last byte must have its MSB set to 0, and all other bytes must have their MSBs set to 1 to indicate there is a next byte. 🧚

The maximum allowed amount of bits to read is 63 bits. If the 9th byte is present and has its MSB set, then the stream must be considered as invalid. No cap, that's a hard limit 💀.

5. Shared Dictionary Stream ✨

Okay bestie, this section is all about the shared dictionary stream 📚🦋. It's how you encode a custom dictionary for brotli, with custom words, custom transforms, or both. A shared dictionary can show up on its own, or as the contents of a resource inside a framing format container 🎀.

A compliant shared brotli dictionary stream must have the following format, so let's go field by field 💅

2 bytes:  File signature in hexadecimal format (bytes 91 and 0).

varint:  LZ77_DICTIONARY_LENGTH.  The number of bytes for an LZ77
   dictionary, or 0 if there is none.  The maximum allowed value is
   the maximum possible sliding window size of brotli or large window
   brotli.

LZ77_DICTIONARY_LENGTH bytes:  Contents of the LZ77 dictionary.

1 byte:  NUM_CUSTOM_WORD_LISTS.  May have a value in range 0 to 64.

NUM_CUSTOM_WORD_LISTS times a word list with the following format
for each word list:

   28 bytes:  SIZE_BITS_BY_LENGTH.  An array of 28 unsigned 8-bit
      integers, indexed by word lengths 4 to 31.  The value
      represents log2(number of words of this length), with the
      exception of 0 meaning 0 words of this length.  The max allowed
      length value is 15 bits.  OFFSETS_BY_LENGTH is computed from
      this as OFFSETS_BY_LENGTH[i + 1] = OFFSETS_BY_LENGTH[i] +
      (SIZE_BITS_BY_LENGTH[i] ? (i << SIZE_BITS_BY_LENGTH[i]) : 0).

   N bytes:  Words dictionary data, where N is OFFSETS_BY_LENGTH[31]
      + (SIZE_BITS_BY_LENGTH[31] ?  (31 << SIZE_BITS_BY_LENGTH[31]) :
      0), with all the words of shortest length first, then all words
      of the next length, and so on, where there are either 0 or a
      positive power of two number of words for each length.

1 byte:  NUM_CUSTOM_TRANSFORM_LISTS.  May have a value in range 0 to
   64.

NUM_CUSTOM_TRANSFORM_LISTS times a transform list with the
following format for each transform list:

   2 bytes:  PREFIX_SUFFIX_LENGTH.  The length of prefix/suffix data.
      Must be at least 1 because the list must always end with a
      zero-length stringlet even if it is empty.

   NUM_PREFIX_SUFFIX times:  Prefix/suffix stringlet.
      NUM_PREFIX_SUFFIX is the number of stringlets parsed and must
      be in range 1..256.

      1 byte:  STRING_LENGTH.  The length of the entry contents.  0
         for the last (terminating) entry of the transform list.  For
         other entries, STRING_LENGTH must be in range 1..255.  The 0
         entry must be present and must be the last byte of the
         PREFIX_SUFFIX_LENGTH bytes of prefix/suffix data, else the
         stream must be rejected as invalid.

      STRING_LENGTH bytes:  Contents of the prefix/suffix.

   1 byte:  NTRANSFORMS.  Number of transformation triplets.

   NTRANSFORMS times the data for each transform listed below:

      1 byte:  Index of prefix in prefix/suffix data; must be less
         than NUM_PREFIX_SUFFIX.

      1 byte:  Index of suffix in prefix/suffix data; must be less
         than NUM_PREFIX_SUFFIX.

      1 byte:  Operation index; must be an index in the table of
         operations listed in Section 3.1.1.

   If and only if at least one transform has operation index
   ShiftFirst or ShiftAll, then NTRANSFORMS times the following:

      2 bytes:  Parameters for the transform.  If the transform does
         not have type ShiftFirst or ShiftAll, the value must be 0.
         ShiftFirst and ShiftAll interpret these bytes as an unsigned
         16-bit integer.

If NUM_CUSTOM_WORD_LISTS > 0 or NUM_CUSTOM_TRANSFORM_LISTS > 0
(else implicitly NUM_DICTIONARIES is 1 and points to the brotli
built-in and there is no context map):

   1 byte:  NUM_DICTIONARIES.  May have a value in range 1 to 64.
      Each dictionary is a combination of a word list and a transform
      list.  Each next dictionary is used when the distance goes
      beyond the previous.  If a CONTEXT_MAP is enabled, then the
      dictionary matching the context is moved to the front in the
      order for this context.

   NUM_DICTIONARIES times the DICTIONARY_MAP, which contains:

      1 byte:  Index into a custom word list or value
         NUM_CUSTOM_WORD_LISTS to indicate using the brotli [RFC7932]
         built-in default word list.

      1 byte:  Index into a custom transform list or value
         NUM_CUSTOM_TRANSFORM_LISTS to indicate using the brotli
         [RFC7932] built-in default transform list.

   1 byte:  CONTEXT_ENABLED.  If 0, there is no context map.  If 1, a
      context map used to select the dictionary is encoded as below.

   If CONTEXT_ENABLED is 1, there is a context map for the 64
   brotli [RFC7932] literals contexts:

      64 bytes:  CONTEXT_MAP.  Index into the DICTIONARY_MAP for the
         first dictionary to use for this context.

Now let's break that down, because there's a lot going on 🫶

The stream kicks off with a 2-byte file signature, bytes 91 and 0 in hex. Then comes LZ77_DICTIONARY_LENGTH, a varint saying how many bytes the LZ77 dictionary is. If there isn't one, it's just 0. The biggest it can be is the maximum possible sliding window size of brotli or large window brotli, so no overachieving here 😭 The actual LZ77 dictionary contents come right after.

Next up are the custom word lists. NUM_CUSTOM_WORD_LISTS is one byte and can be anywhere from 0 to 64. For each word list you get a 28-byte SIZE_BITS_BY_LENGTH array, indexed by word lengths 4 to 31. Each value is log2 of the number of words at that length, and a 0 means there are zero words of that length 🧚 The max allowed length value is 15 bits. From that array you compute OFFSETS_BY_LENGTH with the formula above, and then you read N bytes of word data, where N comes from OFFSETS_BY_LENGTH[31] plus the length-31 term. The words go shortest first, then the next length up, and so on. Each length has either 0 words or a positive power of two's worth.

Then the custom transform lists 💅 NUM_CUSTOM_TRANSFORM_LISTS is again 0 to 64. Each list starts with a 2-byte PREFIX_SUFFIX_LENGTH, which has to be at least 1, because the list always ends with a zero-length stringlet, even if it's empty. Then you parse NUM_PREFIX_SUFFIX stringlets, and that count has to be in range 1..256. Each stringlet has a 1-byte STRING_LENGTH. The terminating entry is the 0 one, and it must be present and must be the last byte of the PREFIX_SUFFIX_LENGTH bytes of prefix/suffix data. If it isn't, the stream must be rejected as invalid. Every other entry's STRING_LENGTH must be in range 1..255 ⚠️

Watch out: a missing or misplaced 0-length terminating entry means the stream must be rejected as invalid. No exceptions, no vibes-based decoding.

After the stringlets, each transform list has NTRANSFORMS triplets. Each one is a prefix index, a suffix index, and an operation index. The prefix and suffix indexes must be less than NUM_PREFIX_SUFFIX, and the operation index must point into the table of operations in Section 3.1.1. If any transform uses ShiftFirst or ShiftAll, then there's one more set: NTRANSFORMS 2-byte parameters. Those parameters have to be 0 for any transform that isn't ShiftFirst or ShiftAll, and for those two, they're read as an unsigned 16-bit integer.

Last part, the dictionary wiring 🎀 This section only applies if NUM_CUSTOM_WORD_LISTS is greater than 0, or NUM_CUSTOM_TRANSFORM_LISTS is greater than 0. Otherwise, the stream implicitly has NUM_DICTIONARIES set to 1, pointing at the brotli built-in, with no context map. When it does apply, NUM_DICTIONARIES can be 1 to 64. Each dictionary pairs a word list with a transform list, and each next dictionary kicks in once the distance goes beyond the previous one. If a context map is enabled, the dictionary matching the context gets moved to the front for that context. Each DICTIONARY_MAP entry is two bytes: a word list index (or NUM_CUSTOM_WORD_LISTS for the brotli [RFC7932] built-in default word list) and a transform list index (or NUM_CUSTOM_TRANSFORM_LISTS for the built-in default transform list). Then CONTEXT_ENABLED decides whether a context map follows. If it's 0, no map. If it's 1, you get 64 CONTEXT_MAP bytes, one for each of the 64 brotli [RFC7932] literal contexts, each pointing into the DICTIONARY_MAP for the first dictionary to use in that context. Iconic structure, honestly 👑

6. Large Window Brotli Compressed Data Stream

Large window brotli is the glow-up that lets the sliding window go beyond the 24-bit maximum of regular brotli [RFC7932] ✨

The compressed data stream stays backwards compatible with brotli [RFC7932], and it may optionally have the following differences.

In the encoding of WBITS in the stream header, the following new pattern of 14 bits is supported:

8 bits:  Value 00010001 to indicate a large window brotli stream.

6 bits:  WBITS.  Must have value in range 10 to 62.

So the 8 bits are 00010001, which flags a large window brotli stream, and then 6 bits of WBITS that must be in range 10 to 62 🌈

Distance alphabet: if the stream is a large window brotli stream, the maximum number of extra bits is 62. The theoretical maximum size of the distance alphabet becomes (16 + NDIRECT + (124 << NPOSTFIX)). That overrides the distance alphabet size given in Section 3.3 of [RFC7932], and it also changes the number of bits in the encoding of the Simple Prefix Code for distances, as described in Section 3.4 of [RFC7932]. There's one more limit, though. Even with that huge alphabet, the alphabet is not allowed to contain a distance symbol able to represent a distance larger than ((1 << 63) - 4) when its extra bits have their maximum value. Whether that can happen depends on NPOSTFIX and NDIRECT 🙈

A decoder that doesn't support 64-bit integers may reject a stream if WBITS is higher than 30, or if a distance symbol from the distance alphabet is able to encode a distance larger than 2147483644. Lowkey a very real "know your limits" moment 💀

7. Shared Brotli Compressed Data Stream

A shared brotli compressed data stream without a framing format stays backwards compatible with brotli [RFC7932]. It has these optional differences:

  • LZ77 dictionaries as described above are supported.
  • Custom static dictionaries replacing or extending the static dictionary of brotli [RFC7932] with different words or transforms are supported.
  • The stream may have the format of regular brotli [RFC7932] or the format of large window brotli as described in Section 6.

Basically, all the custom dictionary stuff from earlier works here too, and you can pick regular or large window brotli for the stream itself 🦋

8. Shared Brotli Framing Format Stream

A compliant shared brotli framing format stream has the format described below 🎀

8.1. Main Format

The start of the file looks like this:

4 bytes:  File signature in hexadecimal format (bytes 0x91, 0x0a,
   0x42, and 0x52).  The first byte contains the invalid WBITS
   combination for brotli [RFC7932] and large window brotli.

1 byte:  Container flags that are 8 bits and have the following
   meanings:

   bits 0 and 1:  Version indicator that must be b'00.  Otherwise,
      the decoder must reject the data stream as invalid.

   bit 2:  If 0, the file contains no final footer, may not contain
      any metadata chunks, may not contain a central directory, and
      may encode only a single resource (using one or more data
      chunks).  If 1, the file may contain one or more resources,
      metadata, and a central directory, and it must contain a final
      footer.

multiple times:  A chunk, each with the format specified in
   Section 8.2.

The file signature is 4 bytes in hex: 0x91, 0x0a, 0x42, and 0x52. That first byte is a sneaky one, because it's the invalid WBITS combination for both brotli [RFC7932] and large window brotli, so nothing can mistake this file for a regular stream 😭🧚

Then there's the 1-byte container flags field. The version indicator lives in bits 0 and 1, and it must be b'00. If it isn't, the decoder must reject the data stream as invalid, so no mystery versions allowed 🙅‍♀️

Bit 2 is the big decider. If it's 0, the file has no final footer, it may not contain any metadata chunks or a central directory, and it may only encode a single resource, using one or more data chunks. If bit 2 is 1, the file may hold one or more resources, plus metadata and a central directory, and in that case it must contain a final footer. Main character energy for the bigger files 👑

Last up, the file is made of chunks, which can show up multiple times, and each chunk follows the format specified in Section 8.2. We'll get to that next, bestie 💖

8.2. Chunk Format 🎀

okay bestie, every chunk starts with a length so the decoder knows how much to read. it's giving very organized energy ✨

  • varint: the length of this chunk. it does not count this varint itself, but it does count all the next header bytes and the data. if the value is 0, the CHUNK_TYPE byte isn't there at all and the chunk type is assumed to be 0 🧚
  • 1 byte: CHUNK_TYPE. here's the menu 💖
    • 0: padding chunk
    • 1: metadata chunk
    • 2: data chunk
    • 3: first partial data chunk
    • 4: middle partial data chunk
    • 5: last partial data chunk
    • 6: footer metadata chunk
    • 7: global metadata chunk
    • 8: repeat metadata chunk
    • 9: central directory chunk
    • 10: final footer

If CHUNK_TYPE is not padding chunk, central directory, or final footer, then there's one more byte:

  • 1 byte: CODEC 🌈
    • 0: uncompressed
    • 1: keep decoder
    • 2: brotli
    • 3: shared brotli

If CODEC is not "uncompressed", you get a varint for the uncompressed size in bytes of the data inside the compressed stream 🍓

If CODEC is "shared brotli", it gets extra spicy 🔥

  • 1 byte: the number of dictionary references. you can have multiple, but there are rules, no cap. there can be 1 serialized dictionary and 15 prefix dictionaries maximum (a serialized dictionary might already contain one of those). circular references are not allowed, meaning no dictionary reference can directly or indirectly use this chunk itself as its dictionary. big no 🙅‍♀️

Per dictionary reference:

  • 1 byte: Flags:
    • bits 0 and 1: dictionary source
      • 00: internal dictionary reference to a full resource by pointer, which can span one or more chunks. it must point to a full data chunk or a first partial data chunk.
      • 01: internal dictionary reference to single chunk contents by pointer. it can point to any chunk with content (data or metadata). if it's a partial data chunk, only this part is the dictionary. in this case, the dictionary type is not allowed to be a serialized dictionary.
      • 10: reference to a dictionary by hash code of a resource. the dictionary can come from an external source, like a different container. the user of the decoder must be able to provide the dictionary contents given its hash code (even if it comes from this container itself), or else it gets treated as an error when the user doesn't have it available.
      • 11: invalid bit combination
    • bits 2 and 3: dictionary type
      • 00: prefix dictionary, set in front of the sliding window
      • 01: serialized dictionary in the shared brotli format as specified in Section 5.
      • 10: invalid bit combination
      • 11: invalid bit combination
    • bits 4-7: must be 0

If hash-based:

  • 1 byte: type of hash used. the only supported value is 3, meaning 256-bit HighwayHash [HWYHASH].
  • 32 bytes: 256-bit HighwayHash checksum to refer to the dictionary.

If pointer based:

  • varint: pointer to its chunk in this container, encoded as a varint. the chunk must come earlier in the container than the current chunk ⏪
  • X bytes: extra header bytes, which depend on CHUNK_TYPE. if they're present, the subsequent sections spell them out 🧁
  • remaining bytes: the chunk contents. the uncompressed data depends on CHUNK_TYPE and gets specified in the next sections. the compressed data follows a format based on CODEC:
    • uncompressed: the raw bytes, no tricks 🫶
    • keep decoder: the continuation of the compressed stream that got interrupted at the end of the previous chunk. the decoder from the previous chunk must be used, and its state from the end of the previous chunk must be kept at the start of decoding this chunk. continuity is everything 💜
    • brotli: the bytes are in brotli format [RFC7932].
    • shared brotli: the bytes are in the shared brotli format specified in Section 7.

8.3. Metadata Format 🦋

every metadata chunk type uses this same format for its uncompressed content. it's giving consistency ✨

Per field:

  • 2 bytes: code to identify this metadata field. it must be two lowercase or two uppercase alpha ASCII characters. if the decoder runs into a lowercase field it doesn't recognize for the current chunk type, or non-ASCII characters, or non-alpha characters, the decoder must reject the data stream as invalid. uppercase codes can be used for custom user metadata and a compliant decoder can ignore them 🙈
  • varint: length of the content of this field in bytes, not counting the code bytes or this varint.
  • N bytes: the contents of this field.

the last field is reached when the chunk content ends. if the length of the last field doesn't end at the same byte as the end of the uncompressed content of the chunk, the decoder must reject the data stream as invalid. no sloppy endings allowed 💅

8.4. Chunk Specifications 🌸

8.4.1. Padding Chunk (Type 0)

all bytes in this chunk must be zero except for the initial varint that specifies the remaining chunk length. 🧸

the varint takes up bytes too, so if you're trying to add a certain number of padding bytes, you have to account for how the length of the varint depends on the value it encodes. math is slay 🧮

a single byte varint with a value of 0 is a padding chunk of length 1. for more padding, use higher varint values. do not use multiple shorter padding chunks, since that's slower to decode 🐌

8.4.2. Metadata Chunk (Type 1)

this chunk has metadata that applies to the resource whose beginning is encoded in the subsequent data chunk or first partial data chunk. 📎

the contents of this chunk follow the format described in Section 8.3.

these field types are recognized:

  • id (N bytes): name field. it may appear 0 or 1 times. the format is: name in UTF-8 encoding, with the length determined by the field length. it's treated generically but may be used as a filename. if it's used as a filename, forward slashes / should be used as directory separators, relative paths should be used, and filenames ending in a slash with 0-length content in the matching data chunk should be treated as an empty directory 📁
  • mt (8 bytes): modification type. it may appear 0 or 1 times. the format is: microseconds since epoch, as a little-endian, signed two's complement 64-bit integer. ⏰
  • custom user field: any two uppercase ASCII characters. your secret stash 🤫

8.4.3. Data Chunk (Type 2)

a data chunk contains the actual data of a resource. this is the main character 👑

this chunk has the following extra header bytes:

  • 1 byte: Flags:
    • bit 0: if true, this is not a resource that should be output implicitly as part of extracting resources from this container. instead, it may be referred to only explicitly, e.g., as a dictionary reference by hash code or offset. this flag should be set for data used as a dictionary, to improve compression of actual resources 🪄
    • bit 1: if true, a hash code is given.
    • bits 2-7: must be zero.

If a hash code is given:

  • 1 byte: type of hash used. the only supported value is 3, indicating 256-bit HighwayHash [HWYHASH].
  • 32 bytes: 256-bit HighwayHash checksum of the uncompressed data.

the uncompressed content bytes of this chunk are the actual data of the resource. ✨

8.4.4. First Partial Data Chunk (Type 3)

this chunk contains partial data of a resource. it's the first chunk in a series that holds the entire data of the resource. the start of the glow-up 🌅

the format is the same as a data chunk (Section 8.4.3), except for the differences noted below.

the second bit of flags must be set to 0, and no hash code is given.

the uncompressed data size is only for this part of the resource, not the full resource. 🧩

8.4.5. Middle Partial Data Chunk (Type 4)

this chunk contains partial data of a resource and is neither the first nor the last part of the full resource. the middle child, but make it iconic 🥹

the format is the same as a data chunk (Section 8.4.3), except for the differences noted below.

the first and second bits of flags must be set to 0.

the uncompressed data size is only for this part of the resource, not the full resource. 🧃

8.4.6. Last Partial Data Chunk (Type 5)

this chunk contains the final piece of partial data of a resource. the grand finale 🎆

the format is the same as a data chunk (Section 8.4.3), except for the differences noted below.

the first bit of flags must be set to 0.

if a hash code is given, the hash code of the full resource (concatenated from all previous chunks and this chunk) goes in this chunk. the full receipts 🧾

the uncompressed data size is only for this part of the resource, not the full resource.

this chunk type means no further chunks encode this resource, so the full resource is now known. the plot is complete, periodt 💀

this metadata applies to the resource whose encoding ended in the preceding data chunk or last partial data chunk. the afterword 🌙

the contents of this chunk follow the format described in Section 8.3.

there are no lowercase field types defined for footer metadata. uppercase field types can be used as custom user data. 💜

8.4.8. Global Metadata Chunk (Type 7)

this metadata applies to the whole container instead of a single resource. it's the whole vibe 🌍

the contents of this chunk follow the format described in Section 8.3.

there are no lowercase field types defined for global metadata. uppercase field types can be used as custom user data. 🫧

8.4.9. Repeat Metadata Chunk (Type 8) 🎀

Okay so repeat metadata chunks are the optional encore 💅. They copy metadata that sits between data chunks. To actually use them, you also need extra info, like pointers back to the original chunks, from the central directory 🗂️.

The contents follow the format in Section 8.3 ✨

This chunk has an extra header byte:

1 byte:  Chunk type of repeated chunk (metadata chunk or footer
   metadata chunk).

Now the rules for this whole set of chunks, bestie 👑:

  • Whether repeat metadata chunks show up at all is totally up to you 🙈
  • If they do show up, they must be present for all metadata chunks and footer metadata chunks.
  • There may be only 1 repeat metadata chunk per repeated metadata chunk.
  • They must appear in the same order as the chunks appear in the container, which is also the same order as listed in the central directory.
  • Compression is allowed here 🗜️. However, it is not allowed to use any internal dictionary except an earlier repeat metadata chunk of this series. And it is not allowed for a metadata chunk to keep the decoder state if the previous chunk is not a repeat metadata chunk. So the series of metadata chunks must be decompressible without using other chunks of the framing format file. No borrowing from the neighbors, periodt 🙅‍♀️

And the fields inside these metadata chunks have their own rules:

  • If a field is present, it must exactly match the corresponding field of the copied chunk.
  • It is allowed to leave out a field that is present in the copied chunk.
  • If a field is present, then it must be present in all other repeat metadata chunks when the copied chunk contains this field. In other words, if you know you can get the name field from a repeat chunk, you know that you will be able to get all names of all resources from all repeat chunks 🧚

8.4.10. Central Directory Chunk (Type 9) 🗺️

The central directory chunk, together with the repeat metadata chunks, is what lets you quickly find and list the compressed resources in the container file. It's giving map energy 🦋

The central directory chunk is always uncompressed, and it doesn't have the codec byte. Instead, it has this format:

varint:  Pointer into the file where the repeat metadata chunks are
   located or 0 if they are not present.

per chunk listed:

   varint:  Pointer into the file where this chunk begins.

   varint:  Number of header bytes N used below.

   N bytes:  Copy of all the header bytes of the pointed at chunk,
      including total size, chunk type byte, codec, uncompressed
      size, dictionary references, and X extra header bytes.  The
      content is not repeated here.

The last listed chunk is reached when the end of the contents of the central directory is reached. If the end doesn't match the last byte of the central directory, the decoder must reject the data stream as invalid 🚫

Also, if the central directory is present, it must list all data and metadata chunks of all types 📋

The final footer chunk closes out the file. It's only there if bit 2 of the initial container flags was set 🧁

This chunk has the following content, which is always uncompressed:

reversed varint:  Size of this entire framing format file, including
   these bytes themselves, or 0 if this size is not given.

reversed varint:  Pointer to the start of the central directory, or 0
   if there is none.

A reversed varint has the same format as a varint, except its bytes are in reversed order. It's built to be parsed from the end of the file back toward the beginning, which is a very main character move 🎬

8.4.12. Chunk Ordering 📜

The chunk ordering has to follow the rules below. If the decoder sees anything otherwise, it must reject the data stream as invalid. Not vibing 💀

  • Padding chunks may be inserted anywhere, even between chunks for which the rules below say no other chunk types may come in between 🫧
  • Metadata chunks must come immediately before the data chunks of the resource they apply to.
  • Footer metadata chunks must come immediately after the data chunks of the resource they apply to.
  • There may be only 0 or 1 metadata chunks per resource.
  • There may be only 0 or 1 footer metadata chunks per resource.
  • A resource must exist out of either 1 data chunk or 1 first partial data chunk, 0 or more middle partial data chunks, and 1 last partial data chunk, in that order.
  • Repeat metadata chunks must follow the rules of Section 8.4.9.
  • There may be only 0 or 1 central directory chunks.
  • If bit 2 of the container flags is set, there may be only a single resource, no metadata chunks of any type, no central directory, and no final footer.
  • If bit 2 of the container flags is not set, there must be exactly 1 final footer chunk, and it must be the last chunk in the file.

9. Security Considerations 🔐

Okay, this part is SERIOUS, so read it slowly 🥺

The security considerations for brotli [RFC7932] apply to shared brotli as well ✨

On top of that, the same considerations apply to the decoding of new file format streams for shared brotli. That includes shared dictionaries, the framing format, and the shared brotli format 🧩

The dictionary must be treated with the same security precautions as the content, because a change to the dictionary can result in a change to the decompressed content. Dictionary drama is real 📖

The CRIME attack [CRIME] shows that it's a bad idea to compress data from mixed sources (e.g., public and private). The data sources include not only the compressed data but also the dictionaries. For example, if you compress secret cookies using a public-data-only dictionary, you still leak information about the cookies 🍪💀

The dictionary and the compressed data can reveal things about each other. Data compressed with the dictionary can reveal contents of the dictionary when an adversary can control parts of the data to compress and see the compressed size. On the other hand, if the adversary can control the dictionary, the adversary can learn information about the compressed data 👀

The most robust defense against CRIME is to not compress private data, e.g., sensitive headers like cookies, or any content with personally identifiable information (PII). The challenge has been to identify secrets within a vast amount of data to be compressed. Cloudflare uses a regular expression [CLOUDFLARE]. Another idea is to extend existing web template systems (e.g., Soy [SOY]) so developers can mark secrets that must not be compressed 🛡️

A less robust idea, but easier to implement, is to randomize the compression algorithm. That means adding randomly generated padding, varying the compression ratio, etc. The tricky part is finding the right balance between cost and security. On one hand, we don't want to add too much padding because it adds a cost to data. On the other hand, we don't want to add too little, because the adversary can detect a small amount of padding with traffic analysis 🎲

Another defense is to not use dictionaries for cross-domain requests, and to only use shared brotli for the response when the origin is the same as where the content is hosted (using CORS). This stops an adversary from using a private dictionary with user secrets to compress content hosted on the adversary's origin. It also helps prevent CRIME attacks that try to benefit from a public dictionary, by preventing data compression with dictionaries for requests that don't originate from the host itself 🚧

The content of the dictionary itself should not be affected by external users. Allowing adversaries to control the dictionary allows a form of chosen plaintext attack 😱 Instead, only base the dictionary on content you control, or on generic large-scale content such as a spoken language. And update the dictionary at large time intervals (days, not seconds) to prevent fast probing ⏳

The use of HighwayHash [HWYHASH] for dictionary identifiers does not guarantee against collisions in an adversarial environment. It's intended for identifying the dictionary within a trusted, known set of dictionaries. In an adversarial environment, users of shared brotli should use another mechanism to validate a negotiated dictionary, such as a cryptographically proven secure hash. Stay safe out there, besties 🫶

10. IANA Considerations 🌷

This document has no IANA actions.

11. References 📚

11.1. Normative References

[HWYHASH] Alakuijala, J., Cox, B., and J. Wassenberg, "Fast keyed hash/pseudo-random function using SIMD multiply and permute", DOI 10.48550/arXiv.1612.06257, February 2017, <https://arxiv.org/abs/1612.06257>.

[RFC7932] Alakuijala, J. and Z. Szabadka, "Brotli Compressed Data Format", RFC 7932, DOI 10.17487/RFC7932, July 2016, <https://www.rfc-editor.org/info/rfc7932>.

11.2. Informative References

[CLOUDFLARE] Loring, B., "A Solution to Compression Oracles on the Web", The Cloudflare Blog, 27 March 2018, <https://blog.cloudflare.com/a-solution-to-compression-oracles-on-the-web/>.

[CRIME] CVE Program, "CVE-2012-4929", <https://www.cve.org/CVERecord?id=CVE-2012-4929>.

[LZ77] Ziv, J. and A. Lempel, "A Universal Algorithm for Sequential Data Compression", IEEE Transactions on Information Theory, vol. 23, no. 3, pp. 337-343, DOI 10.1109/TIT.1977.1055714, May 1977, <https://doi.org/10.1109/TIT.1977.1055714>.

[SOY] Google Developers, "Closure Tools", <https://developers.google.com/closure>.

Acknowledgments 🙌

The authors would like to thank Robert Obryk for suggesting improvements to the format and the text of the specification. Main character behavior from Robert 🌟

Authors' Addresses 💌

Jyrki Alakuijala Google, Inc. Email: jyrki@google.com

Thai Duong Google, Inc. Email: thaidn@google.com

Evgenii Kliuchnikov Google, Inc. Email: eustas@google.com

Zoltan Szabadka Google, Inc. Email: szabadka@google.com

Lode Vandevenne (editor) Google, Inc. Email: lode@google.com

📦 where this page comes from