RFC 7692

Compression Extensions for WebSocket

๐Ÿ“… published December 2015, from ietf

okay bestie, we're getting into RFC 7692 and it's giving protocol main character energy ๐Ÿ’–

RFC 7692: Compression Extensions for WebSocket ๐ŸŽ€

This one was written by T. Yoshino at Google, Inc., and published by the Internet Engineering Task Force (IETF) in December 2015. It's a Standards Track document, so it's the real deal, not a suggestion ๐Ÿ’… The ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates, so it's doing its own thing โœจ


Abstract โœจ

okay so this doc sets up a framework for building WebSocket extensions that add compression ๐Ÿ—œ๏ธ An extension built on it compresses the payload data part of WebSocket data messages, one message at a time, using parameters that get negotiated during the opening handshake. The framework gives a general way to apply a compression algorithm to message contents. Every compression algorithm needs its own defining document that spells out the parameter negotiation and the payload transformation in detail. This document also specifies one specific compression extension that uses the DEFLATE algorithm ๐Ÿฆ‹

Status of This Memo ๐Ÿ“œ

This is an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741.

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc7692.

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents ๐Ÿ“š

1. Introduction ....................................................3
2. Conformance Requirements and Terminology ........................3
3. Complementary Terminology .......................................4
4. WebSocket Per-Message Compression Extension .....................5
5. Extension Negotiation ...........................................5
   5.1. General Negotiation Flow ...................................9
   5.2. Negotiation Examples .......................................9
6. Framing ........................................................10
   6.1. Compression ...............................................10
   6.2. Decompression .............................................12
7. The "permessage-deflate" Extension .............................12
   7.1. Extension Parameters ......................................14
        7.1.1. Context Takeover Control ...........................14
        7.1.2. Limiting the LZ77 Sliding Window Size ..............16
        7.1.3. Examples ...........................................18
   7.2. Message Payload Transformation ............................19
        7.2.1. Compression ........................................19
        7.2.2. Decompression ......................................21
        7.2.3. Examples ...........................................22
   7.3. Implementation Notes ......................................25
8. Security Considerations ........................................25
9. IANA Considerations ............................................26
   9.1. Registration of the "permessage-deflate" WebSocket
        Extension Name ............................................26
   9.2. Registration of the "Per-Message Compressed"
        WebSocket Framing Header Bit ..............................26
10. References ....................................................27
   10.1. Normative References .....................................27
   10.2. Informative References ...................................27
Acknowledgements ..................................................28
Author's Address ..................................................28

1. Introduction ๐ŸŽ€

This document specifies a framework for adding compression to the WebSocket Protocol [RFC6455]. The framework explains how to define WebSocket Per-Message Compression Extensions (PMCEs) for a compression algorithm, based on the extension concept of the WebSocket Protocol specified in Section 9 of [RFC6455]. A WebSocket client and a peer WebSocket server negotiate whether to use a PMCE and work out the parameters needed to configure the compression algorithm during the WebSocket opening handshake. After that, the client and server can exchange data messages whose frames carry compressed data in the payload data portion ๐Ÿ’•

This framework only gives a general method for applying a compression algorithm to message contents. Each individual PMCE has to be specified in its own document, which describes in detail how to negotiate the configuration parameters for that PMCE's compression algorithm and how to transform (compress and decompress) data in the payload data portion ๐Ÿง 

A WebSocket client may offer multiple PMCEs during the opening handshake. The peer server that gets the offer can accept its preferred PMCE or decline all of them ๐Ÿ™ˆ PMCEs use the RSV1 bit of the WebSocket frame header to show whether a message is compressed, so an endpoint can choose not to compress messages whose contents won't compress well ๐Ÿ“

This document also specifies one PMCE based on the DEFLATE [RFC1951] algorithm. DEFLATE is widely available across platforms, and its overhead is small โœจ The extension name for this PMCE is "permessage-deflate". To line up the end of the compressed data on an octet boundary, this extension uses the algorithm from Section 2.1 of [RFC1979]. Endpoints can carry over the LZ77 sliding window [LZ77] that was used to build frames for previous messages, which gives a better compression ratio ๐Ÿ“ˆ For resource-limited devices, this extension also has parameters that limit memory usage for the compression context ๐Ÿ”‹

2. Conformance Requirements and Terminology ๐Ÿ’…

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

Requirements written as imperatives inside algorithms (like "strip any leading space characters" or "return false and abort these steps") are to be read with the force of the key word ("MUST", "SHOULD", "MAY", etc.) used to introduce the algorithm ๐Ÿซถ

Conformance requirements written as algorithms or specific steps can be implemented in any manner, as long as the end result is equivalent. The algorithms in this specification are meant to be easy to understand, not fast ๐Ÿข

This document references the procedure to _Fail the WebSocket Connection_. That procedure is defined in Section 7.1.7 of [RFC6455].

This document also references the event that _The WebSocket Connection is Established_ and the event that _A WebSocket Message Has Been Received_. Those events are defined in Sections 4.1 and 6.2, respectively, of [RFC6455] ๐Ÿ“จ

This document uses the Augmented Backus-Naur Form (ABNF) notation of [RFC5234]. The DIGIT (decimal 0-9) rule is included by reference, as defined in Appendix B.1 of [RFC5234] ๐Ÿ”ข

3. Complementary Terminology ๐ŸŽ€

This document defines some terms about WebSocket and WebSocket extension mechanisms that are underspecified or not defined at all in [RFC6455]. Here they are, bestie ๐Ÿ‘€

data message - a message made of data frames, as defined in Section 5.6 of [RFC6455].

message payload (or payload of a message) - the concatenation of the payload data portion of all data frames (see Section 6.2 of [RFC6455]) that represent a single message.

next extension in use after extension X - the next extension listed after X in the Sec-WebSocket-Extensions header in the server's opening handshake, as defined in Section 9.1 of [RFC6455]. This extension gets applied to outgoing data from the application right after X on the sender side, but it gets applied right before X to incoming data from the underlying transport ๐Ÿ”„

extension in use preceding extension X - the extension listed right before X in the Sec-WebSocket-Extensions header in the server's opening handshake. This extension gets applied to outgoing data from the application right before X on the sender side, but it gets applied right after X to incoming data from the underlying transport ๐Ÿ”

extension negotiation offer - each element in the Sec-WebSocket-Extensions header in the client's opening handshake ๐Ÿ“ค

extension negotiation response - each element in the Sec-WebSocket-Extensions header in the server's opening handshake ๐Ÿ“ฅ

corresponding extension negotiation response for an extension negotiation offer - an extension negotiation response that a server sends back to the peer client, containing the same extension name as the offer and meeting the requirements the offer represents ๐Ÿค

Accepting an extension negotiation offer - including a corresponding extension negotiation response for the offer in the Sec-WebSocket-Extensions header in the server's opening handshake โœ…

Declining an extension negotiation offer - not including a corresponding extension negotiation response for the offer in the Sec-WebSocket-Extensions header in the server's opening handshake โŒ

4. WebSocket Per-Message Compression Extension ๐Ÿฆ‹

WebSocket PMCEs are extensions to the WebSocket Protocol that enable compression. They're built on the extension concept of the WebSocket Protocol specified in Section 9 of [RFC6455]. PMCEs are defined individually for each compression algorithm that gets implemented, and they're registered in the "WebSocket Extension Name Registry" created in Section 11.4 of [RFC6455]. Each PMCE that refers to this framework MUST define the following:

  • The extension name of the PMCE, plus any applicable extension parameters that MUST be included in the Sec-WebSocket-Extensions header during the extension negotiation offer/response.
  • How to interpret the extension parameters exchanged during the opening handshake.
  • How to transform the payload of a message.

One PMCE is defined in Section 7 of this document and is registered in Section 9. Other PMCEs may be defined in the future in other documents ๐Ÿ”ฎ

Section 5 describes the basic extension negotiation process. Section 6 describes how to apply the compression algorithm, with the negotiated parameters, to the contents of WebSocket messages ๐Ÿ’œ

5. Extension Negotiation โœจ

okay so, to even offer a PMCE, a client MUST put the PMCE's extension name in the Sec-WebSocket-Extensions header of its opening handshake ๐Ÿค extension parameters are how you spell out the offer in detail. like, a client can list its favorite config values for the PMCE's compression algorithm ๐Ÿ’…

a client can also throw multiple PMCE choices at the server, one element per PMCE in the Sec-WebSocket-Extensions header ๐Ÿงš that set MAY even include several PMCEs with the same extension name, so you can try the same algorithm with different configs. the ORDER matters too, bc it shows what the client prefers. an element that comes earlier has higher preference. it is recommended that a server accepts PMCEs with higher preference if it supports them ๐Ÿ’–

a PMCE negotiation offer is basically full of requests and/or hints for the server ๐ŸŒธ

requests are the strict ones. a request sets constraints on the server's behavior that MUST be satisfied if the server accepts the offer ๐Ÿ˜ญ for example, say a server sends data compressed with DEFLATE to a client. the server has to keep the original bytes of the data it recently compressed and sent. the client has to keep the result of decompressing the bytes it recently received from the server. that amount of bytes is called the LZ77 window size ๐Ÿฆ‹ the client's LZ77 window size must not be less than the server's. so in a PMCE negotiation offer, the client MUST tell the server its LZ77 window size, so the server uses one that's not bigger than the client's. that restriction is an example of a request โœจ

hints are the chill ones. a hint tells the server something about the client's behavior that the server can either safely ignore or use when it decides its own behavior ๐Ÿซถ for example, say a client sends DEFLATE-compressed data to a server. the client has to keep the original bytes it recently compressed and sent. the server has to keep the result of decompressing what it recently received. the server's LZ77 window size must not be less than the client's. in a PMCE negotiation offer, the client MAY tell the server the maximum LZ77 window size it can afford, so the server can pick one that's not bigger than that max ๐Ÿ“ this is an example of a hint. honestly, using a window bigger than what the client actually uses is just wasting memory ๐Ÿ’€ so with the hint, the server can avoid that waste. but since a hint doesn't actually set constraints on the endpoints, the server MUST use the "agreed parameters" (defined below) to explicitly ask the client not to use an LZ77 window size bigger than the server's own ๐Ÿ‘‘

to accept an offered PMCE, a server MUST include the PMCE's extension name in the Sec-WebSocket-Extensions header of its opening handshake ๐ŸŽ€ the extension parameters in that response are the detailed configs the PMCE will use. these parameters and their values are called "agreed parameters" ๐Ÿ’œ the element MUST represent a PMCE that the server fully supports ๐Ÿ”ฅ and it doesn't have to match the received offer word for word. for example, say a server gets a PMCE offer with a parameter "X" saying the client can turn on a feature called X. the server can accept that offer with an element that leaves out "X", which just means the server chose NOT to turn on feature X. in that case the offer had "X" in it, but the "agreed parameters" don't ๐Ÿ™ˆ

the "agreed parameters" have to show how the requests and hints in the client's offer were handled, plus the server's own requests and hints about the client's behavior ๐ŸŒˆ that way the client can set itself up without having to figure out exactly which offer got accepted. literally so smart ๐Ÿง 

for example, if a client sends one offer with a parameter "enable_compression" and another without it, and the server accepts the first one, the server sends back an element with a parameter that acknowledges "enable_compression" โœจ the acknowledging name doesn't have to match the offer exactly. like, "enable_strong_compression" and "enable_weak_compression" could both be defined as acknowledging parameters for "enable_compression" ๐Ÿ“

compression features can work differently in each direction ๐Ÿฆ‹ so for those, the acknowledging parameter and the one going the other way have to be picked so they're distinguishable. for example, you can stick a "server_" prefix on parameters that affect data the server sends, and a "client_" prefix on parameters that affect data the client sends ๐Ÿซง

a server MUST NOT accept a PMCE negotiation offer together with another extension if the PMCE will conflict with that extension over who gets to use the RSV1 bit ๐Ÿ˜ญ and if a client gets a response accepting a PMCE offer together with such an extension, the client MUST Fail the WebSocket Connection ๐Ÿ’€

a server also MUST NOT accept a PMCE negotiation offer together with another extension if the PMCE will be applied to the output of that extension AND any of these conditions apply to the extension ๐Ÿšซ

  • The extension requires the boundary of frames to be preserved between the output from the extension at the sender and the input to the extension at the receiver.
  • The extension uses the "Extension data" field or any of the reserved bits on the WebSocket header as a per-frame attribute.

and yep, same deal for clients: if a client receives a response accepting a PMCE offer together with such an extension, the client MUST Fail the WebSocket Connection ๐Ÿ™…โ€โ™€๏ธ

a server that declines ALL offered PMCEs MUST NOT include any element with PMCE names ๐Ÿ™ƒ if the server responds with no PMCE element in the Sec-WebSocket-Extensions header, then both endpoints just go without per-message compression once the WebSocket Connection is established ๐ŸŒธ

and if a server gives an invalid response, like accepting a PMCE the client never offered, the client MUST Fail the WebSocket Connection ๐Ÿ˜ค no vibes allowed

finally, if a server responds with a valid PMCE element in the Sec-WebSocket-Extensions header and the WebSocket Connection is established, both endpoints MUST use the algorithm described in Section 6 ๐Ÿ’– and the message payload transformation (compressing and decompressing) procedure of the PMCE, configured with the "agreed parameters" the server sent back, to exchange messages ๐Ÿงš


5.1. General Negotiation Flow

okay this section is the general flow of how the negotiation goes ๐Ÿ—บ๏ธ the detailed handling of parameters has to be spelled out in the document that specifies the PMCE, so this is just the big picture ๐ŸŒท

a client makes an offer that includes parameters saying:

  • Hints about how the client is planning to compress data
  • Requests about how the server compresses data
  • Limitations concerning the client's compression functionality

the peer server then decides how it's going to behave based on those parameters ๐Ÿค” if the server can AND wants to go ahead with this PMCE enabled, it responds to the client with parameters saying:

  • Requests about how the client compresses data
  • How the server will compress data

based on what the server sent back, the client works out its own behavior and decides whether it can AND wants to go ahead with this PMCE enabled ๐Ÿ’… if it can't or doesn't want to, the client starts the closing handshake with close code 1010 ๐Ÿฅ€


5.2. Negotiation Examples

these are example values for the Sec-WebSocket-Extensions header offering PMCEs โœจ permessage-foo and permessage-bar are just hypothetical extension names for PMCEs using the compression algorithms foo and bar ๐Ÿฆ‹

  • Offer the permessage-foo.
permessage-foo
  • Offer the permessage-foo with a parameter x with a value of 10.
permessage-foo; x=10

The value may be quoted.

permessage-foo; x="10"
  • Offer the permessage-foo as first choice and the permessage-bar as a fallback plan.
permessage-foo, permessage-bar
  • Offer the permessage-foo with a parameter use_y, which enables a feature y as first choice, and the permessage-foo without the use_y parameter as a fallback plan.
permessage-foo; use_y, permessage-foo

6. Framing ๐ŸŽ€

PMCEs only operate on data messages ๐Ÿ’–

this document gives the RSV1 bit of the WebSocket header to PMCEs and calls it the "Per-Message Compressed" bit ๐Ÿซถ on a WebSocket connection where a PMCE is in use, this bit tells you whether a message is compressed or not โœจ

a message whose "Per-Message Compressed" bit is set on its first fragment is called a "compressed message" ๐Ÿ—œ๏ธ the frames of a compressed message carry compressed data in the payload data portion. when an endpoint gets a compressed message, it decompresses the concatenation of the compressed data from the message's frames using the decompression procedure of the PMCE in use ๐Ÿงฉ then it takes the bytes that match the application data portion of that decompressed data and uses those for the _A WebSocket Message Has Been Received_ event, instead of the received data as-is ๐ŸŒˆ

a message whose "Per-Message Compressed" bit is unset on its first fragment is called an "uncompressed message" ๐Ÿ“ the frames of an uncompressed message carry the uncompressed original data as-is in the payload data portion. when an endpoint gets an uncompressed message, it just uses the concatenation of the application data portions of the message's frames, as-is, for the _A WebSocket Message Has Been Received_ event ๐Ÿ’œ

6.1. Compression ๐ŸŽ€

okay bestie, here's the compression recipe ๐Ÿ’… An endpoint MUST use the following algorithm to send a message in the form of a compressed message โœจ

  1. Squish the payload of the original message using the compression procedure of the PMCE ๐Ÿ—œ๏ธ The original message can come straight from the application layer, or from the output of another WebSocket extension, depending on which extensions got negotiated.
  2. Now process the compressed data:
    • If this PMCE is the last extension to process outgoing messages, build frame(s) using the compressed data instead of the original data for the message payload, set the "Per-Message Compressed" bit of the first frame, and then send the frame(s) as described in Section 6.1 of [RFC6455] ๐Ÿ’–
    • Otherwise, pass the transformed message payload and modified header values, including the "Per-Message Compressed" bit value set to 1, to the next extension after the PMCE. If the extension expects frames for input, build a frame for the message and pass it ๐Ÿฆ‹

An endpoint MUST use the following algorithm to send a message in the form of an uncompressed message ๐Ÿซถ

  1. Process the original data like this:
    • If this PMCE is the last extension to process outgoing messages, build frame(s) using the original data for the payload data portion as is, unset the "Per-Message Compressed" bit of the first frame, and then send the frame(s) as described in Section 6.1 of [RFC6455] ๐ŸŒˆ
    • Otherwise, pass the message payload and header values to the next extension after the PMCE as is. If the extension expects frames for input, build a frame for the message and pass it ๐Ÿ“

An endpoint MUST NOT set the "Per-Message Compressed" bit of control frames and non-first fragments of a data message. An endpoint receiving such a frame MUST Fail the WebSocket Connection ๐Ÿ˜ญ this one is serious, no cap ๐Ÿšซ

PMCEs do not change the opcode field. The opcode of the first frame of a compressed message indicates the opcode of the original message ๐Ÿงš

The payload data portion in frames generated by a PMCE is not subject to the constraints for the original data type. For example, the concatenation of the output data corresponding to the application data portion of frames of a compressed text message is not required to be valid UTF-8 ๐Ÿ™ˆ At the receiver, though, the payload data portion after decompression is subject to the constraints for the original data type again ๐Ÿ‘‘

6.2. Decompression ๐ŸŒธ

An endpoint MUST use the following algorithm to receive a message in the form of a compressed message. It's giving reverse-engineering energy, but in a cute way ๐Ÿ’œ

  1. Concatenate the payload data portion of the received frames of the compressed message. The received frames may be direct input from the underlying transport, or output of another WebSocket extension, depending on which extensions got negotiated.
  2. Decompress the concatenation by following the decompression procedure of the PMCE ๐Ÿ”“
  3. Now process the decompressed message:
    • If this is the last extension to process incoming messages, deliver the A WebSocket Message Has Been Received event to the application layer with the decompressed message payload and header values, including the "Per-Message Compressed" bit unset to 0 โœจ
    • Otherwise, pass the decompressed message payload and header values, including the "Per-Message Compressed" bit unset to 0, to the extension preceding the PMCE. If the extension expects frames for input, build a frame for the message and pass it ๐Ÿฆ‹

An endpoint MUST use the following algorithm to receive a message in the form of an uncompressed message ๐ŸŽ€

  1. Process the received message like this:
    • If this PMCE is the last extension to process incoming messages, deliver the A WebSocket Message Has Been Received event to the application layer with the received message payload and header values as is ๐Ÿ’–
    • Otherwise, pass the message payload and header values to the extension preceding the PMCE as is. If the extension expects frames for input, build a frame for the message and pass it ๐ŸŒˆ

7. The "permessage-deflate" Extension ๐Ÿฆ‹

okay this is the main event, the specific PMCE everyone's obsessed with ๐Ÿ˜ญ This section defines a specific PMCE called "permessage-deflate". It compresses the payload of a message using the DEFLATE algorithm [RFC1951] and uses the byte boundary alignment method introduced in [RFC1979] ๐Ÿ’…

This section uses the term "byte" with the same meaning as used in [RFC1951], i.e., 8 bits stored or transmitted as a unit (same as an octet) ๐Ÿง 

The registered extension name for this extension is "permessage-deflate" ๐Ÿท๏ธ

Four extension parameters are defined for "permessage-deflate" to help endpoints manage per-connection resource usage ๐Ÿ’œ

  • server_no_context_takeover
  • client_no_context_takeover
  • server_max_window_bits
  • client_max_window_bits

These parameters enable two methods (no_context_takeover and max_window_bits) of constraining memory usage, and each one can be applied independently to either direction of WebSocket traffic ๐ŸŒธ The extension parameters with the client_ prefix are used by the client to configure its compressor, and by the server to configure its decompressor. The extension parameters with the server_ prefix are used by the server to configure its compressor, and by the client to configure its decompressor. All four parameters are defined for both a client's extension negotiation offer and a server's extension negotiation response ๐ŸŽ€

A server MUST decline an extension negotiation offer for this extension if any of the following conditions are met:

  • The negotiation offer contains an extension parameter not defined for use in an offer.
  • The negotiation offer contains an extension parameter with an invalid value.
  • The negotiation offer contains multiple extension parameters with the same name.
  • The server doesn't support the offered configuration.

A client MUST Fail the WebSocket Connection if the peer server accepted an extension negotiation offer for this extension with an extension negotiation response meeting any of the following conditions:

  • The negotiation response contains an extension parameter not defined for use in a response.
  • The negotiation response contains an extension parameter with an invalid value.
  • The negotiation response contains multiple extension parameters with the same name.
  • The client does not support the configuration that the response represents.

The term "LZ77 sliding window" [LZ77] used in this section means the buffer used by the DEFLATE algorithm to store recently processed input. The DEFLATE compression algorithm searches the buffer for a match with the following input ๐Ÿ”

The term "use context takeover" used in this section means that the same LZ77 sliding window used by the endpoint to build frames of the previous sent message is reused to build frames of the next message to be sent ๐Ÿ”

7.1. Extension Parameters ๐Ÿ’œ

7.1.1. Context Takeover Control โœจ

7.1.1.1. The "server_no_context_takeover" Extension Parameter ๐Ÿซถ

A client MAY include the server_no_context_takeover extension parameter in an extension negotiation offer. This extension parameter has no value. By including this extension parameter in an extension negotiation offer, a client prevents the peer server from using context takeover. If the peer server doesn't use context takeover, the client doesn't need to reserve memory to retain the LZ77 sliding window between messages ๐Ÿงบ

Absence of this extension parameter in an extension negotiation offer indicates that the client can decompress a message that the server built using context takeover ๐ŸŒˆ

A server accepts an extension negotiation offer that includes the server_no_context_takeover extension parameter by including the server_no_context_takeover extension parameter in the corresponding extension negotiation response to send back to the client. The server_no_context_takeover extension parameter in an extension negotiation response has no value ๐Ÿ“

It is RECOMMENDED that a server supports the server_no_context_takeover extension parameter in an extension negotiation offer ๐Ÿ’–

A server MAY include the server_no_context_takeover extension parameter in an extension negotiation response even if the extension negotiation offer being accepted by the extension negotiation response didn't include the server_no_context_takeover extension parameter ๐Ÿฆ‹

7.1.1.2. The "client_no_context_takeover" Extension Parameter ๐ŸŒธ

A client MAY include the client_no_context_takeover extension parameter in an extension negotiation offer. This extension parameter has no value. By including this extension parameter in an extension negotiation offer, a client informs the peer server of a hint: even if the server doesn't include the client_no_context_takeover extension parameter in the corresponding extension negotiation response to the offer, the client is not going to use context takeover ๐Ÿคž

A server MAY include the client_no_context_takeover extension parameter in an extension negotiation response. If the received extension negotiation offer includes the client_no_context_takeover extension parameter, the server may either ignore the parameter, or use the parameter to avoid taking over the LZ77 sliding window unnecessarily by including the client_no_context_takeover extension parameter in the corresponding extension negotiation response to the offer. The client_no_context_takeover extension parameter in an extension negotiation response has no value. By including the client_no_context_takeover extension parameter in an extension negotiation response, a server prevents the peer client from using context takeover. This reduces the amount of memory that the server has to reserve for the connection ๐Ÿง 

Absence of this extension parameter in an extension negotiation response indicates that the server can decompress messages built by the client using context takeover ๐ŸŽ€

A client MUST support the client_no_context_takeover extension parameter in an extension negotiation response ๐Ÿ‘‘

7.1.2. Limiting the LZ77 Sliding Window Size ๐Ÿฆ‹

7.1.2.1. The "server_max_window_bits" Extension Parameter ๐Ÿ’–

Okay bestie, clients are allowed (MAY) to include server_max_window_bits in an extension negotiation offer ๐ŸŽ€. Its value is a decimal integer with no leading zeroes, from 8 to 15 inclusive, and it's the base-2 logarithm of the LZ77 sliding window size. It MUST conform to this ABNF:

server-max-window-bits = 1*DIGIT

So when a client sends this, it's basically saying "hey server, please keep your LZ77 window small when you compress stuff for me" ๐Ÿงš. If the server uses a small window, the client can shrink how much memory its own LZ77 window needs. Iconic for the client, ngl ๐Ÿ’….

If the server doesn't support this parameter, it declines the offer ๐Ÿ™…. And if the parameter isn't in the offer at all, the client can receive messages compressed with an LZ77 sliding window of up to 32,768 bytes ๐ŸŒธ.

A server accepts an offer with this parameter by including server_max_window_bits in its response, with the same or a smaller value than the offer. The response version also has a decimal integer with no leading zeroes, from 8 to 15 inclusive, meaning the base-2 logarithm of the LZ77 sliding window size, and it MUST conform to this ABNF:

server-max-window-bits = 1*DIGIT

A server MAY include server_max_window_bits in its response even if the offer it's accepting didn't include the parameter ๐Ÿฆ‹.

7.1.2.2. The "client_max_window_bits" Extension Parameter ๐Ÿ“

A client MAY include client_max_window_bits in an extension negotiation offer. This one is a bit different: it can have no value at all, or a decimal integer with no leading zeroes, from 8 to 15 inclusive, which is again the base-2 logarithm of the LZ77 sliding window size. If there IS a value, it MUST conform to this ABNF:

client-max-window-bits = 1*DIGIT

By putting this in an offer, the client tells the server that it supports the client_max_window_bits parameter in a response ๐Ÿซถ. If the client also attaches a value, that's a hint too. It means that even if the server doesn't send back client_max_window_bits with a bigger value than the offer (or doesn't send it at all), the client still won't use an LZ77 sliding window bigger than the value it offered when compressing messages ๐Ÿ”ฅ.

If a received offer has client_max_window_bits, the server MAY include it in the matching response. If the offer had a value, the server can either ignore that value, or use it to avoid allocating a window that's way bigger than it needs to be. To do the second thing, the server includes client_max_window_bits in the response with a value equal to or smaller than the one it received. That response value is also a decimal integer with no leading zeroes, from 8 to 15 inclusive, the base-2 logarithm of the LZ77 sliding window size, and it MUST conform to this ABNF:

client-max-window-bits = 1*DIGIT

When a server includes this parameter in a response, it limits the LZ77 sliding window the client uses to compress messages ๐Ÿ‘‘. That also means less memory the server has to reserve for its decompression context on the connection. Lowkey genius ๐Ÿง .

If a received offer doesn't have client_max_window_bits, the matching response MUST NOT include client_max_window_bits โŒ. And if the response doesn't include it, the server can receive messages compressed with an LZ77 sliding window of up to 32,768 bytes โœจ.

7.1.3. Examples ๐ŸŽ€

The simplest Sec-WebSocket-Extensions header in a client's opening handshake, offering the "permessage-deflate" extension, looks like this:

Sec-WebSocket-Extensions: permessage-deflate

Since client_max_window_bits isn't in this offer, the server must not accept it with a response that includes client_max_window_bits ๐Ÿ™ˆ. The simplest Sec-WebSocket-Extensions header in a server's opening handshake, accepting "permessage-deflate", is the same:

Sec-WebSocket-Extensions: permessage-deflate

Next, here's an offer from a client asking the server to use an LZ77 sliding window of 1,024 bytes or less, and saying it supports the client_max_window_bits parameter in a response:

Sec-WebSocket-Extensions:
    permessage-deflate;
    client_max_window_bits; server_max_window_bits=10

The server might reject that one, because it doesn't support server_max_window_bits in an offer. That's totally fine if the client can't handle messages compressed with a bigger window. But if the client would be happy with a plain "permessage-deflate" as a fallback, it can offer both options like this ๐Ÿฆ‹:

Sec-WebSocket-Extensions:
    permessage-deflate;
    client_max_window_bits; server_max_window_bits=10,
    permessage-deflate;
    client_max_window_bits

The server can accept "permessage-deflate" by picking any one it supports from the list. If it goes with the first option, its response could look like this:

Sec-WebSocket-Extensions:
    permessage-deflate; server_max_window_bits=10

If it goes with the second option, the response could look like this:

Sec-WebSocket-Extensions: permessage-deflate

7.2. Message Payload Transformation ๐ŸŒˆ

7.2.1. Compression ๐Ÿ’œ

An endpoint uses the following algorithm to compress a message ๐Ÿซง:

  1. Compress all the octets of the payload of the message using DEFLATE.
  2. If the resulting data does not end with an empty DEFLATE block with no compression (the BTYPE bits are set to 00), append an empty DEFLATE block with no compression to the tail end.
  3. Remove 4 octets (that are 0x00 0x00 0xff 0xff) from the tail end. After this step, the last octet of the compressed data contains (possibly part of) the DEFLATE header bits with the BTYPE bits set to 00.

When using DEFLATE in the first step above, a few things are allowed:

  • An endpoint MAY use multiple DEFLATE blocks to compress one message.
  • An endpoint MAY use DEFLATE blocks of any type.
  • An endpoint MAY use both DEFLATE blocks with the BFINAL bit set to 0 and DEFLATE blocks with the BFINAL bit set to 1.
  • When any DEFLATE block with the BFINAL bit set to 1 doesn't end at a byte boundary, an endpoint MUST add minimal padding bits of 0 to make it end at a byte boundary. The next DEFLATE block follows the padded data if any.

An endpoint fragments a compressed message by splitting the result of running this algorithm. Even when only part of the payload is available, a fragment can be built by compressing the available data and choosing the block type appropriately, so that the end of the resulting compressed data is aligned at a byte boundary. Note that for non-final fragments, the removal of 0x00 0x00 0xff 0xff MUST NOT be done ๐Ÿ™…โ€โ™€๏ธ.

An endpoint MUST NOT use an LZ77 sliding window longer than 32,768 bytes to compress messages to send ๐Ÿšซ.

If the "agreed parameters" contain the client_no_context_takeover extension parameter, the client MUST start compressing each new message with an empty LZ77 sliding window. Otherwise, the client MAY take over the LZ77 sliding window used to build the last compressed message. Note that even if the client included client_no_context_takeover in its offer, the client MAY take over the LZ77 sliding window used to build the last compressed message if the "agreed parameters" don't contain client_no_context_takeover. The client-to-server client_no_context_takeover extension parameter is just a hint for the server to build an extension negotiation response ๐ŸŽ€.

If the "agreed parameters" contain the server_no_context_takeover extension parameter, the server MUST start compressing each new message with an empty LZ77 sliding window. Otherwise, the server MAY take over the LZ77 sliding window used to build the last compressed message ๐Ÿงš.

If the "agreed parameters" contain the client_max_window_bits extension parameter with a value of w, the client MUST NOT use an LZ77 sliding window longer than the w-th power of 2 bytes to compress messages to send. Note that even if the client included client_max_window_bits in its offer with a value smaller than the one in the "agreed parameters", the client MAY use an LZ77 sliding window of any size to compress messages to send, as long as that size conforms to the "agreed parameters". The client-to-server client_max_window_bits extension parameter is just a hint for the server to build an extension negotiation response ๐Ÿซถ.

If the "agreed parameters" contain the server_max_window_bits extension parameter with a value of w, the server MUST NOT use an LZ77 sliding window longer than the w-th power of 2 bytes to compress messages to send ๐Ÿ’….

7.2.2. Decompression โœจ

okay bestie, here's how an endpoint decompresses a message ๐Ÿซถ

  1. Append 4 octets of 0x00 0x00 0xff 0xff to the tail end of the payload of the message.
  2. Decompress the resulting data using DEFLATE.

so if the "agreed parameters" include server_no_context_takeover, the client MAY decompress each new message with an empty LZ77 sliding window ๐Ÿงน. otherwise the client MUST decompress each new message using the LZ77 sliding window that was used to process the last compressed message ๐Ÿ”’.

same energy for the server ๐Ÿ’…. if the "agreed parameters" include client_no_context_takeover, the server MAY decompress each new message with an empty LZ77 sliding window. otherwise the server MUST decompress each new message using the LZ77 sliding window used to process the last compressed message.

โš ๏ธ bestie, read this part twice. even if the client put client_no_context_takeover in its offer, the server MUST decompress each new message using the LZ77 sliding window used to process the last compressed message if the "agreed parameters" don't contain client_no_context_takeover. the client-to-server client_no_context_takeover parameter is just a hint for the server to build an extension negotiation response ๐ŸŽ€.

window sizes next ๐ŸชŸ. if the "agreed parameters" include server_max_window_bits with a value of w, the client MAY reduce the size of its LZ77 sliding window to decompress received messages down to the w-th power of 2 bytes. otherwise the client MUST use a 32,768-byte LZ77 sliding window to decompress received messages.

and the server side of that ๐Ÿฆ‹: if the "agreed parameters" include client_max_window_bits with a value of w, the server MAY reduce the size of its LZ77 sliding window to decompress received messages down to the w-th power of 2 bytes. otherwise the server MUST use a 32,768-byte LZ77 sliding window to decompress received messages.

last bit, and it's a big one ๐Ÿ‘‘. even if the client included a client_max_window_bits value in its offer that is smaller than the one in the "agreed parameters", the client MUST use an LZ77 sliding window of a size that conforms to the "agreed parameters" to compress messages it sends. the client-to-server client_max_window_bits parameter is, again, just a hint for the server to build an extension negotiation response ๐Ÿ’–.

7.2.3. Examples

this section is the fun part ๐Ÿงš. it introduces examples of how the "permessage-deflate" extension transforms messages.

7.2.3.1. A Message Compressed Using One Compressed DEFLATE Block

say an endpoint sends the text message "Hello" ๐Ÿ‘‹. if it uses one compressed DEFLATE block (compressed with fixed Huffman code and the "BFINAL" bit not set), the endpoint gets the compressed data for the message payload like this ๐Ÿ“.

the endpoint compresses "Hello" into one compressed DEFLATE block and flushes the resulting data into a byte array using an empty DEFLATE block with no compression:

    0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 0x00 0x00 0xff 0xff

by stripping 0x00 0x00 0xff 0xff from the tail end, the endpoint gets the data for the message payload:

    0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00

say the endpoint sends this compressed message without fragmentation ๐Ÿ™ˆ. it builds one frame by putting all the compressed data in the payload data portion of the frame:

    0xc1 0x07 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00

the first 2 octets (0xc1 0x07) are the WebSocket frame header (FIN=1, RSV1=1, RSV2=0, RSV3=0, opcode=text, MASK=0, Payload length=7). here's what goes in each field ๐Ÿ’…:

     0                   1
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
    +-+-+-+-+-------+-+-------------+
    |F|R|R|R| opcode|M| Payload len |
    |I|S|S|S|       |A|             |
    |N|V|V|V|       |S|             |
    | |1|2|3|       |K|             |
    +-+-+-+-+-------+-+-------------+
    |1|1|0|0|   1   |0|      7      |
    +-+-+-+-+-------+-+-------------+

now say the endpoint sends the compressed message with fragmentation ๐Ÿงฉ. it splits the compressed data into fragments and builds a frame for each fragment. for example, if the fragments are 3 and 4 octets, the first frame is:

    0x41 0x03 0xf2 0x48 0xcd

and the second frame is:

    0x80 0x04 0xc9 0xc9 0x07 0x00

note that the RSV1 bit is set only on the first frame ๐ŸŽ€.

7.2.3.2. Sharing LZ77 Sliding Window

okay this one is giving dรฉjร  vu ๐ŸŒ€. say a client has sent a message "Hello" as a compressed message and will send the same message "Hello" again as a compressed message:

    0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00

that's the payload of the first message the client has sent. if the "agreed parameters" include the client_no_context_takeover extension parameter, the client compresses the payload of the next message into the same bytes (as long as it uses the same "BTYPE" value and "BFINAL" value). so the payload of the second message will be:

    0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00

if the "agreed parameters" did not contain the client_no_context_takeover extension parameter, the client can compress the payload of the next message into fewer bytes by referencing the history in the LZ77 sliding window ๐Ÿช„. so the payload of the second message will be:

    0xf2 0x00 0x11 0x00 0x00

so 2 bytes are saved in total ๐Ÿค‘.

note that even if some uncompressed messages (with the RSV1 bit unset) are inserted between the two "Hello" messages, they don't affect the LZ77 sliding window ๐Ÿฆ‹.

7.2.3.3. Using a DEFLATE Block with No Compression

a DEFLATE block with no compression may be used.

    0xc1 0x0b 0x00 0x05 0x00 0xfa 0xff 0x48 0x65 0x6c 0x6c 0x6f 0x00

this is a frame that makes up a text message "Hello", built using a DEFLATE block with no compression ๐ŸŒธ. the first 2 octets (0xc1 0x0b) are the WebSocket frame header (FIN=1, RSV1=1, RSV2=0, RSV3=0, opcode=text, MASK=0, Payload length=7). note that the RSV1 bit is set for this message (only on the first fragment if the message is fragmented) because the RSV1 bit is set when DEFLATE is applied to the message, including when only DEFLATE blocks with no compression are used ๐Ÿ˜ญ. the 3rd to 13th octets are the payload data, which holds "Hello" compressed using a DEFLATE block with no compression.

7.2.3.4. Using a DEFLATE Block with "BFINAL" Set to 1

on platforms where the flush method using an empty DEFLATE block with no compression isn't available, implementors can choose to flush data using DEFLATE blocks with "BFINAL" set to 1 ๐Ÿ› ๏ธ.

    0xf3 0x48 0xcd 0xc9 0xc9 0x07 0x00 0x00

this is the payload of a message containing "Hello" compressed using a DEFLATE block with "BFINAL" set to 1. the first 7 octets make up a DEFLATE block with "BFINAL" set to 1 and "BTYPE" set to 01, containing "Hello". the last 1 octet (0x00) holds the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, plus 5 padding bits of 0. that octet is needed so the payload can be decompressed the same way as messages flushed using DEFLATE blocks with "BFINAL" unset โœจ.

7.2.3.5. Two DEFLATE Blocks in One Message

two or more DEFLATE blocks can show up in one message ๐Ÿง.

    0xf2 0x48 0x05 0x00 0x00 0x00 0xff 0xff 0xca 0xc9 0xc9 0x07 0x00

the first 3 octets (0xf2 0x48 0x05) and the least significant two bits of the 4th octet (0x00) make up one DEFLATE block with "BFINAL" set to 0 and "BTYPE" set to 01, containing "He". the rest of the 4th octet holds the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, plus the 3 padding bits of 0. together with the following 4 octets (0x00 0x00 0xff 0xff), those header bits make up an empty DEFLATE block with no compression. a DEFLATE block containing "llo" follows the empty one ๐Ÿซง.

7.2.3.6. Generating an Empty Fragment

say an endpoint is sending data of unknown size ๐Ÿ“ฆ. it might hit the end-of-data signal from the data source while its buffer for uncompressed data is empty. in that case, the endpoint just needs to send the last fragment with the FIN bit set to 1 and the payload set to the DEFLATE block(s), which hold 0 bytes of data. if the compression library in use doesn't output anything when its buffer is empty, an empty uncompressed DEFLATE block can be built and used for this purpose, like so ๐Ÿญ:

    0x00

the single octet 0x00 holds the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, plus 5 padding bits of 0.

7.3. Implementation Notes

okay, implementation tips ๐Ÿง . on most common software development platforms, the DEFLATE compression library gives you a way to align compressed data to byte boundaries using an empty DEFLATE block with no compression. for example, zlib [zlib] does this when Z_SYNC_FLUSH is passed to the deflate function ๐Ÿ’œ.

some platforms only provide ways to output and process compressed data with a zlib header and an Adler-32 checksum. on those platforms, developers need to write stub code to remove and complement the zlib and Adler-32 checksum by themselves ๐Ÿ› ๏ธ.

to get a useful compression ratio, an LZ77 sliding window size of 1,024 or more is RECOMMENDED ๐Ÿ“.

if a side disallows context takeover, its endpoint can easily figure out whether or not a certain message will be shorter if compressed ๐ŸŽฏ. otherwise, it's not easy to know whether future messages will benefit from having a certain message compressed. implementors may use some heuristics to figure this out ๐Ÿ”ฎ.


8. Security Considerations ๐Ÿšจ

๐Ÿšจ bestie, this one is serious. there is a known exploit when history-based compression is combined with a secure transport [CRIME]. implementors should pay attention to this when integrating this extension with other extensions or protocols. don't skip it ๐Ÿ’€.

9. IANA Considerations ๐ŸŒˆ

9.1. Registration of the "permessage-deflate" WebSocket Extension Name

IANA has registered the following WebSocket extension name in the "WebSocket Extension Name Registry" defined in [RFC6455] ๐ŸŽ€:

    Extension Identifier
       permessage-deflate

    Extension Common Name
       WebSocket Per-Message Deflate

    Extension Definition
       This document.

    Known Incompatible Extensions
       None

the "permessage-deflate" extension name is used in the Sec-WebSocket-Extensions header in the WebSocket opening handshake to negotiate use of the "permessage-deflate" extension ๐Ÿค.

9.2. Registration of the "Per-Message Compressed" WebSocket Framing Header Bit

IANA has registered the following WebSocket framing header bit in the "WebSocket Framing Header Bits Registry" defined in [RFC6455] ๐Ÿฆ‹:

    Value
       RSV1

    Description
       The "Per-Message Compressed" bit, which indicates whether or not
       the message is compressed.  RSV1 is set for compressed messages
       and unset for uncompressed messages.

    Reference
       Section 6 of this document.

the "Per-Message Compressed" framing header bit is used on the first fragment of data messages to indicate whether the payload of the message is compressed by the PMCE or not ๐Ÿ’–.

10. References ๐Ÿ“š

10.1. Normative References ๐Ÿ“š

okay, these are the ones you have to read to understand the spec ๐Ÿซถ

[CRIME] Rizzo, J. and T. Duong, "The CRIME attack", EKOparty Security Conference, September 2012.

[LZ77] Ziv, J. and A. Lempel, "A Universal Algorithm for Sequential Data Compression", IEEE Transactions on Information Theory, Vol. 23, No. 3, pp. 337-343, DOI 10.1109/TIT.1977.1055714, May 1977, <https://www.cs.duke.edu/courses/spring03/cps296.5/papers/ziv_lempel_1977_universal_algorithm.pdf>.

[RFC1951] Deutsch, P., "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, DOI 10.17487/RFC1951, May 1996, <http://www.rfc-editor.org/info/rfc1951>.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <http://www.rfc-editor.org/info/rfc2119>.

[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/RFC5234, January 2008, <http://www.rfc-editor.org/info/rfc5234>.

[RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol", RFC 6455, DOI 10.17487/RFC6455, December 2011, <http://www.rfc-editor.org/info/rfc6455>.


10.2. Informative References ๐ŸŒธ

these ones are extra reading, not required, but they're giving context and I love a good backstory ๐Ÿงš

[RFC1979] Woods, J., "PPP Deflate Protocol", RFC 1979, DOI 10.17487/RFC1979, August 1996, <http://www.rfc-editor.org/info/rfc1979>.

[zlib] Gailly, J. and M. Adler, "zlib", <http://www.zlib.net/>.


Acknowledgements ๐Ÿ’–

okay, big thank-you to Patrick McManus, who wrote up the first version of a DEFLATE-based compression extension for the WebSocket Protocol. I used that as a reference while writing this spec, so he's lowkey the OG ๐Ÿ‘‘

Thanks also to everyone below, who joined the discussions on the HyBi WG and shared ideas and/or gave really detailed reviews ๐Ÿซถ (the list is probably incomplete, so if your name isn't here, I'm sorry bestie ๐Ÿ˜ญ): Adam Rice, Alexander Philippou, Alexey Melnikov, Arman Djusupov, Bjoern Hoehrmann, Brian McKelvey, Dario Crivelli, Greg Wilkins, Inaki Baz Castillo, Jamie Lokier, Joakim Erdfelt, John A. Tamplin, Julian Reschke, Kenichi Ishibashi, Mark Nottingham, Peter Thorson, Roberto Peon, Salvatore Loreto, Simone Bordet, Tobias Oberstein, and Yutaka Hirano. Just so you know, the people listed didn't necessarily agree with where the work ended up ๐Ÿ’…


Author's Address ๐Ÿฆ‹

Takeshi Yoshino Google, Inc.

Email: tyoshino@google.com

๐Ÿ“ฆ where this page comes from