RFC 7692
Compression Extensions for WebSocket
okay bestie, we're getting into RFC 7692 and it's giving protocol main character energy ๐
RFC 7692: Compression Extensions for WebSocket ๐
This one was written by T. Yoshino at Google, Inc., and published by the Internet Engineering Task Force (IETF) in December 2015. It's a Standards Track document, so it's the real deal, not a suggestion ๐ The ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates, so it's doing its own thing โจ
Abstract โจ
okay so this doc sets up a framework for building WebSocket extensions that add compression ๐๏ธ An extension built on it compresses the payload data part of WebSocket data messages, one message at a time, using parameters that get negotiated during the opening handshake. The framework gives a general way to apply a compression algorithm to message contents. Every compression algorithm needs its own defining document that spells out the parameter negotiation and the payload transformation in detail. This document also specifies one specific compression extension that uses the DEFLATE algorithm ๐ฆ
Status of This Memo ๐
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741.
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc7692.
Copyright Notice ยฉ
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Table of Contents ๐
1. Introduction ....................................................3
2. Conformance Requirements and Terminology ........................3
3. Complementary Terminology .......................................4
4. WebSocket Per-Message Compression Extension .....................5
5. Extension Negotiation ...........................................5
5.1. General Negotiation Flow ...................................9
5.2. Negotiation Examples .......................................9
6. Framing ........................................................10
6.1. Compression ...............................................10
6.2. Decompression .............................................12
7. The "permessage-deflate" Extension .............................12
7.1. Extension Parameters ......................................14
7.1.1. Context Takeover Control ...........................14
7.1.2. Limiting the LZ77 Sliding Window Size ..............16
7.1.3. Examples ...........................................18
7.2. Message Payload Transformation ............................19
7.2.1. Compression ........................................19
7.2.2. Decompression ......................................21
7.2.3. Examples ...........................................22
7.3. Implementation Notes ......................................25
8. Security Considerations ........................................25
9. IANA Considerations ............................................26
9.1. Registration of the "permessage-deflate" WebSocket
Extension Name ............................................26
9.2. Registration of the "Per-Message Compressed"
WebSocket Framing Header Bit ..............................26
10. References ....................................................27
10.1. Normative References .....................................27
10.2. Informative References ...................................27
Acknowledgements ..................................................28
Author's Address ..................................................28
1. Introduction ๐
This document specifies a framework for adding compression to the WebSocket Protocol [RFC6455]. The framework explains how to define WebSocket Per-Message Compression Extensions (PMCEs) for a compression algorithm, based on the extension concept of the WebSocket Protocol specified in Section 9 of [RFC6455]. A WebSocket client and a peer WebSocket server negotiate whether to use a PMCE and work out the parameters needed to configure the compression algorithm during the WebSocket opening handshake. After that, the client and server can exchange data messages whose frames carry compressed data in the payload data portion ๐
This framework only gives a general method for applying a compression algorithm to message contents. Each individual PMCE has to be specified in its own document, which describes in detail how to negotiate the configuration parameters for that PMCE's compression algorithm and how to transform (compress and decompress) data in the payload data portion ๐ง
A WebSocket client may offer multiple PMCEs during the opening handshake. The peer server that gets the offer can accept its preferred PMCE or decline all of them ๐ PMCEs use the RSV1 bit of the WebSocket frame header to show whether a message is compressed, so an endpoint can choose not to compress messages whose contents won't compress well ๐
This document also specifies one PMCE based on the DEFLATE [RFC1951] algorithm. DEFLATE is widely available across platforms, and its overhead is small โจ The extension name for this PMCE is "permessage-deflate". To line up the end of the compressed data on an octet boundary, this extension uses the algorithm from Section 2.1 of [RFC1979]. Endpoints can carry over the LZ77 sliding window [LZ77] that was used to build frames for previous messages, which gives a better compression ratio ๐ For resource-limited devices, this extension also has parameters that limit memory usage for the compression context ๐
2. Conformance Requirements and Terminology ๐
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
Requirements written as imperatives inside algorithms (like "strip any leading space characters" or "return false and abort these steps") are to be read with the force of the key word ("MUST", "SHOULD", "MAY", etc.) used to introduce the algorithm ๐ซถ
Conformance requirements written as algorithms or specific steps can be implemented in any manner, as long as the end result is equivalent. The algorithms in this specification are meant to be easy to understand, not fast ๐ข
This document references the procedure to _Fail the WebSocket Connection_. That procedure is defined in Section 7.1.7 of [RFC6455].
This document also references the event that _The WebSocket Connection is Established_ and the event that _A WebSocket Message Has Been Received_. Those events are defined in Sections 4.1 and 6.2, respectively, of [RFC6455] ๐จ
This document uses the Augmented Backus-Naur Form (ABNF) notation of [RFC5234]. The DIGIT (decimal 0-9) rule is included by reference, as defined in Appendix B.1 of [RFC5234] ๐ข
3. Complementary Terminology ๐
This document defines some terms about WebSocket and WebSocket extension mechanisms that are underspecified or not defined at all in [RFC6455]. Here they are, bestie ๐
data message - a message made of data frames, as defined in Section 5.6 of [RFC6455].
message payload (or payload of a message) - the concatenation of the payload data portion of all data frames (see Section 6.2 of [RFC6455]) that represent a single message.
next extension in use after extension X - the next extension listed after X in the Sec-WebSocket-Extensions header in the server's opening handshake, as defined in Section 9.1 of [RFC6455]. This extension gets applied to outgoing data from the application right after X on the sender side, but it gets applied right before X to incoming data from the underlying transport ๐
extension in use preceding extension X - the extension listed right before X in the Sec-WebSocket-Extensions header in the server's opening handshake. This extension gets applied to outgoing data from the application right before X on the sender side, but it gets applied right after X to incoming data from the underlying transport ๐
extension negotiation offer - each element in the Sec-WebSocket-Extensions header in the client's opening handshake ๐ค
extension negotiation response - each element in the Sec-WebSocket-Extensions header in the server's opening handshake ๐ฅ
corresponding extension negotiation response for an extension negotiation offer - an extension negotiation response that a server sends back to the peer client, containing the same extension name as the offer and meeting the requirements the offer represents ๐ค
Accepting an extension negotiation offer - including a corresponding extension negotiation response for the offer in the Sec-WebSocket-Extensions header in the server's opening handshake โ
Declining an extension negotiation offer - not including a corresponding extension negotiation response for the offer in the Sec-WebSocket-Extensions header in the server's opening handshake โ
4. WebSocket Per-Message Compression Extension ๐ฆ
WebSocket PMCEs are extensions to the WebSocket Protocol that enable compression. They're built on the extension concept of the WebSocket Protocol specified in Section 9 of [RFC6455]. PMCEs are defined individually for each compression algorithm that gets implemented, and they're registered in the "WebSocket Extension Name Registry" created in Section 11.4 of [RFC6455]. Each PMCE that refers to this framework MUST define the following:
- The extension name of the PMCE, plus any applicable extension parameters that MUST be included in the
Sec-WebSocket-Extensionsheader during the extension negotiation offer/response. - How to interpret the extension parameters exchanged during the opening handshake.
- How to transform the payload of a message.
One PMCE is defined in Section 7 of this document and is registered in Section 9. Other PMCEs may be defined in the future in other documents ๐ฎ
Section 5 describes the basic extension negotiation process. Section 6 describes how to apply the compression algorithm, with the negotiated parameters, to the contents of WebSocket messages ๐
5. Extension Negotiation โจ
okay so, to even offer a PMCE, a client MUST put the PMCE's extension name in the Sec-WebSocket-Extensions header of its opening handshake ๐ค extension parameters are how you spell out the offer in detail. like, a client can list its favorite config values for the PMCE's compression algorithm ๐
a client can also throw multiple PMCE choices at the server, one element per PMCE in the Sec-WebSocket-Extensions header ๐ง that set MAY even include several PMCEs with the same extension name, so you can try the same algorithm with different configs. the ORDER matters too, bc it shows what the client prefers. an element that comes earlier has higher preference. it is recommended that a server accepts PMCEs with higher preference if it supports them ๐
a PMCE negotiation offer is basically full of requests and/or hints for the server ๐ธ
requests are the strict ones. a request sets constraints on the server's behavior that MUST be satisfied if the server accepts the offer ๐ญ for example, say a server sends data compressed with DEFLATE to a client. the server has to keep the original bytes of the data it recently compressed and sent. the client has to keep the result of decompressing the bytes it recently received from the server. that amount of bytes is called the LZ77 window size ๐ฆ the client's LZ77 window size must not be less than the server's. so in a PMCE negotiation offer, the client MUST tell the server its LZ77 window size, so the server uses one that's not bigger than the client's. that restriction is an example of a request โจ
hints are the chill ones. a hint tells the server something about the client's behavior that the server can either safely ignore or use when it decides its own behavior ๐ซถ for example, say a client sends DEFLATE-compressed data to a server. the client has to keep the original bytes it recently compressed and sent. the server has to keep the result of decompressing what it recently received. the server's LZ77 window size must not be less than the client's. in a PMCE negotiation offer, the client MAY tell the server the maximum LZ77 window size it can afford, so the server can pick one that's not bigger than that max ๐ this is an example of a hint. honestly, using a window bigger than what the client actually uses is just wasting memory ๐ so with the hint, the server can avoid that waste. but since a hint doesn't actually set constraints on the endpoints, the server MUST use the "agreed parameters" (defined below) to explicitly ask the client not to use an LZ77 window size bigger than the server's own ๐
to accept an offered PMCE, a server MUST include the PMCE's extension name in the Sec-WebSocket-Extensions header of its opening handshake ๐ the extension parameters in that response are the detailed configs the PMCE will use. these parameters and their values are called "agreed parameters" ๐ the element MUST represent a PMCE that the server fully supports ๐ฅ and it doesn't have to match the received offer word for word. for example, say a server gets a PMCE offer with a parameter "X" saying the client can turn on a feature called X. the server can accept that offer with an element that leaves out "X", which just means the server chose NOT to turn on feature X. in that case the offer had "X" in it, but the "agreed parameters" don't ๐
the "agreed parameters" have to show how the requests and hints in the client's offer were handled, plus the server's own requests and hints about the client's behavior ๐ that way the client can set itself up without having to figure out exactly which offer got accepted. literally so smart ๐ง
for example, if a client sends one offer with a parameter "enable_compression" and another without it, and the server accepts the first one, the server sends back an element with a parameter that acknowledges "enable_compression" โจ the acknowledging name doesn't have to match the offer exactly. like, "enable_strong_compression" and "enable_weak_compression" could both be defined as acknowledging parameters for "enable_compression" ๐
compression features can work differently in each direction ๐ฆ so for those, the acknowledging parameter and the one going the other way have to be picked so they're distinguishable. for example, you can stick a "server_" prefix on parameters that affect data the server sends, and a "client_" prefix on parameters that affect data the client sends ๐ซง
a server MUST NOT accept a PMCE negotiation offer together with another extension if the PMCE will conflict with that extension over who gets to use the RSV1 bit ๐ญ and if a client gets a response accepting a PMCE offer together with such an extension, the client MUST Fail the WebSocket Connection ๐
a server also MUST NOT accept a PMCE negotiation offer together with another extension if the PMCE will be applied to the output of that extension AND any of these conditions apply to the extension ๐ซ
- The extension requires the boundary of frames to be preserved between the output from the extension at the sender and the input to the extension at the receiver.
- The extension uses the "Extension data" field or any of the reserved bits on the WebSocket header as a per-frame attribute.
and yep, same deal for clients: if a client receives a response accepting a PMCE offer together with such an extension, the client MUST Fail the WebSocket Connection ๐ โโ๏ธ
a server that declines ALL offered PMCEs MUST NOT include any element with PMCE names ๐ if the server responds with no PMCE element in the Sec-WebSocket-Extensions header, then both endpoints just go without per-message compression once the WebSocket Connection is established ๐ธ
and if a server gives an invalid response, like accepting a PMCE the client never offered, the client MUST Fail the WebSocket Connection ๐ค no vibes allowed
finally, if a server responds with a valid PMCE element in the Sec-WebSocket-Extensions header and the WebSocket Connection is established, both endpoints MUST use the algorithm described in Section 6 ๐ and the message payload transformation (compressing and decompressing) procedure of the PMCE, configured with the "agreed parameters" the server sent back, to exchange messages ๐ง
5.1. General Negotiation Flow
okay this section is the general flow of how the negotiation goes ๐บ๏ธ the detailed handling of parameters has to be spelled out in the document that specifies the PMCE, so this is just the big picture ๐ท
a client makes an offer that includes parameters saying:
- Hints about how the client is planning to compress data
- Requests about how the server compresses data
- Limitations concerning the client's compression functionality
the peer server then decides how it's going to behave based on those parameters ๐ค if the server can AND wants to go ahead with this PMCE enabled, it responds to the client with parameters saying:
- Requests about how the client compresses data
- How the server will compress data
based on what the server sent back, the client works out its own behavior and decides whether it can AND wants to go ahead with this PMCE enabled ๐ if it can't or doesn't want to, the client starts the closing handshake with close code 1010 ๐ฅ
5.2. Negotiation Examples
these are example values for the Sec-WebSocket-Extensions header offering PMCEs โจ permessage-foo and permessage-bar are just hypothetical extension names for PMCEs using the compression algorithms foo and bar ๐ฆ
- Offer the permessage-foo.
permessage-foo
- Offer the permessage-foo with a parameter x with a value of 10.
permessage-foo; x=10
The value may be quoted.
permessage-foo; x="10"
- Offer the permessage-foo as first choice and the permessage-bar as a fallback plan.
permessage-foo, permessage-bar
- Offer the permessage-foo with a parameter use_y, which enables a feature y as first choice, and the permessage-foo without the use_y parameter as a fallback plan.
permessage-foo; use_y, permessage-foo
6. Framing ๐
PMCEs only operate on data messages ๐
this document gives the RSV1 bit of the WebSocket header to PMCEs and calls it the "Per-Message Compressed" bit ๐ซถ on a WebSocket connection where a PMCE is in use, this bit tells you whether a message is compressed or not โจ
a message whose "Per-Message Compressed" bit is set on its first fragment is called a "compressed message" ๐๏ธ the frames of a compressed message carry compressed data in the payload data portion. when an endpoint gets a compressed message, it decompresses the concatenation of the compressed data from the message's frames using the decompression procedure of the PMCE in use ๐งฉ then it takes the bytes that match the application data portion of that decompressed data and uses those for the _A WebSocket Message Has Been Received_ event, instead of the received data as-is ๐
a message whose "Per-Message Compressed" bit is unset on its first fragment is called an "uncompressed message" ๐ the frames of an uncompressed message carry the uncompressed original data as-is in the payload data portion. when an endpoint gets an uncompressed message, it just uses the concatenation of the application data portions of the message's frames, as-is, for the _A WebSocket Message Has Been Received_ event ๐
6.1. Compression ๐
okay bestie, here's the compression recipe ๐ An endpoint MUST use the following algorithm to send a message in the form of a compressed message โจ
- Squish the payload of the original message using the compression procedure of the PMCE ๐๏ธ The original message can come straight from the application layer, or from the output of another WebSocket extension, depending on which extensions got negotiated.
- Now process the compressed data:
- If this PMCE is the last extension to process outgoing messages, build frame(s) using the compressed data instead of the original data for the message payload, set the
"Per-Message Compressed"bit of the first frame, and then send the frame(s) as described in Section 6.1 of [RFC6455] ๐ - Otherwise, pass the transformed message payload and modified header values, including the
"Per-Message Compressed"bit value set to1, to the next extension after the PMCE. If the extension expects frames for input, build a frame for the message and pass it ๐ฆ
- If this PMCE is the last extension to process outgoing messages, build frame(s) using the compressed data instead of the original data for the message payload, set the
An endpoint MUST use the following algorithm to send a message in the form of an uncompressed message ๐ซถ
- Process the original data like this:
- If this PMCE is the last extension to process outgoing messages, build frame(s) using the original data for the payload data portion as is, unset the
"Per-Message Compressed"bit of the first frame, and then send the frame(s) as described in Section 6.1 of [RFC6455] ๐ - Otherwise, pass the message payload and header values to the next extension after the PMCE as is. If the extension expects frames for input, build a frame for the message and pass it ๐
- If this PMCE is the last extension to process outgoing messages, build frame(s) using the original data for the payload data portion as is, unset the
An endpoint MUST NOT set the "Per-Message Compressed" bit of control frames and non-first fragments of a data message. An endpoint receiving such a frame MUST Fail the WebSocket Connection ๐ญ this one is serious, no cap ๐ซ
PMCEs do not change the opcode field. The opcode of the first frame of a compressed message indicates the opcode of the original message ๐ง
The payload data portion in frames generated by a PMCE is not subject to the constraints for the original data type. For example, the concatenation of the output data corresponding to the application data portion of frames of a compressed text message is not required to be valid UTF-8 ๐ At the receiver, though, the payload data portion after decompression is subject to the constraints for the original data type again ๐
6.2. Decompression ๐ธ
An endpoint MUST use the following algorithm to receive a message in the form of a compressed message. It's giving reverse-engineering energy, but in a cute way ๐
- Concatenate the payload data portion of the received frames of the compressed message. The received frames may be direct input from the underlying transport, or output of another WebSocket extension, depending on which extensions got negotiated.
- Decompress the concatenation by following the decompression procedure of the PMCE ๐
- Now process the decompressed message:
- If this is the last extension to process incoming messages, deliver the A WebSocket Message Has Been Received event to the application layer with the decompressed message payload and header values, including the
"Per-Message Compressed"bit unset to0โจ - Otherwise, pass the decompressed message payload and header values, including the
"Per-Message Compressed"bit unset to0, to the extension preceding the PMCE. If the extension expects frames for input, build a frame for the message and pass it ๐ฆ
- If this is the last extension to process incoming messages, deliver the A WebSocket Message Has Been Received event to the application layer with the decompressed message payload and header values, including the
An endpoint MUST use the following algorithm to receive a message in the form of an uncompressed message ๐
- Process the received message like this:
- If this PMCE is the last extension to process incoming messages, deliver the A WebSocket Message Has Been Received event to the application layer with the received message payload and header values as is ๐
- Otherwise, pass the message payload and header values to the extension preceding the PMCE as is. If the extension expects frames for input, build a frame for the message and pass it ๐
7. The "permessage-deflate" Extension ๐ฆ
okay this is the main event, the specific PMCE everyone's obsessed with ๐ญ This section defines a specific PMCE called "permessage-deflate". It compresses the payload of a message using the DEFLATE algorithm [RFC1951] and uses the byte boundary alignment method introduced in [RFC1979] ๐
This section uses the term "byte" with the same meaning as used in [RFC1951], i.e., 8 bits stored or transmitted as a unit (same as an octet) ๐ง
The registered extension name for this extension is "permessage-deflate" ๐ท๏ธ
Four extension parameters are defined for "permessage-deflate" to help endpoints manage per-connection resource usage ๐
server_no_context_takeoverclient_no_context_takeoverserver_max_window_bitsclient_max_window_bits
These parameters enable two methods (no_context_takeover and max_window_bits) of constraining memory usage, and each one can be applied independently to either direction of WebSocket traffic ๐ธ The extension parameters with the client_ prefix are used by the client to configure its compressor, and by the server to configure its decompressor. The extension parameters with the server_ prefix are used by the server to configure its compressor, and by the client to configure its decompressor. All four parameters are defined for both a client's extension negotiation offer and a server's extension negotiation response ๐
A server MUST decline an extension negotiation offer for this extension if any of the following conditions are met:
- The negotiation offer contains an extension parameter not defined for use in an offer.
- The negotiation offer contains an extension parameter with an invalid value.
- The negotiation offer contains multiple extension parameters with the same name.
- The server doesn't support the offered configuration.
A client MUST Fail the WebSocket Connection if the peer server accepted an extension negotiation offer for this extension with an extension negotiation response meeting any of the following conditions:
- The negotiation response contains an extension parameter not defined for use in a response.
- The negotiation response contains an extension parameter with an invalid value.
- The negotiation response contains multiple extension parameters with the same name.
- The client does not support the configuration that the response represents.
The term "LZ77 sliding window" [LZ77] used in this section means the buffer used by the DEFLATE algorithm to store recently processed input. The DEFLATE compression algorithm searches the buffer for a match with the following input ๐
The term "use context takeover" used in this section means that the same LZ77 sliding window used by the endpoint to build frames of the previous sent message is reused to build frames of the next message to be sent ๐
7.1. Extension Parameters ๐
7.1.1. Context Takeover Control โจ
7.1.1.1. The "server_no_context_takeover" Extension Parameter ๐ซถ
A client MAY include the server_no_context_takeover extension parameter in an extension negotiation offer. This extension parameter has no value. By including this extension parameter in an extension negotiation offer, a client prevents the peer server from using context takeover. If the peer server doesn't use context takeover, the client doesn't need to reserve memory to retain the LZ77 sliding window between messages ๐งบ
Absence of this extension parameter in an extension negotiation offer indicates that the client can decompress a message that the server built using context takeover ๐
A server accepts an extension negotiation offer that includes the server_no_context_takeover extension parameter by including the server_no_context_takeover extension parameter in the corresponding extension negotiation response to send back to the client. The server_no_context_takeover extension parameter in an extension negotiation response has no value ๐
It is RECOMMENDED that a server supports the server_no_context_takeover extension parameter in an extension negotiation offer ๐
A server MAY include the server_no_context_takeover extension parameter in an extension negotiation response even if the extension negotiation offer being accepted by the extension negotiation response didn't include the server_no_context_takeover extension parameter ๐ฆ
7.1.1.2. The "client_no_context_takeover" Extension Parameter ๐ธ
A client MAY include the client_no_context_takeover extension parameter in an extension negotiation offer. This extension parameter has no value. By including this extension parameter in an extension negotiation offer, a client informs the peer server of a hint: even if the server doesn't include the client_no_context_takeover extension parameter in the corresponding extension negotiation response to the offer, the client is not going to use context takeover ๐ค
A server MAY include the client_no_context_takeover extension parameter in an extension negotiation response. If the received extension negotiation offer includes the client_no_context_takeover extension parameter, the server may either ignore the parameter, or use the parameter to avoid taking over the LZ77 sliding window unnecessarily by including the client_no_context_takeover extension parameter in the corresponding extension negotiation response to the offer. The client_no_context_takeover extension parameter in an extension negotiation response has no value. By including the client_no_context_takeover extension parameter in an extension negotiation response, a server prevents the peer client from using context takeover. This reduces the amount of memory that the server has to reserve for the connection ๐ง
Absence of this extension parameter in an extension negotiation response indicates that the server can decompress messages built by the client using context takeover ๐
A client MUST support the client_no_context_takeover extension parameter in an extension negotiation response ๐
7.1.2. Limiting the LZ77 Sliding Window Size ๐ฆ
7.1.2.1. The "server_max_window_bits" Extension Parameter ๐
Okay bestie, clients are allowed (MAY) to include server_max_window_bits in an extension negotiation offer ๐. Its value is a decimal integer with no leading zeroes, from 8 to 15 inclusive, and it's the base-2 logarithm of the LZ77 sliding window size. It MUST conform to this ABNF:
server-max-window-bits = 1*DIGIT
So when a client sends this, it's basically saying "hey server, please keep your LZ77 window small when you compress stuff for me" ๐ง. If the server uses a small window, the client can shrink how much memory its own LZ77 window needs. Iconic for the client, ngl ๐ .
If the server doesn't support this parameter, it declines the offer ๐ . And if the parameter isn't in the offer at all, the client can receive messages compressed with an LZ77 sliding window of up to 32,768 bytes ๐ธ.
A server accepts an offer with this parameter by including server_max_window_bits in its response, with the same or a smaller value than the offer. The response version also has a decimal integer with no leading zeroes, from 8 to 15 inclusive, meaning the base-2 logarithm of the LZ77 sliding window size, and it MUST conform to this ABNF:
server-max-window-bits = 1*DIGIT
A server MAY include server_max_window_bits in its response even if the offer it's accepting didn't include the parameter ๐ฆ.
7.1.2.2. The "client_max_window_bits" Extension Parameter ๐
A client MAY include client_max_window_bits in an extension negotiation offer. This one is a bit different: it can have no value at all, or a decimal integer with no leading zeroes, from 8 to 15 inclusive, which is again the base-2 logarithm of the LZ77 sliding window size. If there IS a value, it MUST conform to this ABNF:
client-max-window-bits = 1*DIGIT
By putting this in an offer, the client tells the server that it supports the client_max_window_bits parameter in a response ๐ซถ. If the client also attaches a value, that's a hint too. It means that even if the server doesn't send back client_max_window_bits with a bigger value than the offer (or doesn't send it at all), the client still won't use an LZ77 sliding window bigger than the value it offered when compressing messages ๐ฅ.
If a received offer has client_max_window_bits, the server MAY include it in the matching response. If the offer had a value, the server can either ignore that value, or use it to avoid allocating a window that's way bigger than it needs to be. To do the second thing, the server includes client_max_window_bits in the response with a value equal to or smaller than the one it received. That response value is also a decimal integer with no leading zeroes, from 8 to 15 inclusive, the base-2 logarithm of the LZ77 sliding window size, and it MUST conform to this ABNF:
client-max-window-bits = 1*DIGIT
When a server includes this parameter in a response, it limits the LZ77 sliding window the client uses to compress messages ๐. That also means less memory the server has to reserve for its decompression context on the connection. Lowkey genius ๐ง .
If a received offer doesn't have client_max_window_bits, the matching response MUST NOT include client_max_window_bits โ. And if the response doesn't include it, the server can receive messages compressed with an LZ77 sliding window of up to 32,768 bytes โจ.
7.1.3. Examples ๐
The simplest Sec-WebSocket-Extensions header in a client's opening handshake, offering the "permessage-deflate" extension, looks like this:
Sec-WebSocket-Extensions: permessage-deflate
Since client_max_window_bits isn't in this offer, the server must not accept it with a response that includes client_max_window_bits ๐. The simplest Sec-WebSocket-Extensions header in a server's opening handshake, accepting "permessage-deflate", is the same:
Sec-WebSocket-Extensions: permessage-deflate
Next, here's an offer from a client asking the server to use an LZ77 sliding window of 1,024 bytes or less, and saying it supports the client_max_window_bits parameter in a response:
Sec-WebSocket-Extensions:
permessage-deflate;
client_max_window_bits; server_max_window_bits=10
The server might reject that one, because it doesn't support server_max_window_bits in an offer. That's totally fine if the client can't handle messages compressed with a bigger window. But if the client would be happy with a plain "permessage-deflate" as a fallback, it can offer both options like this ๐ฆ:
Sec-WebSocket-Extensions:
permessage-deflate;
client_max_window_bits; server_max_window_bits=10,
permessage-deflate;
client_max_window_bits
The server can accept "permessage-deflate" by picking any one it supports from the list. If it goes with the first option, its response could look like this:
Sec-WebSocket-Extensions:
permessage-deflate; server_max_window_bits=10
If it goes with the second option, the response could look like this:
Sec-WebSocket-Extensions: permessage-deflate
7.2. Message Payload Transformation ๐
7.2.1. Compression ๐
An endpoint uses the following algorithm to compress a message ๐ซง:
- Compress all the octets of the payload of the message using DEFLATE.
- If the resulting data does not end with an empty DEFLATE block with no compression (the
BTYPEbits are set to00), append an empty DEFLATE block with no compression to the tail end. - Remove 4 octets (that are
0x00 0x00 0xff 0xff) from the tail end. After this step, the last octet of the compressed data contains (possibly part of) the DEFLATE header bits with theBTYPEbits set to00.
When using DEFLATE in the first step above, a few things are allowed:
- An endpoint MAY use multiple DEFLATE blocks to compress one message.
- An endpoint MAY use DEFLATE blocks of any type.
- An endpoint MAY use both DEFLATE blocks with the
BFINALbit set to 0 and DEFLATE blocks with theBFINALbit set to 1. - When any DEFLATE block with the
BFINALbit set to 1 doesn't end at a byte boundary, an endpoint MUST add minimal padding bits of 0 to make it end at a byte boundary. The next DEFLATE block follows the padded data if any.
An endpoint fragments a compressed message by splitting the result of running this algorithm. Even when only part of the payload is available, a fragment can be built by compressing the available data and choosing the block type appropriately, so that the end of the resulting compressed data is aligned at a byte boundary. Note that for non-final fragments, the removal of 0x00 0x00 0xff 0xff MUST NOT be done ๐
โโ๏ธ.
An endpoint MUST NOT use an LZ77 sliding window longer than 32,768 bytes to compress messages to send ๐ซ.
If the "agreed parameters" contain the client_no_context_takeover extension parameter, the client MUST start compressing each new message with an empty LZ77 sliding window. Otherwise, the client MAY take over the LZ77 sliding window used to build the last compressed message. Note that even if the client included client_no_context_takeover in its offer, the client MAY take over the LZ77 sliding window used to build the last compressed message if the "agreed parameters" don't contain client_no_context_takeover. The client-to-server client_no_context_takeover extension parameter is just a hint for the server to build an extension negotiation response ๐.
If the "agreed parameters" contain the server_no_context_takeover extension parameter, the server MUST start compressing each new message with an empty LZ77 sliding window. Otherwise, the server MAY take over the LZ77 sliding window used to build the last compressed message ๐ง.
If the "agreed parameters" contain the client_max_window_bits extension parameter with a value of w, the client MUST NOT use an LZ77 sliding window longer than the w-th power of 2 bytes to compress messages to send. Note that even if the client included client_max_window_bits in its offer with a value smaller than the one in the "agreed parameters", the client MAY use an LZ77 sliding window of any size to compress messages to send, as long as that size conforms to the "agreed parameters". The client-to-server client_max_window_bits extension parameter is just a hint for the server to build an extension negotiation response ๐ซถ.
If the "agreed parameters" contain the server_max_window_bits extension parameter with a value of w, the server MUST NOT use an LZ77 sliding window longer than the w-th power of 2 bytes to compress messages to send ๐
.
7.2.2. Decompression โจ
okay bestie, here's how an endpoint decompresses a message ๐ซถ
- Append 4 octets of 0x00 0x00 0xff 0xff to the tail end of the payload of the message.
- Decompress the resulting data using DEFLATE.
so if the "agreed parameters" include server_no_context_takeover, the client MAY decompress each new message with an empty LZ77 sliding window ๐งน. otherwise the client MUST decompress each new message using the LZ77 sliding window that was used to process the last compressed message ๐.
same energy for the server ๐
. if the "agreed parameters" include client_no_context_takeover, the server MAY decompress each new message with an empty LZ77 sliding window. otherwise the server MUST decompress each new message using the LZ77 sliding window used to process the last compressed message.
โ ๏ธ bestie, read this part twice. even if the client putclient_no_context_takeoverin its offer, the server MUST decompress each new message using the LZ77 sliding window used to process the last compressed message if the "agreed parameters" don't containclient_no_context_takeover. the client-to-serverclient_no_context_takeoverparameter is just a hint for the server to build an extension negotiation response ๐.
window sizes next ๐ช. if the "agreed parameters" include server_max_window_bits with a value of w, the client MAY reduce the size of its LZ77 sliding window to decompress received messages down to the w-th power of 2 bytes. otherwise the client MUST use a 32,768-byte LZ77 sliding window to decompress received messages.
and the server side of that ๐ฆ: if the "agreed parameters" include client_max_window_bits with a value of w, the server MAY reduce the size of its LZ77 sliding window to decompress received messages down to the w-th power of 2 bytes. otherwise the server MUST use a 32,768-byte LZ77 sliding window to decompress received messages.
last bit, and it's a big one ๐. even if the client included a client_max_window_bits value in its offer that is smaller than the one in the "agreed parameters", the client MUST use an LZ77 sliding window of a size that conforms to the "agreed parameters" to compress messages it sends. the client-to-server client_max_window_bits parameter is, again, just a hint for the server to build an extension negotiation response ๐.
7.2.3. Examples
this section is the fun part ๐ง. it introduces examples of how the "permessage-deflate" extension transforms messages.
7.2.3.1. A Message Compressed Using One Compressed DEFLATE Block
say an endpoint sends the text message "Hello" ๐. if it uses one compressed DEFLATE block (compressed with fixed Huffman code and the "BFINAL" bit not set), the endpoint gets the compressed data for the message payload like this ๐.
the endpoint compresses "Hello" into one compressed DEFLATE block and flushes the resulting data into a byte array using an empty DEFLATE block with no compression:
0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 0x00 0x00 0xff 0xff
by stripping 0x00 0x00 0xff 0xff from the tail end, the endpoint gets the data for the message payload:
0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00
say the endpoint sends this compressed message without fragmentation ๐. it builds one frame by putting all the compressed data in the payload data portion of the frame:
0xc1 0x07 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00
the first 2 octets (0xc1 0x07) are the WebSocket frame header (FIN=1, RSV1=1, RSV2=0, RSV3=0, opcode=text, MASK=0, Payload length=7). here's what goes in each field ๐ :
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-------+-+-------------+
|F|R|R|R| opcode|M| Payload len |
|I|S|S|S| |A| |
|N|V|V|V| |S| |
| |1|2|3| |K| |
+-+-+-+-+-------+-+-------------+
|1|1|0|0| 1 |0| 7 |
+-+-+-+-+-------+-+-------------+
now say the endpoint sends the compressed message with fragmentation ๐งฉ. it splits the compressed data into fragments and builds a frame for each fragment. for example, if the fragments are 3 and 4 octets, the first frame is:
0x41 0x03 0xf2 0x48 0xcd
and the second frame is:
0x80 0x04 0xc9 0xc9 0x07 0x00
note that the RSV1 bit is set only on the first frame ๐.
7.2.3.2. Sharing LZ77 Sliding Window
okay this one is giving dรฉjร vu ๐. say a client has sent a message "Hello" as a compressed message and will send the same message "Hello" again as a compressed message:
0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00
that's the payload of the first message the client has sent. if the "agreed parameters" include the client_no_context_takeover extension parameter, the client compresses the payload of the next message into the same bytes (as long as it uses the same "BTYPE" value and "BFINAL" value). so the payload of the second message will be:
0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00
if the "agreed parameters" did not contain the client_no_context_takeover extension parameter, the client can compress the payload of the next message into fewer bytes by referencing the history in the LZ77 sliding window ๐ช. so the payload of the second message will be:
0xf2 0x00 0x11 0x00 0x00
so 2 bytes are saved in total ๐ค.
note that even if some uncompressed messages (with the RSV1 bit unset) are inserted between the two "Hello" messages, they don't affect the LZ77 sliding window ๐ฆ.
7.2.3.3. Using a DEFLATE Block with No Compression
a DEFLATE block with no compression may be used.
0xc1 0x0b 0x00 0x05 0x00 0xfa 0xff 0x48 0x65 0x6c 0x6c 0x6f 0x00
this is a frame that makes up a text message "Hello", built using a DEFLATE block with no compression ๐ธ. the first 2 octets (0xc1 0x0b) are the WebSocket frame header (FIN=1, RSV1=1, RSV2=0, RSV3=0, opcode=text, MASK=0, Payload length=7). note that the RSV1 bit is set for this message (only on the first fragment if the message is fragmented) because the RSV1 bit is set when DEFLATE is applied to the message, including when only DEFLATE blocks with no compression are used ๐ญ. the 3rd to 13th octets are the payload data, which holds "Hello" compressed using a DEFLATE block with no compression.
7.2.3.4. Using a DEFLATE Block with "BFINAL" Set to 1
on platforms where the flush method using an empty DEFLATE block with no compression isn't available, implementors can choose to flush data using DEFLATE blocks with "BFINAL" set to 1 ๐ ๏ธ.
0xf3 0x48 0xcd 0xc9 0xc9 0x07 0x00 0x00
this is the payload of a message containing "Hello" compressed using a DEFLATE block with "BFINAL" set to 1. the first 7 octets make up a DEFLATE block with "BFINAL" set to 1 and "BTYPE" set to 01, containing "Hello". the last 1 octet (0x00) holds the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, plus 5 padding bits of 0. that octet is needed so the payload can be decompressed the same way as messages flushed using DEFLATE blocks with "BFINAL" unset โจ.
7.2.3.5. Two DEFLATE Blocks in One Message
two or more DEFLATE blocks can show up in one message ๐ง.
0xf2 0x48 0x05 0x00 0x00 0x00 0xff 0xff 0xca 0xc9 0xc9 0x07 0x00
the first 3 octets (0xf2 0x48 0x05) and the least significant two bits of the 4th octet (0x00) make up one DEFLATE block with "BFINAL" set to 0 and "BTYPE" set to 01, containing "He". the rest of the 4th octet holds the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, plus the 3 padding bits of 0. together with the following 4 octets (0x00 0x00 0xff 0xff), those header bits make up an empty DEFLATE block with no compression. a DEFLATE block containing "llo" follows the empty one ๐ซง.
7.2.3.6. Generating an Empty Fragment
say an endpoint is sending data of unknown size ๐ฆ. it might hit the end-of-data signal from the data source while its buffer for uncompressed data is empty. in that case, the endpoint just needs to send the last fragment with the FIN bit set to 1 and the payload set to the DEFLATE block(s), which hold 0 bytes of data. if the compression library in use doesn't output anything when its buffer is empty, an empty uncompressed DEFLATE block can be built and used for this purpose, like so ๐ญ:
0x00
the single octet 0x00 holds the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, plus 5 padding bits of 0.
7.3. Implementation Notes
okay, implementation tips ๐ง . on most common software development platforms, the DEFLATE compression library gives you a way to align compressed data to byte boundaries using an empty DEFLATE block with no compression. for example, zlib [zlib] does this when Z_SYNC_FLUSH is passed to the deflate function ๐.
some platforms only provide ways to output and process compressed data with a zlib header and an Adler-32 checksum. on those platforms, developers need to write stub code to remove and complement the zlib and Adler-32 checksum by themselves ๐ ๏ธ.
to get a useful compression ratio, an LZ77 sliding window size of 1,024 or more is RECOMMENDED ๐.
if a side disallows context takeover, its endpoint can easily figure out whether or not a certain message will be shorter if compressed ๐ฏ. otherwise, it's not easy to know whether future messages will benefit from having a certain message compressed. implementors may use some heuristics to figure this out ๐ฎ.
8. Security Considerations ๐จ
๐จ bestie, this one is serious. there is a known exploit when history-based compression is combined with a secure transport [CRIME]. implementors should pay attention to this when integrating this extension with other extensions or protocols. don't skip it ๐.
9. IANA Considerations ๐
9.1. Registration of the "permessage-deflate" WebSocket Extension Name
IANA has registered the following WebSocket extension name in the "WebSocket Extension Name Registry" defined in [RFC6455] ๐:
Extension Identifier
permessage-deflate
Extension Common Name
WebSocket Per-Message Deflate
Extension Definition
This document.
Known Incompatible Extensions
None
the "permessage-deflate" extension name is used in the Sec-WebSocket-Extensions header in the WebSocket opening handshake to negotiate use of the "permessage-deflate" extension ๐ค.
9.2. Registration of the "Per-Message Compressed" WebSocket Framing Header Bit
IANA has registered the following WebSocket framing header bit in the "WebSocket Framing Header Bits Registry" defined in [RFC6455] ๐ฆ:
Value
RSV1
Description
The "Per-Message Compressed" bit, which indicates whether or not
the message is compressed. RSV1 is set for compressed messages
and unset for uncompressed messages.
Reference
Section 6 of this document.
the "Per-Message Compressed" framing header bit is used on the first fragment of data messages to indicate whether the payload of the message is compressed by the PMCE or not ๐.
10. References ๐
10.1. Normative References ๐
okay, these are the ones you have to read to understand the spec ๐ซถ
[CRIME] Rizzo, J. and T. Duong, "The CRIME attack", EKOparty Security Conference, September 2012.
[LZ77] Ziv, J. and A. Lempel, "A Universal Algorithm for Sequential Data Compression", IEEE Transactions on Information Theory, Vol. 23, No. 3, pp. 337-343, DOI 10.1109/TIT.1977.1055714, May 1977, <https://www.cs.duke.edu/courses/spring03/cps296.5/papers/ziv_lempel_1977_universal_algorithm.pdf>.
[RFC1951] Deutsch, P., "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, DOI 10.17487/RFC1951, May 1996, <http://www.rfc-editor.org/info/rfc1951>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <http://www.rfc-editor.org/info/rfc2119>.
[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/RFC5234, January 2008, <http://www.rfc-editor.org/info/rfc5234>.
[RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol", RFC 6455, DOI 10.17487/RFC6455, December 2011, <http://www.rfc-editor.org/info/rfc6455>.
10.2. Informative References ๐ธ
these ones are extra reading, not required, but they're giving context and I love a good backstory ๐ง
[RFC1979] Woods, J., "PPP Deflate Protocol", RFC 1979, DOI 10.17487/RFC1979, August 1996, <http://www.rfc-editor.org/info/rfc1979>.
[zlib] Gailly, J. and M. Adler, "zlib", <http://www.zlib.net/>.
Acknowledgements ๐
okay, big thank-you to Patrick McManus, who wrote up the first version of a DEFLATE-based compression extension for the WebSocket Protocol. I used that as a reference while writing this spec, so he's lowkey the OG ๐
Thanks also to everyone below, who joined the discussions on the HyBi WG and shared ideas and/or gave really detailed reviews ๐ซถ (the list is probably incomplete, so if your name isn't here, I'm sorry bestie ๐ญ): Adam Rice, Alexander Philippou, Alexey Melnikov, Arman Djusupov, Bjoern Hoehrmann, Brian McKelvey, Dario Crivelli, Greg Wilkins, Inaki Baz Castillo, Jamie Lokier, Joakim Erdfelt, John A. Tamplin, Julian Reschke, Kenichi Ishibashi, Mark Nottingham, Peter Thorson, Roberto Peon, Salvatore Loreto, Simone Bordet, Tobias Oberstein, and Yutaka Hirano. Just so you know, the people listed didn't necessarily agree with where the work ended up ๐
Author's Address ๐ฆ
Takeshi Yoshino Google, Inc.
Email: tyoshino@google.com
๐ฆ where this page comes from
- ietf: rfc7692 2015-12
- ๐ girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms