RFC 9628

RTP Payload Format for VP9 Video

📅 published March 2025, from ietf

RFC 9628: RTP Payload Format for VP9 Video 🎀 (Part 1 of 6)

okay bestie, let's get into it 💖 This is RFC 9628, and it's all about how to send VP9 video over RTP, which is the thing that makes your video calls not freeze every five seconds. Here's the title block, so you know who's behind it.

  • Number and title: RFC 9628, RTP Payload Format for VP9 Video
  • Authors: J. Uberti (OpenAI); S. Holmer, M. Flodman, and D. Hong (Google); J. Lennox (8x8 / Jitsi)
  • Category: Standards Track
  • ISSN: 2070-1721
  • Date: March 2025
  • Obsoletes / Updates: no Obsoletes or Updates line appears in this title block

Abstract ✨

okay so the abstract is giving "this payload format is for everyone" 🌈 It describes an RTP payload format for the VP9 video codec, and it's built to work for everything from low-bitrate peer-to-peer calls to high-bitrate video conferences. It also has support for temporal and spatial scalability, which is how you get multiple quality levels out of one stream. iconic, honestly 💅

Status of This Memo 🦋

ok so this one is an Internet Standards Track document. It's a product of the Internet Engineering Task Force (IETF), it represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication. If you want more on Internet Standards, check Section 2 of RFC 7841.

For the current status, any errata, or to give feedback, go to https://www.rfc-editor.org/info/rfc9628.

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, since they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Revised BSD License.

Table of Contents 🧚

1.  Introduction
2.  Conventions
3.  Media Format Description
4.  Payload Format
  4.1.  RTP Header Usage
  4.2.  VP9 Payload Descriptor
    4.2.1.  Scalability Structure (SS)
  4.3.  Frame Fragmentation
  4.4.  Scalable Encoding Considerations
  4.5.  Example of a VP9 RTP Stream
    4.5.1.  Reference Picture Use for Scalable Structure
5.  Feedback Messages and Header Extensions
  5.1.  Reference Picture Selection Indication (RPSI)
  5.2.  Full Intra Request (FIR)
  5.3.  Layer Refresh Request (LRR)
6.  Payload Format Parameters
  6.1.  SDP Parameters
    6.1.1.  Mapping of Media Subtype Parameters to SDP
    6.1.2.  Offer/Answer Considerations
7.  Media Type Definition
8.  Security Considerations
9.  Congestion Control
10. IANA Considerations
11. References
  11.1.  Normative References
  11.2.  Informative References
Acknowledgments
Authors' Addresses

1. Introduction ✨

okay so this doc describes an RTP [RFC3550] payload specification for sending video streams encoded with the VP9 video codec [VP9-BITSTREAM]. The format works in both peer-to-peer and video conferencing apps, so it's very flexible, bestie 🫶

VP9 was developed by Google and it's the successor to VP8 [RFC6386]. Beyond the compression improvements and other general upgrades over VP8, VP9 is also designed to allow spatially scalable video encoding. It's giving "one stream, many sizes" 🌸

2. Conventions 💅

Here's the key-word rule. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. So a lowercase version is just a regular word, but the capitals are the real deal 🔥

3. Media Format Description 🌈

The VP9 codec can keep up to eight reference frames, and any new frame can reference up to three of them.

VP9 also lets a frame use a frame of a different resolution as a reference. To be specific, a frame can use any references whose width and height fall between 1/16th of the current frame's size and twice the current frame's size, inclusive. That's how internal resolution changes can happen without needing keyframes. Such a glow-up 🦋

Put together, these features let an encoder build different kinds of coarse-grained scalability, including temporal, spatial, and quality scalability modes, plus combinations of them, all without any explicit scalable coding tools. Literally so efficient 😭

Here's the vocab. Temporal layers define different frame rates of video. Spatial and quality layers define different, and possibly dependent, representations of one input frame. Spatial layers let a frame be encoded at different resolutions. Quality layers let a frame be encoded at the same resolution but at different qualities, which means different amounts of coding error. VP9 treats quality layers as spatial layers with no resolution changes, so from here on, the term "spatial layer" stands for both spatial and quality layers. 💜

This payload format spec defines how those temporal and spatial scalability layers can be described and communicated. ✨

Temporal and spatial scalability layers are tagged with non-negative integer IDs. The lowest layer of either type has an ID of zero, and it's sometimes called the "base" temporal or spatial layer. 🍓

Layers are designed, and they MUST be encoded, so that if any layer and all higher layers are removed from the bitstream along either the spatial or temporal dimension, the remaining bitstream is still correctly decodable. That's the main rule, bestie, so don't mess it up 🙈

For terminology, this doc uses "frame" to mean a single encoded VP9 frame for a particular resolution and/or quality. It uses "picture" to mean all the representations (frames) at one instant in time. So a picture is made of one or more frames, each encoding a different spatial layer. 🎀

Within a picture, a frame with spatial-layer ID equal to S, where S > 0, can depend on a frame of the same picture with a lower spatial-layer ID. This "inter-layer" dependency can give you extra coding gain, compared with using only "inter-picture" dependency, where a frame depends on a previously coded frame in time. To keep things simple, this format assumes that, within a picture and when inter-layer dependency is used, a spatial-layer S frame can depend only on the immediately previous spatial-layer S-1 frame, when S > 0. Also, if inter-picture dependency is used, a spatial-layer S frame is assumed to depend only on a previously coded spatial-layer S frame. 🧠

With those simplifications for inter-layer and inter-picture dependencies, a flag, the D bit described below, indicates whether a spatial-layer SID frame depends on the spatial-layer SID-1 frame. Given the D bit, a receiver only needs to also know the inter-picture dependency structure for a given spatial-layer frame to figure out whether it can be decoded. There are two ways to describe that inter-picture dependency structure: "flexible mode" and "non-flexible mode". An encoder can only switch between the two on the first packet of a keyframe with a temporal-layer ID equal to zero. 🦋

In flexible mode, each packet can contain up to three reference indices, which identify all frames referenced by the frame sent in the current packet for inter-picture prediction. This, along with the D bit, lets a receiver tell whether a frame is decodable and helps it understand the temporal-layer structure. Since it's signaled in each packet, you can have very flexible temporal-layer hierarchies and scalability structures that change dynamically. Main character energy 👑

In non-flexible mode, frames are encoded using a fixed, recurring pattern of dependencies. The set of pictures that repeat in this pattern is known as a "Picture Group" (or "PG"). In this mode, the inter-picture dependencies (the reference indices) of the PG MUST be pre-specified as part of the Scalability Structure (SS) data. Each packet has an index that points to one of the described pictures in the PG, and from that, you can figure out which pictures the current packet's picture references for inter-picture prediction. 🌸

Quick aside, because this trips people up 🙈

   |  Note: A "Picture Group" or "PG", as used in this document, is
   |  not the same thing as the term "Group of Pictures" as it is
   |  commonly used in video coding, i.e., to mean an independently
   |  decodable run of pictures beginning with a keyframe.

The SS data can also be used to specify the resolution of each spatial layer present in the VP9 stream, for both flexible and non-flexible modes. ✨

4. Payload Format 🎀

This section describes how the encoded VP9 bitstream gets wrapped up in RTP. To handle network losses, using RTP/AVPF [RFC4585] is RECOMMENDED. Also, all integer fields in this spec are encoded as unsigned integers in network octet order. 🫶

That's Part 1 done, bestie. Part 2 picks up with Section 4.1, so stay tuned 🌈💖

4.1. RTP Header Usage 🎀

Okay so this is the general RTP payload format for VP9, and the diagram is giving main character energy 💅 Here it is:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |V=2|P|X|  CC   |M|     PT      |       sequence number         |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                           timestamp                           |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |           synchronization source (SSRC) identifier            |
     +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
     |            contributing source (CSRC) identifiers             |
     |                             ....                              |
     +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
     |            VP9 payload descriptor (integer #octets)           |
     :                                                               :
     |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                               :                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
     |                                                               |
     +                                                               |
     :                          VP9 payload                          :
     |                                                               |
     |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                               :    OPTIONAL RTP padding       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 1: General RTP Payload Format for VP9

Okay so the payload descriptor is covered in Section 4.2 🧚, and the VP9 payload itself is described in [VP9-BITSTREAM]. Bestie, the thing to remember is that OPTIONAL RTP padding MUST NOT be included unless the P bit is set 🙈

Marker bit (M): This bit MUST be set to one for the final packet of the highest spatial-layer frame (the final packet of the picture); otherwise, it is zero. Unless spatial scalability is in use for this picture, this bit will have the same value as the E bit described in Section 4.2. Note this bit MUST be set to one for the target spatial-layer frame if a stream is being rewritten to remove higher spatial layers. ✨

Payload Type (PT): In line with the policy in Section 3 of [RFC3551], applications using the VP9 RTP payload profile MUST assign a dynamic payload type number to be used in each RTP session and provide a mechanism to indicate the mapping. See Section 6.1 for the mechanism to be used with the Session Description Protocol (SDP) [RFC8866]. 🦋

Timestamp: The RTP timestamp [RFC3550] indicates the time when the input frame was sampled, at a clock rate of 90 kHz. If the input picture is encoded with multiple frames, all of the frames of the picture MUST have the same timestamp. 🌸

Now here's the tea 🍓 If a frame has the VP9 show_frame field set to zero (i.e., it is meant only to populate a reference buffer without being output), its timestamp MAY alternatively be set to be the same as the subsequent frame with show_frame equal to one. This is honestly so convenient for playing out pre-encoded content packaged with VP9 "superframes", which typically bundle show_frame==0 frames with a subsequent show_frame==1 frame 📦 Every picture containing a frame with show_frame==1, however, MUST have a unique timestamp modulo the 2^32 wrap of the field. No copying allowed, periodt 💀

The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, SSRC, and CSRC identifiers) are used as specified in Section 5.1 of [RFC3550]. Lowkey the whole header is just standard RTP doing its thing 💜

4.2. VP9 Payload Descriptor 🎀

Okay bestie, so in flexible mode (the F bit set to one 💅), the first octets after the RTP header are the VP9 payload descriptor, and it has this structure 👑:

         0 1 2 3 4 5 6 7
        +-+-+-+-+-+-+-+-+
        |I|P|L|F|B|E|V|Z| (REQUIRED)
        +-+-+-+-+-+-+-+-+
   I:   |M| PICTURE ID  | (REQUIRED)
        +-+-+-+-+-+-+-+-+
   M:   | EXTENDED PID  | (RECOMMENDED)
        +-+-+-+-+-+-+-+-+
   L:   | TID |U| SID |D| (Conditionally RECOMMENDED)
        +-+-+-+-+-+-+-+-+                             -\
   P,F: | P_DIFF      |N| (Conditionally REQUIRED)    - up to 3 times
        +-+-+-+-+-+-+-+-+                             -/
   V:   | SS            |
        | ..            |
        +-+-+-+-+-+-+-+-+

         Figure 2: Flexible Mode Format for VP9 Payload Descriptor

Now non-flexible mode (the F bit set to zero) is the same vibe, but the first octets after the RTP header still make up the VP9 payload descriptor, just with a slightly different shape ✨:

         0 1 2 3 4 5 6 7
        +-+-+-+-+-+-+-+-+
        |I|P|L|F|B|E|V|Z| (REQUIRED)
        +-+-+-+-+-+-+-+-+
   I:   |M| PICTURE ID  | (RECOMMENDED)
        +-+-+-+-+-+-+-+-+
   M:   | EXTENDED PID  | (RECOMMENDED)
        +-+-+-+-+-+-+-+-+
   L:   | TID |U| SID |D| (Conditionally RECOMMENDED)
        +-+-+-+-+-+-+-+-+
        |   TL0PICIDX   | (Conditionally REQUIRED)
        +-+-+-+-+-+-+-+-+
   V:   | SS            |
        | ..            |
        +-+-+-+-+-+-+-+-+

       Figure 3: Non-Flexible Mode Format for VP9 Payload Descriptor

Unless I say otherwise 🫶, these field descriptions work for both the Figure 2 and Figure 3 formats.

I: Picture ID (PID) present. When this is set to one, the OPTIONAL PID MUST be there after the mandatory first octet, specified as below. Otherwise, PID MUST NOT be there at all. If the V bit was set in the stream's most recent start of a keyframe (i.e., the SS field was present) and the F bit is set to zero (i.e., non-flexible scalability mode is in use), then this bit MUST be set on every packet 😭.

P: Inter-picture predicted frame. When this is set to zero, the frame does not use inter-picture prediction. In that case, up-switching to a current spatial layer's frame is possible from a directly lower spatial-layer frame. P SHOULD also be set to zero when encoding a layer synchronization frame in response to a Layer Refresh Request (LRR) [RFC9627] message (see Section 5.3) 🦋. When P is set to zero, the Temporal-layer ID (TID) field (described below) MUST also be set to zero (if present). Also, the P bit does not forbid intra-picture, inter-layer prediction from earlier frames of the same picture, if there are any 🌸.

L: Layer indices present. When this is set to one, the one or two octets after the mandatory first octet and the PID (if present) are described by "Layer indices" below. If the F bit (described below) is set to one (flexible mode), then only one octet is present for the layer indices. Otherwise, if the F bit is set to zero (non-flexible mode), then two octets are present for the layer indices 🍓.

F: Flexible mode. When set to one, this means flexible mode. If the P bit is also set to one, then the octets after the mandatory first octet, the PID, and the layer indices (if present) are described by "reference indices" below. This bit MUST only be set to one if the I bit is also set to one. If the I bit is set to zero, then this bit MUST also be set to zero and ignored by receivers. (Flexible mode's reference indices are offsets from the Picture ID field, so they would have no meaning if I weren't set 🙈.) The value of the F bit MUST only change on the first packet of a key picture. A "key picture" is a picture whose base spatial-layer frame is a keyframe, which means it completely resets the encoder state. This packet will have its P bit equal to zero, its SID or L bit (described below) equal to zero, and its B bit (described below) equal to one 👑.

B: Start of Frame. This bit MUST be set to one if the first payload octet of the RTP packet is the beginning of a new VP9 frame. Otherwise, it MUST NOT be one. Note that this frame might not be the first frame of a picture 🧚.

E: End of Frame. This bit MUST be set to one for the final RTP packet of a VP9 frame. Otherwise, it is zero. This lets a decoder finish decoding the frame, when it otherwise might have to wait for the next packet to explicitly know the frame is complete ✨. Note that if spatial scalability is in use, more frames from the same picture may follow. See the description of the B bit above.

V: Scalability Structure (SS) data present. When set to one, the OPTIONAL SS data MUST be present in the payload descriptor. Otherwise, the SS data MUST NOT be present 💜.

Z: Not a reference frame for upper spatial layers. If set to one, this indicates that frames with higher spatial layers SID+1 and greater, in the current and following pictures, do not depend on the current spatial-layer SID frame. This lets a decoder that's targeting a higher spatial layer know it can safely discard this packet's frame without processing it, without having to wait for the D bit in the higher-layer frame (see below) 🌈.

The mandatory first octet is followed by the extension data fields that are enabled 🎀:

M: The most significant bit of the first octet is an extension flag. The field MUST be present if the I bit is equal to one. If M is set, the PID field MUST contain 15 bits. Otherwise, it MUST contain 7 bits. See PID below 💖.

Picture ID (PID): This is the Picture ID, represented in 7 or 15 bits depending on the M bit. It's a running index of the pictures, where the sender increments the value by one for each picture it sends. (Note, however, that because a middlebox can discard pictures where the SS permits it, Picture IDs as received by a receiver might not be contiguous 🦋.) This field MUST be present if the I bit is equal to one. If M is set to zero, 7 bits carry the PID. Else, if M is set to one, 15 bits carry the PID in network byte order. The sender may choose between a 7- or 15-bit index. The PID SHOULD start on a random number and MUST wrap after reaching the maximum ID (0x7f or 0x7fff, depending on the index size chosen). The receiver MUST NOT assume that the number of bits in the PID stays the same through the session. If this field transitions from 7 bits to 15 bits, the value is zero-extended (i.e., the value after 0x6e is 0x006f). If the field transitions from 15 bits to 7 bits, it is truncated (i.e., the value after 0x1bbe is 0x3f) 🌸.

In non-flexible mode (when the F bit is set to zero), this PID is used as an index to the PG specified in the SS data below. In this mode, the PID of the keyframe corresponds to the first specified frame in the PG. Then subsequent PIDs are mapped to subsequently specified frames in the PG (modulo N_G, specified in the SS data below), respectively 🍓.

All frames of the same picture MUST have the same PID value 💅.

Frames (and their corresponding pictures) with the VP9 show_frame field equal to zero MUST have distinct PID values from subsequent pictures with show_frame equal to one. So a picture (as defined in this specification) is different from a VP9 superframe 🧚.

All frames of the same picture MUST have the same value for show_frame 🔥.

Layer indices: This field is optional but RECOMMENDED whenever you're encoding with layers 🌈. For both flexible and non-flexible modes, one octet is used to specify a layer frame's Temporal-layer ID (TID) and Spatial-layer ID (SID), as shown in both Figures 2 and 3. Additionally, a bit (U) is used to indicate that the current frame is a "switching up point" frame. Another bit (D) is used to indicate whether inter-layer prediction is used for the current frame ⭐.

In non-flexible mode (when the F bit is set to zero), another octet is used to represent the Temporal Layer 0 Picture Index (8 bits) (TL0PICIDX), as depicted in Figure 3. The TL0PICIDX is present so that all minimally required frames (the base temporal-layer frames) can be tracked 🫶.

The TID and SID fields indicate the temporal and spatial layers, and they can help middleboxes and endpoints quickly identify which layer a packet belongs to 🦋.

  • TID: The temporal-layer ID of the current frame. In non-flexible mode, if a PID is mapped to a picture in a specified PG, then the value of the TID MUST match the corresponding TID value of the mapped picture in the PG 💖.
  • U: Switching up point. When this bit is set to one, and the current picture has a temporal-layer ID equal to value T, then subsequent pictures with temporal-layer ID values higher than T will not depend on any picture before the current picture (in decode order) with a temporal-layer ID value greater than T 👑.
  • SID: The spatial-layer ID of the current frame. Note that frames with spatial-layer SID > 0 may depend on a decoded spatial-layer SID-1 frame within the same picture. Different frames of the same picture MUST have distinct spatial-layer IDs, and frames' spatial layers MUST appear in increasing order within the frame 🔥.
  • D: Inter-layer dependency is used. D MUST be set to one if and only if the current spatial-layer SID frame depends on the spatial-layer SID-1 frame of the same picture. Otherwise, it MUST be set to zero. For the base-layer frame (with SID equal to zero), the D bit MUST be set to zero 💀.
  • TL0PICIDX: Temporal Layer 0 Picture Index (8 bits). TL0PICIDX is only present in non-flexible mode (F = 0). It's a running index for the temporal base-layer pictures, i.e., the pictures with a TID set to zero. If the TID is larger than zero, TL0PICIDX indicates which temporal base-layer picture the current picture depends on. TL0PICIDX MUST be incremented by one when the TID is equal to zero. The index SHOULD start on a random number and MUST restart at zero after reaching the maximum number 255 🌸.

Reference indices: When P and F are both set to one, indicating a non-keyframe in flexible mode, then at least one reference index MUST be specified as below. Additional reference indices (a total of up to three reference indices are allowed) may be specified using the N bit below. When either P or F is set to zero, then no reference index is specified 🧚.

  • P_DIFF: The reference index (in 7 bits) specified as the relative PID from the current picture. For example, when P_DIFF=3 on a packet containing the picture with PID 112, the picture refers back to the picture with PID 109. This calculation is done modulo the size of the PID field, i.e., either 7 or 15 bits. A P_DIFF value of zero is invalid 🙈.
  • N: 1 if there is additional P_DIFF following the current P_DIFF ✨.

4.2.1. Scalability Structure (SS) ✨

okay so the SS data is the part that tells you the resolution of each frame in a picture 🖼️ and how the pictures in a PG depend on each other 🧚. if the V bit in the VP9 payload descriptor is set, the SS data shows up in the spot marked in Figures 2 and 3 💅

        +-+-+-+-+-+-+-+-+
   V:   | N_S |Y|G|-|-|-|
        +-+-+-+-+-+-+-+-+              -\
   Y:   |     WIDTH     | (OPTIONAL)    .
        +               +               .
        |               | (OPTIONAL)    .
        +-+-+-+-+-+-+-+-+               . - N_S + 1 times
        |     HEIGHT    | (OPTIONAL)    .
        +               +               .
        |               | (OPTIONAL)    .
        +-+-+-+-+-+-+-+-+              -/
   G:   |      N_G      | (OPTIONAL)
        +-+-+-+-+-+-+-+-+                            -\
   N_G: | TID |U| R |-|-| (OPTIONAL)                 .
        +-+-+-+-+-+-+-+-+              -\            . - N_G times
        |    P_DIFF     | (OPTIONAL)    . - R times  .
        +-+-+-+-+-+-+-+-+              -/            -/

Figure 4: VP9 Scalability Structure

here's what each field means 🌸

  • N_S: Number of Spatial Layers Minus 1. N_S + 1 tells you how many spatial layers are in the VP9 stream 🎀
  • Y: each spatial layer's frame resolution is present. when this is set to one, the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be present for each layer frame. otherwise, the resolution MUST NOT be present 🫶
  • G: the PG description present flag 🦋
  • -: a bit reserved for future use. it MUST be set to zero and MUST be ignored by the receiver 🙈
  • N_G: N_G tells you the number of pictures in a PG 💜 if N_G is greater than zero, the SS data lets the inter-picture dependency structure of the VP9 stream be pre-declared, instead of being indicated on the fly with every packet. if N_G is greater than zero, then for the N_G pictures in the PG, each picture's Temporal-layer ID (TID), switch up point (U), and reference indices (P_DIFFs) are specified.

the first picture specified in the PG MUST have a TID set to zero 👑

G set to zero or N_G set to zero means either there is only one temporal layer (for non-flexible mode) or no fixed inter-picture dependency information is present (for flexible mode) going forward in the bitstream 🌈

note that for a given picture, all frames follow the same inter-picture dependency structure. however, the frame rate of each spatial layer can be different from each other. this can be described with the use of the D bit described above. the specified dependency structure in the SS data MUST be for the highest frame rate layer ⭐

  • R: the number of P_DIFF fields that are present 🍓

in a scalable stream sent with a fixed pattern, the SS data SHOULD be included in the first packet of every key frame. this is a packet with the P bit equal to zero, SID or L bit equal to zero, and B bit equal to one. the SS data MUST only be changed on the picture that corresponds to the first picture specified in the previous SS data's PG (if the previous SS data's N_G was greater than zero) 💖

4.3. Frame Fragmentation

VP9 frames get chopped into packets in RTP sequence number order 🧩 they start with a packet that has the B bit set and end with a packet that has the E bit set. there is no mechanism for finer-grained access to parts of a VP9 frame, so that's just how it is 🤷‍♀️

4.4. Scalable Encoding Considerations

okay so beyond reference frames, VP9 has a few more kinds of inter-frame dependencies, mostly involving probability tables for the entropy and tree encoders 🧠 in VP9 syntax, the syntax element "error_resilient_mode" resets this extra inter-frame data, so a frame's syntax can be decoded on its own 🔥

because scalable streams have their own requirements, a VP9 encoder making one needs to make sure a frame does not depend on a previous frame (from the same picture or an earlier one) that can legitimately be removed from the stream 😭 so a frame that follows a frame that might be removed (in full decode order) MUST be encoded with "error_resilient_mode" set to true 🚨

for spatially scalable streams, this means "error_resilient_mode" needs to be turned on for the base spatial layer. it can be turned off for higher spatial layers, assuming they're sent with inter-layer dependency (i.e., with the D bit set) 🌸 for streams that are only temporally scalable without spatial scalability, "error_resilient_mode" can additionally be turned off for any picture that immediately follows a temporal-layer 0 frame 🦋

4.5. Example of a VP9 RTP Stream

Time for a real example 🎬

4.5.1. Reference Picture Use for Scalable Structure

as we covered in Section 3, the VP9 codec can keep up to eight reference frames, and any new frame can reference or update up to three of them 🧚 this section shows one way to build a scalable structure (with three spatial layers and three temporal layers) out of these reference frames ✨

           +==========+=========+============+=========+
           | Temporal | Spatial | References | Updates |
           +==========+=========+============+=========+
           |    0     |    0    |     0      |    0    |
           +----------+---------+------------+---------+
           |    0     |    1    |    0,1     |    1    |
           +----------+---------+------------+---------+
           |    0     |    2    |    1,2     |    2    |
           +----------+---------+------------+---------+
           |    2     |    0    |     0      |    6    |
           +----------+---------+------------+---------+
           |    2     |    1    |    1,6     |    7    |
           +----------+---------+------------+---------+
           |    2     |    2    |    2,7     |    -    |
           +----------+---------+------------+---------+
           |    1     |    0    |     0      |    3    |
           +----------+---------+------------+---------+
           |    1     |    1    |    1,3     |    4    |
           +----------+---------+------------+---------+
           |    1     |    2    |    2,4     |    5    |
           +----------+---------+------------+---------+
           |    2     |    0    |     3      |    6    |
           +----------+---------+------------+---------+
           |    2     |    1    |    4,6     |    7    |
           +----------+---------+------------+---------+
           |    2     |    2    |    5,7     |    -    |
           +----------+---------+------------+---------+

Table 1: Example Scalability Structure

this structure is built so the U bit can always be set 💅

5. Feedback Messages and Header Extensions ✨

5.1. Reference Picture Selection Indication (RPSI)

the RPSI is a payload-specific feedback message that lives inside the RTCP-based feedback format 📨 a receiver generates it, and it can be used in two ways 🫶 either it signals a preferred reference picture when the decoder has detected a loss (ideally one the decoder knows is perfect), or it works as positive feedback to acknowledge that certain reference pictures decoded correctly 🎀 the positive feedback method is useful for VP9 used for point-to-point (unicast) communication 💜

RPSI works best for VP9 when it's paired with a special update pattern for the codec's two special reference frames, the golden frame and the altref frame 🌈 they get updated in an alternating leapfrog fashion 🐸 when a receiver has received and correctly decoded a golden or altref frame, and that frame had a Picture ID in the payload descriptor, the receiver can just send an RPSI message back to the sender to acknowledge it 🙌 the message body (i.e., the "native RPSI bit string" in [RFC4585]) is simply the (7- or 15-bit) Picture ID of the received frame 🍓

Note: because all frames of the same picture must have the same inter-picture reference structure, there is no need for a message to specify which frame is being selected.

5.2. Full Intra Request (FIR)

the Full Intra Request (FIR) [RFC5104] RTCP feedback message lets a receiver ask for a full state refresh of an encoded stream 🔄

when a FIR request comes in, a VP9 sender MUST send a picture with a keyframe for its spatial-layer 0 layer frame, and then send frames without inter-picture prediction (P=0) for any higher-layer frames 🧸

5.3. Layer Refresh Request (LRR)

the Layer Refresh Request (LRR) [RFC9627] lets a receiver ask for a single layer of a spatially or temporally encoded stream to be refreshed, without necessarily messing with the stream's other layers 🦋

               +---------------+---------------+
               |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
               +---------------+---------+-----+
               |   RES   | TID | RES     | SID |
               +---------------+---------+-----+

Figure 5: LRR Index Format

Figure 5 shows the format of an LRR's layer index fields for VP9 streams 🌸 both "RES" fields MUST be set to zero on transmission and ignored on reception 🙈 see Section 4.2 for details on the TID and SID fields 💖

identifying a layer refresh frame can be derived from the reference IDs of each frame by backtracking the dependency chain until you hit a point where only decodable frames are being referenced 🧩 so for both the flexible and the non-flexible mode, it's recommended that when switching up points are being encoded in response to an LRR, those packets contain layer indices and the reference field or fields. that way the decoder or selective forwarding middleboxes [RFC7667] can make this derivation 🔍

Example:

LRR {1,0}, {2,1} is sent by a Multipoint Control Unit (MCU) when it is currently relaying {1,0} to a receiver that wants to upgrade to {2,1} 👑 in response, the encoder should encode the next frames in layers {1,1} and {2,1} by only referring to frames in {1,0} or {0,0} ✨

in the non-flexible mode, periodic upgrade frames can be defined by the layer structure of the SS, so periodic upgrade frames can be automatically identified by the Picture ID 🥰

6. Payload Format Parameters ✨

okay bestie, this payload format has three optional parameters: max-fr, max-fs, and profile-id. 🎀

The max-fr and max-fs parameters are how a receiver says what it can handle 🫶. If the implementation is willing to receive media, both parameters MUST be provided. These parameters MUST NOT be used for any other purpose. A media sender SHOULD NOT send media with a frame rate or frame size exceeding the max-fr and max-fs values signaled. (There may be scenarios, such as pre-encoded media or selective forwarding middleboxes [RFC7667], where a media sender does not have media available that fits within a receiver's max-fs and max-fr values. In those scenarios, a sender MAY exceed the signaled values. 💀)

  • max-fr: The value of max-fr is an integer indicating the maximum frame rate, in frames per second, that the decoder is capable of decoding.
  • max-fs: The value of max-fs is an integer indicating the maximum frame size, in macroblocks, that the decoder is capable of decoding.

The decoder can handle this frame size as long as the width and height of the frame in macroblocks are each less than int(sqrt(max-fs * 8)). For instance, a max-fs of 1200 (capable of supporting 640x480 resolution) will support widths and heights up to 1552 pixels (97 macroblocks). 📐

  • profile-id: The value of profile-id is an integer indicating the default coding profile. That means the subset of coding tools that may have been used to generate the stream, or that the receiver supports. Table 2 lists all of the profiles defined in Section 7.2 of [VP9-BITSTREAM] and the corresponding integer values to be used. 🧚

If no profile-id is present, Profile 0 MUST be inferred. (The profile-id parameter was added pretty late in the development of this specification, so some existing implementations may not send it. 🙈)

Informative note: See Table 3 for capabilities of coding profiles defined in Section 7.2 of [VP9-BITSTREAM]. 🌸

A receiver MUST ignore any parameter unspecified in this specification. 👑

                      +=========+============+
                      | Profile | profile-id |
                      +=========+============+
                      |    0    |     0      |
                      +---------+------------+
                      |    1    |     1      |
                      +---------+------------+
                      |    2    |     2      |
                      +---------+------------+
                      |    3    |     3      |
                      +---------+------------+

                              Table 2:
                       Correspondence between
                         profile-id to VP9
                          Profile Integer
+=========+===========+=================+==========================+
| Profile | Bit Depth | SRGB Colorspace |    Chroma Subsampling    |
+=========+===========+=================+==========================+
|    0    |     8     |        No       |        YUV 4:2:0         |
+---------+-----------+-----------------+--------------------------+
|    1    |     8     |       Yes       | YUV 4:2:2,4:4:0 or 4:4:4 |
+---------+-----------+-----------------+--------------------------+
|    2    |  10 or 12 |        No       |        YUV 4:2:0         |
+---------+-----------+-----------------+--------------------------+
|    3    |  10 or 12 |       Yes       | YUV 4:2:2,4:4:0 or 4:4:4 |
+---------+-----------+-----------------+--------------------------+

                  Table 3: Profile Capabilities

  |  Note: SRGB (often sRGB) = Standard Red-Green-Blue

6.1. SDP Parameters 🎶

6.1.1. Mapping of Media Subtype Parameters to SDP

The media type video/vp9 string gets mapped to fields in the Session Description Protocol (SDP) [RFC8866] like this:

  • The media name in the m= line of SDP MUST be video.
  • The encoding name in the a=rtpmap line of SDP MUST be VP9 (the media subtype).
  • The clock rate in the a=rtpmap line MUST be 90000.
  • The parameters max-fr and max-fs MUST be included in the a=fmtp line of SDP if the receiver wishes to declare its receiver capabilities. These parameters are expressed as a media subtype string in the form of a semicolon-separated list of parameter=value pairs.
  • The OPTIONAL parameter profile-id, when present, SHOULD be included in the a=fmtp line of SDP. This parameter is expressed as a media subtype string in the form of a parameter=value pair. When the parameter is not present, a value of 0 MUST be inferred for profile-id. 🫧

6.1.1.1. Example

Here's what a media representation looks like in SDP, and it's honestly so clean 😭:

m=video 49170 RTP/AVPF 98
a=rtpmap:98 VP9/90000
a=fmtp:98 max-fr=30;max-fs=3600;profile-id=0

6.1.2. Offer/Answer Considerations

When VP9 is offered over RTP using SDP in an Offer/Answer model [RFC3264] for negotiation for unicast usage, the following limitations and rules apply. Pay attention, this part is important 👀:

  • The parameter identifying a media format configuration for VP9 is profile-id. This media format configuration parameter MUST be used symmetrically. That means the answerer MUST either maintain this configuration parameter or remove the media format (payload type) completely if it is not supported. 🔁
  • The max-fr and max-fs parameters are used declaratively to describe receiver capabilities, even in the Offer/Answer model. The values in an answer describe the answerer's capabilities, so their values are set independently of the values in the offer. 🙌
  • To simplify the handling and matching of these configurations, the same RTP payload type number used in the offer SHOULD also be used in the answer and in a subsequent offer, as specified in [RFC3264]. An answer or subsequent offer MUST NOT contain the payload type number used in the offer unless the profile-id value is exactly the same as in the original offer. However, max-fr and max-fs parameters MAY be changed in subsequent offers and answers, with the same payload type number, if an endpoint wishes to change its declared receiver capabilities. 🦋

7. Media Type Definition 📝

This registration uses the template defined in [RFC6838] and following [RFC4855].

Type name:  video

Subtype name:  VP9

Required parameters:  N/A

Optional parameters:  There are three optional parameters: max-fr,
   max-fs, and profile-id.  See Section 6 for their definition.

Encoding considerations:  This media type is framed in RTP and
   contains binary data; see Section 4.8 of [RFC6838].

Security considerations:  See Section 8 of RFC 9628.

Interoperability considerations:  None

Published specification:  VP9 bitstream format [VP9-BITSTREAM] and
   RFC 9628.

Applications that use this media type:  For example, video over IP,
   video conferencing.

Fragment identifier considerations:  N/A

Additional information:  None

Person & email address to contact for further information:  Jonathan
   Lennox <jonathan.lennox@8x8.com>

Intended usage:  COMMON

Restrictions on usage:  This media type depends on RTP framing;
   hence, it is only defined for transfer via RTP [RFC3550].

Author:  Jonathan Lennox <jonathan.lennox@8x8.com>

Change controller:  IETF AVTCore Working Group delegated from the
   IETF.

8. Security Considerations 🔐

RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550], and in any applicable RTP profile such as RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/SAVPF [RFC5124]. However, as "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] discusses, it is not an RTP payload format's responsibility to discuss or mandate what solutions are used to meet the basic security goals like confidentiality, integrity, and source authenticity for RTP in general. That responsibility lies with anyone using RTP in an application. They can find guidance on available security mechanisms in "Options for Securing RTP Sessions" [RFC7201]. Applications SHOULD use one or more appropriate strong security mechanisms. 🛡️

Implementations of this RTP payload format need to take appropriate security considerations into account. It is extremely important for the decoder to be robust against malicious or malformed payloads, and to make sure they do not cause the decoder to overrun its allocated memory or otherwise misbehave. An overrun in allocated memory could lead to arbitrary code execution by an attacker. This is not a drill 🚨. The same applies to the encoder, even though problems in encoders are (typically) rarer.

This RTP payload format and its media decoder do not exhibit any significant non-uniformity in the receiver-side computational complexity for packet processing. So they are unlikely to pose a denial-of-service threat due to the receipt of pathological data. The RTP payload format also doesn't contain any active content, which is a huge win 🌈.

9. Congestion Control 🚦

Congestion control for RTP SHALL be used in accordance with [RFC3550], and with any applicable RTP profile, e.g., [RFC3551]. In a real-time encoding scenario, the congestion control mechanism can adapt the transmission rate by instructing the encoder to encode at a certain target rate. Media-aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2 to identify non-reference frames and discard them in order to reduce network congestion. 🧹

Note that discarding non-reference frames cannot be done if the stream is encrypted, because the non-reference marker is encrypted. 🙅‍♀️

10. IANA Considerations 🗂️

IANA has registered the media type "video/vp9" as specified in Section 7. The media type has also been added to the "RTP Payload Format Media Types" registry of the "Real-Time Transport Protocol (RTP) Parameters" registry group (https://www.iana.org/assignments/rtp-parameters) as follows:

Media Type:  video
Subtype:  VP9
Clock Rate (Hz):  90000
Reference:  RFC 9628

11. References 📚

11.1. Normative References

okay so these are the receipts 🧾✨ the normative ones are the must-reads, aka the specs this doc basically can't function without 💅

11.2. Informative References

these ones are the bonus reading 🦋 handy for the background and the "why does this exist" energy, but you don't strictly need them to get the main story 🌸

Acknowledgments 🫶

big thank-yous to Alex Eleftheriadis, Yuki Ito, Won Kap Jang, Sergio Garcia Murillo, Roi Sasson, Timothy Terriberry, Emircan Uysaler, and Thomas Volkert 🥹💖 they all commented on the development of this document and gave super helpful feedback, so they're basically the hype squad behind it 👑

Authors' Addresses 💌

these are the people who wrote it, and where to find them 🧚

Justin Uberti
OpenAI
1455 3rd St
San Francisco, CA 94158
United States of America
Email: justin@uberti.name


Stefan Holmer
Google, Inc.
Kungsbron 2
SE-111 22 Stockholm
Sweden
Email: holmer@google.com


Magnus Flodman
Google, Inc.
Kungsbron 2
SE-111 22 Stockholm
Sweden
Email: mflodman@google.com


Danny Hong
Google, Inc.
315 Hudson St.
New York, NY 10013
United States of America
Email: dannyhong@google.com


Jonathan Lennox
8x8, Inc. / Jitsi
Jersey City, NJ 07302
United States of America
Email: jonathan.lennox@8x8.com
📦 where this page comes from