RFC 7741
RTP Payload Format for VP8 Video
RFC 7741: RTP Payload Format for VP8 Video 🎀
Okay bestie, let's dig into this one! 💖 RFC 7741 is an IETF Standards Track document, which means it's an official Internet standard and not just a "hey, here's an idea" memo. It was published in March 2016 (ISSN 2070-1721) and it's all about how to send VP8 video over RTP. 🦋
The main authors are P. Westin, H. Lundin, J. Uberti, and F. Galligan from Google, plus M. Glover from Twitter. 👑 This title block doesn't list anything the document obsoletes or updates, so there's no older RFC it's replacing here. 🤷♀️
Abstract ✨
This memo describes an RTP payload format for the VP8 video codec. 🎥 It's super versatile, too, because it works for everything from low-bitrate peer-to-peer calls all the way up to high-bitrate video conferences. 💅
Status of This Memo 🌸
This is an Internet Standards Track document. 🔥
It's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check out Section 2 of RFC 5741. 📚
For the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7741. 🧚
Copyright Notice 💜
Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were in effect on the date of publication. Please actually read these, bestie, because they explain your rights and restrictions. 🙈 Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License describes. 🫶
Table of Contents 🗺️
1. Introduction ....................................................3
2. Conventions, Definitions, and Abbreviations .....................3
3. Media Format Description ........................................4
4. Payload Format ..................................................5
4.1. RTP Header Usage ...........................................6
4.2. VP8 Payload Descriptor .....................................7
4.3. VP8 Payload Header ........................................11
4.4. Aggregated and Fragmented Payloads ........................12
4.5. Example Algorithms ........................................13
4.5.1. Frame Reconstruction Algorithm .....................13
4.5.2. Partition Reconstruction Algorithm .................13
4.6. Examples of VP8 RTP Stream ................................14
4.6.1. Key Frame in a Single RTP Packet ...................14
4.6.2. Non-discardable VP8 Interframe in a Single
RTP Packet; No PictureID ...........................14
4.6.3. VP8 Partitions in Separate RTP Packets .............15
4.6.4. VP8 Frame Fragmented across RTP Packets ............16
4.6.5. VP8 Frame with Long PictureID ......................18
5. Using VP8 with RPSI and SLI Feedback ...........................18
5.1. RPSI ......................................................18
5.2. SLI .......................................................19
5.3. Example ...................................................19
6. Payload Format Parameters ......................................21
6.1. Media Type Definition .....................................21
6.2. SDP Parameters ............................................23
6.2.1. Mapping of Media Subtype Parameters to SDP .........23
6.2.2. Offer/Answer Considerations ........................23
7. Security Considerations ........................................24
8. Congestion Control .............................................24
9. IANA Considerations ............................................24
10. References ....................................................25
10.1. Normative References .....................................25
10.2. Informative References ...................................26
Authors' Addresses ................................................281. Introduction ✨
This memo describes an RTP payload specification for sending video streams encoded with the VP8 video codec [RFC6386]. 🎬 The format works for both peer-to-peer and video-conferencing apps, so it's giving flexible. 💅
Okay, so VP8 splits each frame into square sub-blocks of pixels called "macroblocks" (see Section 2 of [RFC6386]). It predicts those sub-blocks from blocks it has already built, then adjusts those predictions (and makes up the unpredicted blocks) using a discrete cosine transform (DCT from here on). 🌈 There's one special case where VP8 uses a "Walsh-Hadamard" transform (WHT from here on) instead of a DCT. ✨ An encoded VP8 frame gets split into two or more partitions, as described in [RFC6386]. The first partition (prediction or mode) holds the prediction mode parameters and motion vectors for every macroblock. 🧚 The other partitions all hold the quantized DCT/WHT coefficients for the residuals. Depending on the encoder settings, a frame can have 1, 2, 4, or 8 DCT/WHT partitions. 🍓
So what does this payload format actually unlock? Here's the tea: ☕
- Partition boundaries get taken into account, which helps with loss robustness and makes packet-loss concealment at the decoder way easier. 🛡️
- Temporal scalability. 🦋
- Advanced use of reference frames, so error recovery is way more efficient. 🔁
- Frames that don't affect the decoding of any other frame get marked, so a server or media-aware network element can throw them out if it needs to. 🗑️💀
2. Conventions, Definitions, and Abbreviations 📖
Here's the important part: the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in [RFC2119]. 👑 Those capitalised words are the real rules, so don't skim past them, bestie! 🫶
This document uses the definitions from [RFC6386]. These are the terms you'll see a lot: 🌸
Key frames: Frames that get decoded without referring to any other frame in a sequence (also called intraframes and I-frames). 🔑
Interframes: Frames that get encoded with reference to prior frames, specifically all prior frames up to and including the most recent key frame (also called prediction frames and P-frames). 🎞️
Golden and altref frames: Alternate prediction frames. Blocks in an interframe may be predicted using blocks in the immediately previous frame, as well as the most recent golden frame or altref frame. ✨ Every key frame is automatically golden and altref, and any interframe may optionally replace the most recent golden or altref frame. 💛
Macroblock: A square array of pixels whose Y (luminance) dimensions are 16x16 pixels and whose U and V (chrominance) dimensions are 8x8 pixels. 🧩
Two definitions come from [RFC4585] and are used here too: 💜
RPSI: Reference picture selection indication. It's a feedback message that tells the encoder the decoder has correctly decoded a certain frame. 🎯
SLI: Slice loss indication. It's a feedback message that lets a decoder tell an encoder it has spotted the loss or corruption of one or several macroblocks. 😭
3. Media Format Description 🎀
VP8 uses three different reference frames for interframe prediction: the previous frame, the golden frame, and the altref frame. 🖼️ Blocks in an interframe may be predicted from the immediately previous frame, plus the most recent golden frame or altref frame. Every key frame is automatically golden and altref, and any interframe may optionally replace the most recent golden or altref frame. 💅 Golden and altref frames can also help frames survive when packets get dropped. The payload specification in this memo has elements that enable advanced use of the reference frames, for example to improve loss robustness. 🛡️
One specific use case for those three reference frame types is temporal scalability. ⏳ If you set up the reference hierarchy the right way, you can encode up to five temporal layers. (How to set up that hierarchy for temporal scalability is not within the scope of this memo, so it's not my problem to solve here. 🙈) Temporal scalability is supported by the optional TL0PICIDX and TID/Y/KEYIDX fields, which are described in Section 4.2. For a general overview of temporal scalability in video coding, see [Sch07]. 📚
VP8 also does data partitioning on the encoded data. An encoded VP8 frame can be split into two or more partitions, as described in "VP8 Data Format and Decoding Guide" [RFC6386]. The first partition (prediction or mode) holds the prediction mode parameters and motion vectors for all macroblocks. The other partitions hold the transform coefficients for the residuals. 🧠 The first partition can be decoded without the remaining residual partitions, and the later partitions can still be useful even if part of the frame gets lost. 🌟
Because of that, this document RECOMMENDS that the sender packetize the frame so each data partition goes in its own separate packet (or packets). 💌 That can help with error concealment on the decoder side. The payload format in Section 4 has fields that let you identify the partitions even if the first partition isn't available. 🔍 The sender can also choose to bundle the data partitions into a single data stream and split it into several packets without caring about partition boundaries. 🎁 The receiver can then use the length information in the first partition to find the partitions while decoding. 🧩
Section 4 describes the format specification. Section 5 describes a method for acknowledging receipt of reference frames using RTCP techniques. 📡
The payload partitioning and the acknowledgement method are the reasons behind three of the fields in the payload format: the PID, 1st partition size, and PictureID fields. 🥰 The ability to encode a temporally scalable stream is the reason for the TL0PICIDX and TID fields. 🦋
4. Payload Format 💖
This section explains how the encoded VP8 bitstream gets encapsulated in RTP. 📦 To handle network losses, using RTP/AVPF [RFC4585] is RECOMMENDED. 🫶 Also, every integer field in the specifications is encoded as an unsigned integer in network octet order. 🔢✨
4.1. RTP Header Usage ✨
okay bestie, here's the general RTP payload format for VP8 💖 It's giving a clean little layout, so let's look at it together 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| VP8 payload descriptor (integer #octets) |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : VP8 payload header (3 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| VP8 pyld hdr : |
+-+-+-+-+-+-+-+-+ |
: Octets 4..N of VP8 payload :
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+So the VP8 payload descriptor and VP8 payload header get their own deep dives in Sections 4.2 and 4.3 🎀 Till then, the important bit is that OPTIONAL RTP padding MUST NOT be included unless the P bit is set 🙈 Also, this figure only shows the first packet in a frame. Later packets won't have the VP8 payload header and will just carry on with more octets of the frame payload 🧚
Figure 1
- Marker bit (M): MUST be set for the very last packet of each encoded frame, same as the normal M bit usage in video formats 🎬 This is what lets a decoder finish decoding the picture. Otherwise it might have to sit and wait for the next packet just to find out the frame is done 😭
- Payload type (PT): The assignment of an RTP payload type for this packet format is outside the scope of this document, so it won't be specified here 💅
- Timestamp: The RTP timestamp tells you when the frame was sampled ⏰ The clock granularity is 90 kHz, so a delta of 1 means 1/90,000 of a second ✨
The rest of the RTP Fixed Header Fields (V, P, X, CC, sequence number, SSRC, and CSRC identifiers) work exactly as described in Section 5.1 of [RFC3550] 🌸 No cap, that's the whole vibe 💜
4.2. VP8 Payload Descriptor
okay bestie, this is where the VP8 payload descriptor lives 🎀 it sits right after the RTP header, and the first octet is the MVP. the single-octet PictureID version is on the left (M bit set to 0), and the dual-octet version is on the right (M bit set to 1) ✨
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
|X|R|N|S|R| PID | (REQUIRED) |X|R|N|S|R| PID | (REQUIRED)
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
X: |I|L|T|K| RSV | (OPTIONAL) X: |I|L|T|K| RSV | (OPTIONAL)
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
I: |M| PictureID | (OPTIONAL) I: |M| PictureID | (OPTIONAL)
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
L: | TL0PICIDX | (OPTIONAL) | PictureID |
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
T/K: |TID|Y| KEYIDX | (OPTIONAL) L: | TL0PICIDX | (OPTIONAL)
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
T/K: |TID|Y| KEYIDX | (OPTIONAL)
+-+-+-+-+-+-+-+-+
Figure 2X is the extended control bits flag 🦋 When it's set to 1, the extension octet MUST be provided right after that mandatory first octet. If it's 0, all the optional fields MUST be omitted. heads up, this X bit is NOT the same as the X bit in the RTP header, so don't mix them up 🙈
R is a bit reserved for future use 🌸 It MUST be set to 0, and the receiver MUST ignore it.
N is the non-reference frame flag. When it's 1, the frame can be thrown away without messing up any other frame, past or future. If the reference status of the frame is unknown, N SHOULD be set to 0 so you don't accidentally toss frames that are needed for reference 💅
Informative note: this document doesn't explain how to figure out whether an encoded frame is non-reference. That info is preferably provided by the encoder implementation.
S marks the start of a VP8 partition 👑 S SHOULD be set to 1 when the first payload octet of the RTP packet is the beginning of a new VP8 partition, and it MUST NOT be 1 otherwise. S MUST be set to 1 for the first packet of each encoded frame, no exceptions bestie 🔥
PID is the partition index, and it tells you which VP8 partition the first payload octet of the packet belongs to ✨ The first VP8 partition (the one with modes and motion vectors) MUST be labeled with PID = 0. PID SHOULD go up by 1 for each subsequent partition, but it MAY stay at 0 for all packets. PID can't be larger than 7. If more than one packet in an encoded frame has the same PID, then S MUST NOT be set on any packet except the first packet with that PID 🧚
When the X bit is 1 in the first octet, the Extended Control Bits field octet MUST be provided as the second octet. If X is 0, that Extended Control Bits octet MUST NOT be present, and no extensions (I, L, T, or K) are permitted. it's all or nothing, iykyk 💀
I means PictureID is present 🌈 When I is 1, the PictureID MUST be present after the extension bit field and specified as described below. Otherwise, PictureID MUST NOT be present.
L means TL0PICIDX is present ⭐ When L is 1, TL0PICIDX MUST be present and specified as below, and the T bit MUST be set to 1. Otherwise, TL0PICIDX MUST NOT be present.
T means TID is present 🦋 When T is 1, the TID/Y/KEYIDX octet MUST be present, and the TID|Y part of that octet MUST be specified as described below. If K is 1 but T is 0, the TID/Y/KEYIDX octet MUST still be present, but the TID field MUST be ignored. If neither T nor K is 1, the TID/Y/KEYIDX octet MUST NOT be present.
K means KEYIDX is present 🔑 When K is 1, the TID/Y/KEYIDX octet MUST be present, and the KEYIDX part of that octet MUST be specified as described below. If T is 1 but K is 0, the TID/Y/KEYIDX octet MUST be present, but the KEYIDX field MUST be ignored. If neither T nor K is 1, the TID/Y/KEYIDX octet MUST NOT be present.
RSV is more reserved bits for future use 🌸 They MUST be set to 0, and the receiver MUST ignore them.
After the extension bit field come the extension data fields that are enabled, in order. let's go through them 💖
The PictureID extension shows up when I is 1 ✨ If I is 1, the PictureID extension field MUST be present, and it MUST NOT be present otherwise. It has two parts:
M is the most significant bit of the first octet, and it's an extension flag 🎀 If M is set, the rest of the PictureID field MUST contain 15 bits. Otherwise, it MUST contain 7 bits. (this M bit is also NOT the same as the M bit in the RTP header, so no mixing it up, bestie 🙈)
PictureID is 7 or 15 bits (shown left and right, respectively, in Figure 2), not counting the M bit 🧚 It's a running index of the frames. It MAY start at a random value, it MUST increase by 1 for each subsequent frame, and it MUST wrap back to 0 after hitting the maximum ID (all bits set). The bits go from most significant to least significant, starting with the first bit after the M bit. The sender picks a 7-bit or 15-bit index and sets the M bit to match. The receiver MUST NOT assume that the number of bits in PictureID stays the same through the whole session, so don't get attached to one size 💅 Once the sender has sent a 7-bit PictureID with all bits set to 1, it may either wrap the PictureID to 0 or extend to 15 bits and keep incrementing.
The TL0PICIDX extension shows up when L is 1 🦋 If L is 1, the TL0PICIDX extension field MUST be present, and it MUST NOT be present otherwise. It has just one part:
TL0PICIDX is an 8-bit temporal level zero index 👑 It's a running index for the temporal base layer frames, meaning the frames with TID set to 0. If TID is larger than 0, TL0PICIDX tells you which base-layer frame the current image depends on. TL0PICIDX MUST be incremented when TID is 0. It MAY start at a random value, and it MUST wrap to 0 after reaching the maximum number, 255. Using TL0PICIDX depends on TID being present, so it's RECOMMENDED that TID be used whenever TL0PICIDX is. basically they're a power couple 💖
The TID/Y/KEYIDX extension shows up when either T or K is 1 🌈 If either one is 1, the TID/Y/KEYIDX extension field MUST be present. It MUST NOT be present if both T and K are zero. It has three parts:
TID is a 2-bit temporal-layer index ✨ The TID field MUST be ignored by the receiver when T is 0. It tells you which temporal layer the packet represents.
The lowest layer, the base layer, MUST have TID set to 0. Higher layers SHOULD increment TID according to their position in the layer hierarchy 🧚
Y is the 1-bit layer sync bit ⭐ Y SHOULD be set to 1 if the current frame depends only on the base-layer (TID = 0) frame whose TL0PICIDX matches the current frame's TL0PICIDX. Y MUST be set to 0 if the current frame depends on any frame other than that base-layer (TID = 0) frame with the same TL0PICIDX. Y MUST also be set to 0 if any frame after the current frame depends on a non-base-layer frame that's older than the base-layer frame with the same TL0PICIDX as the current frame. If Y is set while T is 0, the current frame MUST only depend on a past base-layer (TID=0) key frame, as signaled by a change in the KEYIDX field. On top of that, this frame MUST NOT depend on any of the three codec buffers (as defined by [RFC6386]) that have been updated since the last time the KEYIDX field changed 🔥
Informative note: this document doesn't explain how to determine the dependency status of a frame. That info is preferably provided by the encoder implementation. If the status is unknown, Y can safely be set to 0.
KEYIDX is a 5-bit temporal key frame index 🔑 The KEYIDX field MUST be ignored by the receiver when K is 0. It's a running index for key frames. KEYIDX MAY start at a random value, and it MUST wrap to 0 after reaching the maximum number, 31. When it's in use, KEYIDX SHOULD be present for both key frames and interframes. The sender MUST increment KEYIDX for key frames that carry parameter updates critical to interpreting later frames, and it SHOULD leave KEYIDX unchanged for key frames that don't carry those critical updates. If KEYIDX is present, a receiver SHOULD NOT decode an interframe unless it has received and decoded a key frame with the same KEYIDX after the last KEYIDX wraparound 💖
Informative note: this document doesn't explain how to tell whether a key frame updates critical parameters. That info is preferably provided by the encoder implementation. A sender that doesn't have this info may either omit the KEYIDX field (set K to 0) or increment KEYIDX on every key frame. The upside of the latter is that any key-frame loss gets detected by the receiver, which can then ask for a retransmission or request a new key frame 🫶
Informative note: implementations that splice VP8 streams have to make sure the rules for incrementing TL0PICIDX and KEYIDX are followed across the splice. That will likely mean rewriting the values of TL0PICIDX and KEYIDX after the splice. this is a lot of careful bookkeeping, bestie 😭✨
4.3. VP8 Payload Header 🎀
okay so the very start of an encoded VP8 frame is called an "uncompressed data chunk" in Section 9.1 of [RFC6386], and it's giving double duty 💖 because it also works as the payload header in this RTP format. The codec bitstream has two versions of that chunk: a 3-octet one for interframes and a 10-octet one for key frames. The first 3 octets are the same in both. For a key frame, the remaining 7 octets just count as part of the rest of the payload in this RTP format. Also, the header only shows up in packets where the S bit is equal to one and the PID is equal to zero in the payload descriptor. Later packets for the same frame don't carry it 🫶
the length of the first partition can always be found from the first partition-size parameter in the VP8 payload header ✨ The VP8 bitstream format [RFC6386] says that if multiple DCT/WHT partitions get made, the spot where each partition starts is found at the end of the first (prediction or mode) partition. In this RTP payload spec, those location offsets are treated as part of the first partition 🧚
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|Size0|H| VER |P|
+-+-+-+-+-+-+-+-+
| Size1 |
+-+-+-+-+-+-+-+-+
| Size2 |
+-+-+-+-+-+-+-+-+
| Octets 4..N of|
| VP8 payload |
: :
+-+-+-+-+-+-+-+-+
| OPTIONAL RTP |
| padding |
: :
+-+-+-+-+-+-+-+-+
Figure 3A packetizer needs access to the P bit. The other fields are defined in [RFC6386], Section 9.1, and their meanings don't influence the packetization process. None of these fields get modified by the packetization process 💅
P: Inverse key frame flag. When set to 0, the current frame is a key frame. When set to 1, the current frame is an interframe. Defined in [RFC6386]
4.4. Aggregated and Fragmented Payloads 🧩
an encoded VP8 frame can be split into two or more partitions, as described in Section 1. It is OPTIONAL for a packetizer implementing this RTP specification to pay attention to the partition boundaries within an encoded frame. If packetization of a frame happens without considering the partition boundaries, the PID field MAY be set to 0 for all packets, and the S bit MUST NOT be set to 1 for any packet other than the first.
If the preferred usage suggested in Section 3 is followed, with each packet carrying data from exactly one partition, the S bit and PID fields described in Section 4.2 SHOULD be used to indicate what the packet contains. The PID field should indicate which partition the first octet of the payload belongs to, and the S bit indicates that the packet starts on a new partition 🎯
If the packetizer does not pay attention to the partition boundaries, one packet can contain a fragment of a partition, a complete partition, or an aggregate of fragments and partitions. There is no explicit signaling of partition boundaries in the payload, so the partition lengths at the end of the first partition have to be used to identify the boundaries. Partitions MUST be aggregated in decoding order. Two fragments from different partitions MAY be aggregated into the same packet along with one or more complete partitions 🌸
In all cases, the payload of a packet MUST contain data from only one video frame. Consequently, the set of packets carrying the data from a particular frame will contain exactly one VP8 Payload Header (see Section 4.3), carried in the first packet of the frame. The last, or only, packet carrying data for the frame MUST have the M bit set in the RTP header 🔥
4.5. Example Algorithms 🧠
4.5.1. Frame Reconstruction Algorithm
Example of frame reconstruction algorithm 👑
- Collect all packets with a given RTP timestamp.
- Go through packets in order, sorted by sequence numbers. If packets are missing, send NACK as defined in [RFC4585], or decode with missing partitions, see Section 4.5.2 below.
- A frame is complete if the frame has no missing sequence numbers, the first packet in the frame contains
S=1withpartId=0, and the last packet in the frame has the marker bit set.
4.5.2. Partition Reconstruction Algorithm
Example of partition reconstruction algorithm. The algorithm only applies for the RECOMMENDED use case, with partitions in separate packets 🦋
- Scan for the start of a new partition;
S=1. - Continue scan to detect end of partition. Hence, a new
S=1(previous packet was the end of the partition) is found, or the marker bit is set. If a loss is detected before the end of the partition, abandon all packets in this partition and continue the scan, repeating from step 1. - Store the packets in the complete partition, and continue the scan, repeating from step 1 until end of frame is reached.
- Send all complete partitions to the decoder. If no complete partition is found, discard the whole frame.
4.6. Examples of VP8 RTP Stream 🎬
A few examples of how the VP8 RTP payload can be used are included below 🌈
4.6.1. Key Frame in a Single RTP Packet
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 1 |
+-+-+-+-+-+-+-+-+
|1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
|Size0|1| VER |0| P = 0
+-+-+-+-+-+-+-+-+
| Size1 |
+-+-+-+-+-+-+-+-+
| Size2 |
+-+-+-+-+-+-+-+-+
| VP8 payload |
+-+-+-+-+-+-+-+-+4.6.2. Non-discardable VP8 Interframe in a Single RTP Packet; No PictureID
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 1 |
+-+-+-+-+-+-+-+-+
|0|0|0|1|0|0 0 0| X = 0; S = 1; PID = 0
+-+-+-+-+-+-+-+-+
|Size0|1| VER |1| P = 1
+-+-+-+-+-+-+-+-+
| Size1 |
+-+-+-+-+-+-+-+-+
| Size2 |
+-+-+-+-+-+-+-+-+
| VP8 payload |
+-+-+-+-+-+-+-+-+4.6.3. VP8 Partitions in Separate RTP Packets
First RTP packet; complete first partition.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 0 |
+-+-+-+-+-+-+-+-+
|1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
|Size0|1| VER |1| P = 1
+-+-+-+-+-+-+-+-+
| Size1 |
+-+-+-+-+-+-+-+-+
| Size2 |
+-+-+-+-+-+-+-+-+
| Octets 4..L of|
| first VP8 |
| partition |
: :
+-+-+-+-+-+-+-+-+Second RTP packet; complete second partition.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 1 |
+-+-+-+-+-+-+-+-+
|1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
| Remaining VP8 |
| partitions |
: :
+-+-+-+-+-+-+-+-+4.6.4. VP8 Frame Fragmented across RTP Packets
First RTP packet; complete first partition.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 0 |
+-+-+-+-+-+-+-+-+
|1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
|Size0|1| VER |1| P = 1
+-+-+-+-+-+-+-+-+
| Size1 |
+-+-+-+-+-+-+-+-+
| Size2 |
+-+-+-+-+-+-+-+-+
| Complete |
| first |
| partition |
: :
+-+-+-+-+-+-+-+-+Second RTP packet; first fragment of second partition.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 0 |
+-+-+-+-+-+-+-+-+
|1|0|0|1|0|0 0 1| X = 1; S = 1; PID = 1
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
| First fragment|
| of second |
| partition |
: :
+-+-+-+-+-+-+-+-+Third RTP packet; second fragment of second partition.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 0 |
+-+-+-+-+-+-+-+-+
|1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
| Mid fragment |
| of second |
| partition |
: :
+-+-+-+-+-+-+-+-+Fourth RTP packet; last fragment of second partition.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 1 |
+-+-+-+-+-+-+-+-+
|1|0|0|0|0|0 0 1| X = 1; S = 0; PID = 1
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1
+-+-+-+-+-+-+-+-+
|0 0 0 1 0 0 0 1| PictureID = 17
+-+-+-+-+-+-+-+-+
| Last fragment |
| of second |
| partition |
: :
+-+-+-+-+-+-+-+-+4.6.5. VP8 Frame with Long PictureID
PictureID = 4711 = 001001001100111 binary (first 7 bits: 0010010, last 8 bits: 01100111).
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| RTP header |
| M = 1 |
+-+-+-+-+-+-+-+-+
|1|0|0|1|0|0 0 0| X = 1; S = 1; PID = 0
+-+-+-+-+-+-+-+-+
|1|0|0|0|0 0 0 0| I = 1;
+-+-+-+-+-+-+-+-+
|1 0 0 1 0 0 1 0| Long PictureID flag = 1
|0 1 1 0 0 1 1 1| PictureID = 4711
+-+-+-+-+-+-+-+-+
|Size0|1| VER |1|
+-+-+-+-+-+-+-+-+
| Size1 |
+-+-+-+-+-+-+-+-+
| Size2 |
+-+-+-+-+-+-+-+-+
| Octets 4..N of|
| VP8 payload |
: :
+-+-+-+-+-+-+-+-+5. Using VP8 with RPSI and SLI Feedback 🔁
The VP8 payload descriptor defined in Section 4.2 contains an optional PictureID parameter. This parameter is included mainly to enable use of reference picture selection indication (RPSI) and slice loss indication (SLI), both defined in [RFC4585] 🦋✨
5.1. RPSI 🦋
okay so RPSI is a payload-specific feedback message that lives inside the RTCP-based feedback format 💅 The receiver is the one who generates it, and it has two jobs. Job one: when the decoder notices a loss, it can ask for a preferred reference picture, ideally one the decoder knows is perfect 🙈 Job two: it can be positive feedback, basically a little "got it, this one decoded fine" 💖 The positive version is especially useful for VP8 in point-to-point (unicast) calls.
The RPSI slaps even harder when it's paired with a special update pattern for VP8's two special reference frames, the golden frame and the altref frame. They take turns getting updated in an alternating leapfrog fashion 🐸✨ When a receiver successfully decodes a golden or altref frame, and that frame has a PictureID in its payload descriptor, the receiver can just send an RPSI message back to the sender to say "yep, received!" The message body (the "native RPSI bit string" in [RFC4585]) is simply that PictureID 🎀
5.2. SLI 🌸
SLI is the other payload-specific feedback message in the RTCP-based feedback format, and it's the receiver's way of shouting "something got corrupted or lost in this frame!" 😭 Here's the layout, bestie:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| First | Number | PictureID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4So First is the macroblock address (in scan order) of the first lost block, and Number is how many lost blocks there are, as defined in [RFC4585] 🧚 PictureID is the six least significant bits of the codec-specific picture identifier where the loss or corruption happened. For VP8, that codec-specific identifier is basically the PictureID of the current frame, read straight from the payload descriptor 👑
Now here's the lowkey clutch part. If the current frame's payload descriptor doesn't have a PictureID, the receiver MAY send the last received PictureID+1 in the SLI message instead 🍓 The receiver MAY also set First to 0 and Number to the total number of macroblocks per frame, even if only part of the frame is corrupted. Like, go big or go home 💅
When the sender gets an SLI, it can use what it already learned from the latest RPSI message ✨ If the last golden or altref frame is known to have arrived safely, the sender can encode the next frame with reference to that established reference. Iconic behaviour, fr fr 🔥
5.3. Example 🎀
Okay, this is the part where it all clicks, I promise 🫶 The example shows how RPSI and SLI work together. Here the encoder isn't allowed to update the altref frame until the last golden frame it sent has been acknowledged with an RPSI message. If no update comes back within some time, a new golden frame update gets sent instead. Once the new golden frame is established and acknowledged, the same rule applies when updating the altref frame. Rules, but make it cute 💜
Check out the timeline below, it's giving storyboard energy 🎬
+-------+-------------------+-------------------------+-------------+
| Event | Sender | Receiver | Established |
| | | | reference |
+-------+-------------------+-------------------------+-------------+
| 1000 | Send golden frame | | |
| | PictureID = 0 | | |
| | | | |
| | | Receive and decode | |
| | | golden frame | |
| | | | |
| 1001 | | Send RPSI(0) | |
| | | | |
| 1002 | Receive RPSI(0) | | golden |
| | | | |
| ... | (sending regular | | |
| | frames) | | |
| | | | |
| 1100 | Send altref frame | | |
| | PictureID = 100 | | |
| | | | |
| | | Altref corrupted or | golden |
| | | lost | |
| | | | |
| 1101 | | Send SLI(100) | golden |
| | | | |
| 1102 | Receive SLI(100) | | |
| | | | |
| 1103 | Send frame with | | |
| | reference to | | |
| | golden | | |
| | | | |
| | | Receive and decode | golden |
| | | frame (decoder state | |
| | | restored) | |
| | | | |
| ... | (sending regular | | |
| | frames) | | |
| | | | |
| 1200 | Send altref frame | | |
| | PictureID = 200 | | |
| | | | |
| | | Receive and decode | golden |
| | | altref frame | |
| | | | |
| 1201 | | Send RPSI(200) | |
| | | | |
| 1202 | Receive RPSI(200) | | altref |
| | | | |
| ... | (sending regular | | |
| | frames) | | |
| | | | |
| 1300 | Send golden frame | | |
| | PictureID = 300 | | |
| | | | |
| | | Receive and decode | altref |
| | | golden frame | |
| | | | |
| 1301 | | Send RPSI(300) | altref |
| | | | |
| 1302 | RPSI lost | | |
| | | | |
| 1400 | Send golden frame | | |
| | PictureID = 400 | | |
| | | | |
| | | Receive and decode | altref |
| | | golden frame | |
| | | | |
| 1401 | | Send RPSI(400) | |
| | | | |
| 1402 | Receive RPSI(400) | | golden |
+-------+-------------------+-------------------------+-------------+
Table 1: Example Signaling between Sender and ReceiverThe scheme is robust to lost feedback messages, which is honestly so reassuring 🌈 If an RPSI gets lost, the sender will try to update the golden (or altref) again after a while, without releasing the established reference. And if an SLI gets lost, the receiver can keep sending SLI messages at any interval allowed by the RTCP sending timing restrictions in [RFC4585], as long as the picture is still corrupted 🦋
6. Payload Format Parameters 💖
This payload format has two optional parameters, so you get to pick your vibe 💅
6.1. Media Type Definition 🌸
This registration follows the template in [RFC6838] and [RFC4855] 🎀 Here's the template, filled in:
Type name: video
Subtype name: VP8
Required parameters: None.
Optional parameters:
These parameters are used to signal the capabilities of a receiver
implementation. If the implementation is willing to receive
media, both parameters MUST be provided. These parameters MUST
NOT be used for any other purpose.
max-fr: The value of max-fr is an integer indicating the maximum
frame rate in units of frames per second that the decoder is
capable of decoding.
max-fs: The value of max-fs is an integer indicating the maximum
frame size in units of macroblocks that the decoder is capable
of decoding.
The decoder is capable of decoding this frame size as long as
the width and height of the frame in macroblocks are less than
int(sqrt(max-fs * 8)). For instance, a max-fs of 1200 (capable
of supporting 640x480 resolution) will support widths and
heights up to 1552 pixels (97 macroblocks).
Encoding considerations:
This media type is framed in RTP and contains binary data; see
Section 4.8 of [RFC6838].
Security considerations: See Section 7 of RFC 7741.
Interoperability considerations: None.
Published specification: VP8 bitstream format [RFC6386] and RFC
7741.
Applications that use this media type:
For example: Video over IP, video conferencing.
Fragment identifier considerations: N/A.
Additional information: None.
Person & email address to contact for further information:
Patrik Westin, patrik.westin@gmail.com
Intended usage: COMMON
Restrictions on usage:
This media type depends on RTP framing, and hence it is only
defined for transfer via RTP [RFC3550].
Author: Patrik Westin, patrik.westin@gmail.com
Change controller:
IETF Payload Working Group delegated from the IESG.So, the optional parameters are for signalling what a receiver can handle 🧚 If an implementation is willing to receive media, it MUST provide both of them, and they MUST NOT be used for any other purpose 🙅♀️
- max-fr is an integer giving the maximum frame rate, in frames per second, that the decoder can decode ⭐
- max-fs is an integer giving the maximum frame size, in macroblocks, that the decoder can decode 🌈
For max-fs, the decoder can handle the frame size as long as the width and height in macroblocks are each less than int(sqrt(max-fs * 8)). For example, a max-fs of 1200 (enough for 640x480) will support widths and heights up to 1552 pixels (97 macroblocks) 🍓
6.2. SDP Parameters 💜
The receiver MUST ignore any fmtp parameter that isn't specified in this memo 🙈
6.2.1. Mapping of Media Subtype Parameters to SDP 🦋
The video/VP8 media type maps onto fields in the Session Description Protocol (SDP) [RFC4566] like this:
- The media name in the
m=line of SDP MUST bevideo. - The encoding name in the
a=rtpmapline of SDP MUST beVP8(the media subtype). - The clock rate in the
a=rtpmapline MUST be90000. - The parameters
max-fsandmax-frMUST be included in thea=fmtpline if the SDP is used to declare receiver capabilities. These parameters are written as a media subtype string, a semicolon-separated list of parameter=value pairs ✨
6.2.1.1. Example ✨
Okay, here's how it looks when you write it up in SDP 📝
m=video 49170 RTP/AVPF 98
a=rtpmap:98 VP8/90000
a=fmtp:98 max-fr=30; max-fs=3600;6.2.2. Offer/Answer Considerations 🤝
VP8 decode complexity is roughly linear with the number of pixels encoded, so the bigger the frame, the more work it is 💅 The max-fr and max-fs parameters are defined in Section 6.1, where the macroblock size is 16x16 pixels as defined in [RFC6386]. The max-fs and max-fr parameters MUST be used to establish these limits 🎀
7. Security Considerations 🔐
RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550], and in any applicable RTP profile such as RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/SAVPF [RFC5124]. Now, bestie, "Securing the RTP Protocol Framework: Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] explains that it's not an RTP payload format's job to discuss or mandate which solutions get used for the basic security goals (confidentiality, integrity, and source authenticity) for RTP in general 🙅♀️ That responsibility lies with whoever is using RTP in an application. They can find guidance on available security mechanisms and important considerations in "Options for Securing RTP Sessions" [RFC7201]. Applications SHOULD use one or more appropriate strong security mechanisms. This is serious, babes 🚨 The rest of this security considerations section talks about the security-impacting properties of the payload format itself.
This RTP payload format and its media decoder do not show any significant difference in receiver-side computational complexity for packet processing, so they are unlikely to pose a denial-of-service threat from receiving pathological data 🛡️ The RTP payload format also does not contain any active content 💜
8. Congestion Control 🚦
Congestion control for RTP SHALL be used in accordance with RFC 3550 [RFC3550] and with any applicable RTP profile; e.g., RFC 3551 [RFC3551]. In a real-time encoding scenario, the congestion control mechanism can adapt the transmission rate by telling the encoder to encode at a certain target rate 🎯 Media-aware network elements MAY use the information in the VP8 payload descriptor in Section 4.2 to identify non-reference frames and discard them in order to reduce network congestion.
Heads up: discarding non-reference frames cannot be done if the stream is encrypted, because the non-reference marker is encrypted 🙈
9. IANA Considerations 📋
The IANA has registered a media type as described in Section 6.1 🌸
10. References 📚
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <http://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003,
<http://www.rfc-editor.org/info/rfc3551>.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, DOI 10.17487/RFC4566,
July 2006, <http://www.rfc-editor.org/info/rfc4566>.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585,
DOI 10.17487/RFC4585, July 2006,
<http://www.rfc-editor.org/info/rfc4585>.
[RFC4855] Casner, S., "Media Type Registration of RTP Payload
Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007,
<http://www.rfc-editor.org/info/rfc4855>.
[RFC6386] Bankoski, J., Koleszar, J., Quillio, L., Salonen, J.,
Wilkins, P., and Y. Xu, "VP8 Data Format and Decoding
Guide", RFC 6386, DOI 10.17487/RFC6386, November 2011,
<http://www.rfc-editor.org/info/rfc6386>.
[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type
Specifications and Registration Procedures", BCP 13,
RFC 6838, DOI 10.17487/RFC6838, January 2013,
<http://www.rfc-editor.org/info/rfc6838>.10.2. Informative References
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, DOI 10.17487/RFC3711, March 2004,
<http://www.rfc-editor.org/info/rfc3711>.
[RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for
Real-time Transport Control Protocol (RTCP)-Based Feedback
(RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February
2008, <http://www.rfc-editor.org/info/rfc5124>.
[RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP
Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014,
<http://www.rfc-editor.org/info/rfc7201>.
[RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP
Framework: Why RTP Does Not Mandate a Single Media
Security Solution", RFC 7202, DOI 10.17487/RFC7202, April
2014, <http://www.rfc-editor.org/info/rfc7202>.
[Sch07] Schwarz, H., Marpe, D., and T. Wiegand, "Overview of the
Scalable Video Coding Extension of the H.264/AVC
Standard", IEEE Transactions on Circuits and Systems for
Video Technology, Volume 17: Issue 9,
DOI 10.1109/TCSVT.2007.905532, September 2007,
<http://dx.doi.org/10.1109/TCSVT.2007.905532>.Authors' Addresses 🖊️
Patrik Westin
Google, Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
United States
Email: patrik.westin@gmail.com
Henrik F Lundin
Google, Inc.
Kungsbron 2
Stockholm 11122
Sweden
Email: hlundin@google.com
Michael Glover
Twitter Boston
10 Hemlock Way
Durham, NH 03824
United States
Email: michaelglover262@gmail.com
Justin Uberti
Google, Inc.
747 6th Street South
Kirkland, WA 98033
United States
Email: justin@uberti.name
Frank Galligan
Google, Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
United States
Email: fgalligan@google.comThat's the whole spec, bestie! 🌈👑
📦 where this page comes from
- ietf: rfc7741 2016-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms