RFC 7742

WebRTC Video Processing and Codec Requirements

📅 published March 2016, from ietf

RFC 7742: WebRTC Video Processing and Codec Requirements ✨

Okay bestie, this is RFC 7742, titled "WebRTC Video Processing and Codec Requirements" 🎀 It was written by A.B. Roach from Mozilla and published in March 2016 as a Standards Track document from the IETF, with ISSN 2070-1721. The header doesn't list anything it obsoletes or updates, so it stands on its own 💅 This is part 1 of 2, so more is coming, periodt 🦋

Abstract ✨

okay so this spec covers what WebRTC apps need to do to send and receive video over a network 📹 It lays out the video processing that's required, plus the video codecs and their parameters. literally the whole video-side rulebook 💖

Status of This Memo 📜

This is an Internet Standards Track document 🌸

It's a product of the Internet Engineering Task Force (IETF) 🧚 It represents the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check Section 2 of RFC 5741 ✨

For the current status of this document, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7742 🔗

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they describe your rights and restrictions for this document 🫶 Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 💜

Table of Contents 🗺️

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   2
3.  Pre- and Post-Processing  . . . . . . . . . . . . . . . . . .   3
  3.1.  Camera-Source Video . . . . . . . . . . . . . . . . . . .   3
  3.2.  Screen-Source Video . . . . . . . . . . . . . . . . . . .   4
4.  Stream Orientation  . . . . . . . . . . . . . . . . . . . . .   4
5.  Mandatory-to-Implement Video Codec  . . . . . . . . . . . . .   5
6.  Codec-Specific Considerations . . . . . . . . . . . . . . . .   6
  6.1.  VP8 . . . . . . . . . . . . . . . . . . . . . . . . . . .   6
  6.2.  H.264 . . . . . . . . . . . . . . . . . . . . . . . . . .   6
7.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   8
  8.1.  Normative References  . . . . . . . . . . . . . . . . . .   8
  8.2.  Informative References  . . . . . . . . . . . . . . . . .   9
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  10
Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  10

1. Introduction ✨

One of the big jobs of WebRTC endpoints is sending and receiving interactive video 🎥 The video could come from a camera, a screen recording, a stored file, or some other source 🌈 This spec gives the requirements and considerations for WebRTC apps to send and receive video across a network, and it covers the video processing that's required plus the video codecs and their parameters 💅

Heads up though: this document only talks about video-codec handling 🙈 Transport-related stuff for moving media streams across the network is covered in [WebRTC-RTP-USAGE] 🦋

2. Terminology 📖

okay so the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are read the way [RFC2119] describes 💜 Here are the definitions the doc uses:

  • A WebRTC browser (also called a WebRTC User Agent or WebRTC UA) is something that conforms to both the protocol specification and the Javascript API (see [RTCWEB-OVERVIEW]). 🌐
  • A WebRTC non-browser is something that conforms to the protocol specification, but it does not claim to implement the Javascript API. This can also be called a "WebRTC device" or "WebRTC native application". 🔧
  • A WebRTC endpoint is either a WebRTC browser or a WebRTC non-browser. It conforms to the protocol specification. 🎀
  • A WebRTC-compatible endpoint is an endpoint that is able to successfully communicate with a WebRTC endpoint but may fail to meet some requirements of a WebRTC endpoint. This may limit where in the network such an endpoint can be attached, or it may limit the security guarantees that it offers to others. It is not constrained by this specification; when it is mentioned at all, it is to note the implications on WebRTC-compatible endpoints of the requirements placed on WebRTC endpoints. 🧩

These definitions also show up in [RTCWEB-OVERVIEW], and that doc is the place to check for more info 🌸

3. Pre- and Post-Processing 🎨

This section is all about guidance on pre- and post-processing of video streams ✨

Unless the Session Description Protocol (SDP) or the codec says otherwise, the color space SHOULD be sRGB [SRGB]. Just so we're clear, that's the color space you get from codepoint 1 of "ColourPrimaries", as defined in [IEC23001-8] 🌈

Unless the SDP or codec says otherwise, the video scan pattern for video codecs is Y'CbCr 4:2:0 📺

3.1. Camera-Source Video 📷

This doc puts no normative requirements on camera capture 🙌 Still, implementors are encouraged to use the following features, if they're doable on their platform:

  • Automatic focus, if it applies to the camera in use 🔍
  • Automatic white balance 🤍
  • Automatic light-level control ☀️
  • Dynamic frame rate for video capture based on the actual encoding in use (e.g., if encoding at 15 fps due to bandwidth constraints, low light conditions, or application settings, the camera will ideally capture at 15 fps rather than a higher rate). 🎞️

3.2. Screen-Source Video 🖥️

If the video source is some portion of a computer screen (e.g., desktop or application sharing), then the considerations in this section also apply 🪟

Screen-sourced video can change resolution (due to, e.g., window resizing and similar operations), so WebRTC-video recipients MUST be prepared to handle midstream resolution changes in a way that preserves their utility 🙈 The precise handling (e.g., resizing the element a video is rendered in versus scaling down the received stream; decisions around letter/pillarboxing) is left to the application's discretion 💅

Heads up: the default video-scan format (Y'CbCr 4:2:0) is known to be less than optimal for representing screen content from most systems in use when this doc was written, which generally use RGB with at least 24 bits per sample 😭 In the future, it may be advisable to use video codecs optimized for screen content for this type of content ✨

Also, pay attention to the requirements in Section 5.2 of [WebRTC-SEC-ARCH] and the considerations in Section 4.1.1. of [WebRTC-SEC] 🔐

4. Stream Orientation 🔄

In some circumstances, especially ones involving mobile devices, the orientation of the camera may not match the orientation the encoder uses 📱 Even bigger deal: the orientation may change over the course of a call, so the receiver has to change the orientation it renders the stream in 🦋

The sender can simply change the pre-encoding orientation of frames, but that may not be practical or efficient (especially when the camera interface returns pre-compressed video frames) 🙃 Note that this behavior adds another way for a screen's resolution to change mid-stream, on top of the ones mentioned in Section 3.2 👀

To cover these situations, WebRTC implementations that can generate media in orientations other than the default MUST support generating the R0 and R1 bits of the Coordination of Video Orientation (CVO) mechanism described in Section 7.4.5 of [TS26.114] and MUST send them for all orientations when the peer indicates support for the mechanism. They MAY support sending the other bits in the CVO extension, including the higher-resolution rotation bits. All implementations SHOULD support interpretation of the R0 and R1 bits and MAY support the other CVO bits. 🌟

Further, some codecs support in-band signaling of orientation (for example, the SEI "Display Orientation" messages in H.264 and H.265 [H265]). If CVO has been negotiated, then the sender MUST NOT make use of such codec-specific mechanisms. However, when support for CVO is not signaled in the SDP, then such implementations MAY make use of the codec-specific mechanisms instead. 🧚

(more coming in part 2: Section 5 onward, the codec stuff, security, references, and the rest of the fam 💖)

6.1. VP8 🌸

okay so for VP8 (the codec from [RFC6386]), endpoints MUST support the payload formats in [RFC7741]. no skipping that bestie 💅

On top of the [RFC6236] mechanism, VP8 encoders MUST limit the streams they send so they fit inside whatever the receivers put in the max-fr and max-fs SDP attributes. the receiver basically says "here's my limit" and the sender has to respect it 🫶

Unless it's signaled otherwise, implementations that use VP8 MUST encode and decode pixels with an implied 1:1 (square) aspect ratio. so square pixels are the default, periodt ✨

6.2. H.264 🎀

For the [H264] codec, endpoints MUST support the payload formats defined in [RFC6184]. they also MUST support Constrained Baseline Profile Level 1.2, and SHOULD support H.264 Constrained High Profile Level 1.3. the MUST is the main character requirement and the SHOULD is a super strong recommendation 💖

H.264 implementations have used a ginormous pile of optional parameters (chaotic, I know 💀). so to keep everyone's vibes compatible, these parameter settings are specified:

  • packetization-mode: Packetization-mode 1 MUST be supported. Other modes MAY be negotiated and used. 🦋
  • profile-level-id: Implementations MUST include this parameter within SDP and MUST interpret it when receiving it. 🌈
  • max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br: these parameters let an implementation say it can handle certain H.264 features at higher rates and values than the ones signaled by its level (set with profile-level-id). implementations MAY include these parameters in their SDP, but they SHOULD interpret them when receiving them, so they can send the highest quality video possible ⭐
  • sprop-parameter-sets: H.264 lets sequence and picture information go both in-band and out-of-band. WebRTC implementations MUST signal this information in-band. which means WebRTC implementations MUST NOT include this parameter in the SDP they generate. not me being the one who has to remember this 🙈

H.264 codecs MAY send, and MUST support proper interpretation of, Supplemental Enhancement Information (SEI) "filler payload" and "full frame freeze" messages. the "full frame freeze" ones are used in video-switching MCUs to keep a stable decoded picture while switching between different input streams 🧚

When the video orientation (CVO) RTP header extension is NOT signaled as part of the SDP, H.264 implementations MAY send Display Orientation SEI messages, and SHOULD support proper interpretation of them. (the SHOULD NOT vibes are not here, so don't worry 😌)

Implementations MAY send and act upon "User data registered by Rec. ITU-T T.35" and "User data unregistered" messages. even if they don't act on them, implementations MUST be prepared to receive such messages without any ill effects. receiving them should never break anything, full stop 🫧

Unless it's signaled otherwise, implementations that use H.264 MUST encode and decode pixels with an implied 1:1 (square) aspect ratio. same energy as VP8, square pixels all the way 🍓

7. Security Considerations 🔐

okay this part is SERIOUS bestie 😭 this specification doesn't introduce any new mechanisms or security concerns beyond what's in the other documents it references. in WebRTC, video is protected using Datagram Transport Layer Security (DTLS) / Secure Real-time Transport Protocol (SRTP). the full security discussion lives in [WebRTC-SEC] and [WebRTC-SEC-ARCH], so go read those, they're iconic 👑

implementors should consider whether variable bitrate video codecs are appropriate for their application. why? because the degree of inter-frame change (and, by inference, how much motion is in the frame) can be deduced by an eavesdropper just by looking at the video stream's bitrate. eek, it's giving surveillance 👀💀

implementors making use of H.264 are also advised to take careful note of the "Security Considerations" section of [RFC6184], paying special regard to the normative requirement about SEI messages. read the fine print, it matters 🫶

8. References ✨

8.1. Normative References

8.2. Informative References

  • [H265]: ITU-T, "High efficiency video coding", ITU-T Recommendation H.265, April 2015, http://www.itu.int/rec/T-REC-H.265. 🎞️
  • [RTCWEB-OVERVIEW]: Alvestrand, H., "Overview: Real Time Protocols for Browser-based Applications", Work in Progress, draft-ietf-rtcweb-overview-14, June 2015. 🧭
  • [WebRTC-RTP-USAGE]: Perkins, C., Westerlund, M., and J. Ott, "Web Real-Time Communication (WebRTC): Media Transport and Use of RTP", Work in Progress, draft-ietf-rtcweb-rtp-usage-25, June 2015. 🚚
  • [WebRTC-SEC]: Rescorla, E., "Security Considerations for WebRTC", Work in Progress, draft-ietf-rtcweb-security-08, February 2015. 🔐
  • [WebRTC-SEC-ARCH]: Rescorla, E., "WebRTC Security Architecture", Work in Progress, draft-ietf-rtcweb-security-arch-11, March 2015. 🏰

Acknowledgements 🫶

The author would like to thank Gaelle Martin-Cocher, Stephan Wenger, and Bernard Aboba for their detailed feedback and help with this document. thanks to Cullen Jennings for providing text and review, and to Russ Housley for a careful final review. this document includes text that originally appeared in "WebRTC Codec and Media Processing Requirements" (March 2012). the real MVPs 👑💖

Author's Address 💌

Adam Roach

Mozilla

Dallas

United States

Phone: +1 650 903 0800 x863

Email: adam@nostrum.com

📦 where this page comes from