RFC 8761
Video Codec Requirements and Evaluation Methodology
📄 RFC 8761: Video Codec Requirements and Evaluation Methodology 💖
okay bestie, this one is from the IETF (the internet's standards crew 🧚) and it's published as Informational, which means it's not a Standards Track spec. it's giving "here's what we want, and here's how to check it" 💅
- Authors: A. Filippov (Huawei Technologies), A. Norkin (Netflix), and J.R. Alvarez (Huawei Technologies) ✨
- Status: Informational (not Standards Track) 🎀
- ISSN: 2070-1721 🌸
- Date: April 2020 📅
- Obsoletes and updates: neither! its header doesn't list anything it obsoletes or updates, so it's a solo act 🦋
Abstract 🌈
so the whole point of this doc is to lay out what a video codec designed mainly for the Internet should be able to do 🎬 and then it gives you an evaluation methodology, which is basically a way to measure compression efficiency and figure out if those requirements actually got met 💜 slay.
Status of This Memo 📝
ngl this section is the "here's what kind of document this is" part. it is not an Internet Standards Track specification, it's just published for info 🫶
it IS a product of the IETF, it represents the IETF community's consensus, it got public review, and the IESG (the Internet Engineering Steering Group 👑) approved it for publication. not every IESG-approved doc is a candidate for any level of Internet Standard though, so check out Section 2 of RFC 7841 if you want the deets 🔥
more info about the current status, errata, and how to send feedback lives at https://www.rfc-editor.org/info/rfc8761 🧚
Copyright Notice 💅
copyright (c) 2020 by the IETF Trust and the people listed as the document authors, all rights reserved 🩷
this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they were on the publication date. please actually read those, bestie, because they explain your rights and restrictions 🙈
also, if you pull Code Components out of this doc, they have to include the Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License says 💀
Table of Contents 🍓
1. Introduction
2. Terminology Used in This Document
2.1. Definitions
2.2. Abbreviations
3. Applications
3.1. Internet Video Streaming
3.2. Internet Protocol Television (IPTV)
3.3. Video Conferencing
3.4. Video Sharing
3.5. Screencasting
3.6. Game Streaming
3.7. Video Monitoring and Surveillance
4. Requirements
4.1. General Requirements
4.1.1. Coding Efficiency
4.1.2. Profiles and Levels
4.1.3. Bitstream Syntax
4.1.4. Parsing and Identification of Sample Components
4.1.5. Perceptual Quality Tools
4.1.6. Buffer Model
4.1.7. Integration
4.2. Basic Requirements
4.2.1. Input Source Formats
4.2.2. Coding Delay
4.2.3. Complexity
4.2.4. Scalability
4.2.5. Error Resilience
4.3. Optional Requirements
4.3.1. Input Source Formats
4.3.2. Scalability
4.3.3. Complexity
4.3.4. Coding Efficiency
5. Evaluation Methodology
6. Security Considerations
7. IANA Considerations
8. References
8.1. Normative References
8.2. Informative References
Acknowledgments
Authors' Addresses1. Introduction ✨
okay so this section sets the stage 🎀 it lays out the requirements for a video codec built mainly for the Internet, and that covers a LOT: Internet video streaming, IPTV, peer-to-peer video conferencing, video sharing, screencasting, game streaming, and video monitoring and surveillance 🎬
for each of those apps, it gives typical resolutions, frame rates, and picture-access modes. it also covers requirements for sending data over networks where packets get lost 📦 which is a big deal, bestie.
quick note on scope: when the doc talks about data protection, it only means methods designed and built to protect data inside the video codec itself. there are already tons of techniques that protect generic data over lossy networks, so the codec side is the focus 🧚
the vibe check: from a theory standpoint, both packet-loss and bit-error robustness can help video codecs 🌸 but in practice, packet losses are a bigger problem than bit corruption in IP networks 😭
and here's the lowkey important bit. how much delay you can tolerate is tied up with how error-robust your video stream needs to be 🫶 specifically:
- if delay isn't crucial for your app, reliable transport protocols like TCP can be used, because they retransmit undelivered packets and that guarantees correct decoding of the transmitted data 💜
- if delay has to stay low, then either the data transmission should be error free (for example, by using managed networks), or the compressed video stream should be error resilient 🔥
so, error resilience is useful for delay-critical apps because it helps keep delay low in a packet-loss environment. main character energy for error-resilient streams 👑
2. Terminology Used in This Document 🦋
2.1. Definitions 💖
here are the definitions, and they're kinda important so pay attention 🙈
High dynamic range imaging a set of techniques that lets you capture a greater dynamic range of exposures or values than normal digital imaging. basically, a wider range of values between light and dark areas. the goal is to accurately show the huge range of intensity levels you see in things like exterior scenes with light-colored stuff hit by direct sunlight and areas of deep shadow [7] 🌤️🌑
Random access period the amount of time between the two closest independently decodable frames (pictures) ⏱️✨
RD-point a point in a two-dimensional rate-distortion space, where the bitrate and the quality metric are used as the x- and y-coordinates, respectively 📈
Visually lossless compression a form of lossy compression where the data that gets lost after the file is compressed and decompressed is not detectable to the eye. the compressed data look identical to the uncompressed data [8] 👀✨
Wide color gamut a certain complete color subset (for example, the one considered in ITU-R BT.2020 [1]) that supports a wider range of colors than conventional color gamuts (for example, the ones considered in ITU-R BT.601 [17] or BT.709 [20]). "wider" here means an extended range of colors that a specific input or output device (like a video camera, monitor, or printer) can generate, and that a color model can interpret 🌈
2.2. Abbreviations 🍓
AI All-Intra (each picture is intra-coded)
BD-Rate Bjontegaard Delta Rate
FIZD just the First picture is Intra-coded, Zero structural
Delay
FPS Frames per Second
GOP Group of Picture
GPU Graphics Processing Unit
HBR High Bitrate Range
HDR High Dynamic Range
HRD Hypothetical Reference Decoder
HEVC High Efficiency Video Coding
IPTV Internet Protocol Television
LBR Low Bitrate Range
MBR Medium Bitrate Range
MOS Mean Opinion Score
MS-SSIM Multi-Scale Structural Similarity quality index
PAM Picture Access Mode
PSNR Peak Signal-to-Noise Ratio
QoS Quality of Service
QP Quantization Parameter
RA Random Access
RAP Random Access Period
RD Rate-Distortion
SEI Supplemental Enhancement Information
SIMD Single Instruction, Multiple Data
SNR Signal-to-Noise Ratio
UGC User-Generated Content
VDI Virtual Desktop Infrastructure
VUI Video Usability Information
WCG Wide Color Gamut3. Applications 🎀
okay this section is the app roundup 📱 it gives an overview of the video codec applications that are currently out there on the Internet market, and each one has different use cases 🧚
here's the key thing: each use case defines a target platform, and that means different communication channels get involved (wired or wireless, for example). those channels have different QoS and bandwidth. wired channels are way more free from error than wireless ones, so they need different QoS approaches 🔌📶
the target platform, the channel bandwidth, and the channel quality together decide the resolutions, frame rates, and either the quality or the bitrates for the video streams being encoded or decoded 💅
and by default, the color format YCbCr 4:2:0 is assumed for all the application scenarios below 🌸
3.1. Internet Video Streaming
okay bestie, so this one is all about movies, TV series and shows, and animation 🎬✨ Internet video streaming runs on a whole mix of client devices, and the network conditions keep changing, so the adaptive streaming model became super popular 💅 The idea is that video gets encoded at different quality levels and resolutions, and the client picks one based on what it can handle and how much bandwidth it has right now 🦋 Table 1 shows an example combo of resolutions and bitrates, and it's giving organized energy 🎀
A video encoding pipeline for on-demand Internet video streaming usually goes like this 🌸
- Video is encoded in the cloud by software encoders. ☁️
- Source video is split into chunks, each of which is encoded separately, in parallel. 🧩
- Closed-GOP encoding with intrapicture intervals of 2-5 seconds (or longer) is used. ⏱️
- Encoding is perceptually optimized. Perceptual quality is important and should be considered during the codec development. 👀
+------------+-----+------------------------------------------------+
| Resolution | PAM | Frame Rate, FPS ** |
| * | | |
+============+=====+================================================+
| 4K, | RA | 24/1.001, 24, 25, |
| 3840x2160 | | 30/1.001, 30, 50, |
+------------+-----+ 60/1.001, 60, 100, |
| 2K | RA | 120/1.001, 120 |
| (1080p), | | |
| 1920x1080 | | |
+------------+-----+ |
| 1080i, | RA | |
| 1920x1080* | | |
+------------+-----+ |
| 720p, | RA | |
| 1280x720 | | |
+------------+-----+ |
| 576p | RA | |
| (EDTV), | | |
| 720x576 | | |
+------------+-----+ |
| 576i | RA | |
| (SDTV), | | |
| 720x576* | | |
+------------+-----+ |
| 480p | RA | |
| (EDTV), | | |
| 720x480 | | |
+------------+-----+ |
| 480i | RA | |
| (SDTV), | | |
| 720x480* | | |
+------------+-----+ |
| 512x384 | RA | |
+------------+-----+ |
| QVGA, | RA | |
| 320x240 | | |
+------------+-----+------------------------------------------------+
Table 1: Internet Video Streaming: Typical Values of Resolutions,
Frame Rates, and PAMs*Note: Interlaced content can be handled at the higher system level and not necessarily by using specialized video coding tools. It is included in this table only for the sake of completeness, as most video content today is in the progressive format. 📺
**Note: The set of frame rates presented in this table is taken from Table 2 in [1]. 🎞️
The characteristics and requirements of this application scenario are as follows 💖
- High encoder complexity (up to 10x and more) can be tolerated since encoding happens once and in parallel for different segments. 🙌
- Decoding complexity should be kept at reasonable levels to enable efficient decoder implementation. 📱
- Support and efficient encoding of a wide range of content types and formats is required:
- High Dynamic Range (HDR), Wide Color Gamut (WCG), high-resolution (currently, up to 4K), and high-frame-rate content are important use cases; the codec should be able to encode such content efficiently. 🌈
- Improvement of coding efficiency at both lower and higher resolutions is important since low resolutions are used when streaming in low-bandwidth conditions. 📉
- Improvement on both "easy" and "difficult" content in terms of compression efficiency at the same quality level contributes to the overall bitrate/storage savings. 💰
- Film grain (and sometimes other types of noise) is often present in movies and similar content; this is usually part of the creative intent. 🎥
- Significant improvements in compression efficiency between generations of video standards are desirable since this scenario typically assumes long-term support of legacy video codecs. 🕰️
- Random access points are inserted frequently (one per 2-5 seconds) to enable switching between resolutions and fast-forward playback. ⏩
- The elementary stream should have a model that allows easy parsing and identification of the sample components. 🔍
- Middle QP values are normally used in streaming; this is also the range where compression efficiency is important for this scenario. 🎯
- Scalability or other forms of supporting multiple quality representations are beneficial if they do not incur significant bitrate overhead and if mandated in the first version. 🧚
3.2. Internet Protocol Television (IPTV)
IPTV is a service for delivering TV content over IP-based networks 📡 It can be sorted into two main groups based on how the delivery works 🫶
- unicast (e.g., for video on demand), where delay is not crucial; and
- multicast/broadcast (e.g., for transmitting news) where zapping (i.e., stream changing) delay is important.
In the IPTV scenario, traffic is transmitted over managed (QoS-based) networks 🔒 Typical content here is news, movies, cartoons, series, TV shows, etc. 🍿 One important requirement for both groups is that random access to pictures (i.e., the random access period (RAP)) should be kept small enough (approximately 1-5 seconds) ⚡ Optional requirements are as follows 🌟
- Temporal (frame-rate) scalability; and
- Resolution and quality (SNR) scalability.
For this application, typical values of resolutions, frame rates, and PAMs are presented in Table 2 🦋
+------------+-----+------------------------------------------------+
| Resolution | PAM | Frame Rate, FPS ** |
| * | | |
+============+=====+================================================+
| 2160p | RA | 24/1.001, 24, 25, |
| (4K), | | 30/1.001, 30, 50, |
| 3840x2160 | | 60/1.001, 60, 100, |
+------------+-----+ 120/1.001, 120 |
| 1080p, | RA | |
| 1920x1080 | | |
+------------+-----+ |
| 1080i, | RA | |
| 1920x1080* | | |
+------------+-----+ |
| 720p, | RA | |
| 1280x720 | | |
+------------+-----+ |
| 576p | RA | |
| (EDTV), | | |
| 720x576 | | |
+------------+-----+ |
| 576i | RA | |
| (SDTV), | | |
| 720x576* | | |
+------------+-----+ |
| 480p | RA | |
| (EDTV), | | |
| 720x480 | | |
+------------+-----+ |
| 480i | RA | |
| (SDTV), | | |
| 720x480* | | |
+------------+-----+------------------------------------------------+
Table 2: IPTV: Typical Values of Resolutions, Frame Rates, and PAMs*Note: Interlaced content can be handled at the higher system level and not necessarily by using specialized video coding tools. It is included in this table only for the sake of completeness, as most video content today is in a progressive format. 📺
**Note: The set of frame rates presented in this table is taken from Table 2 in [1]. 🎞️
Okay bestie, let's get into it! 🎀
Wait, I'm not going to add that opener. Let me restart cleanly, picking up exactly where part 2 stopped. 💅
3.3. Video Conferencing ✨
This is a form of video connection over the Internet. 🌸 It lets users set up connections with two or more people using two-way video and audio, so they can talk in real time. 📞 Both stationary and mobile devices can be used here, which is giving total flexibility. 💖 The main requirements are:
- Delay should be kept as low as possible (the preferable and maximum end-to-end delay values should be less than 100 ms [9] and 320 ms [2], respectively); 😭
- Temporal (frame-rate) scalability; and 🎞️
- Error robustness. 🛡️
Support of resolution and quality (SNR) scalability is highly desirable. 👑 Typical values of resolutions, frame rates, and PAMs for this application are shown in Table 3. 🦋
+------------------+-----------------+------+
| Resolution | Frame Rate, FPS | PAM |
+==================+=================+======+
| 1080p, 1920x1080 | 15, 30 | FIZD |
+------------------+-----------------+------+
| 720p, 1280x720 | 30, 60 | FIZD |
+------------------+-----------------+------+
| 4CIF, 704x576 | 30, 60 | FIZD |
+------------------+-----------------+------+
| 4SIF, 704x480 | 30, 60 | FIZD |
+------------------+-----------------+------+
| VGA, 640x480 | 30, 60 | FIZD |
+------------------+-----------------+------+
| 360p, 640x360 | 30, 60 | FIZD |
+------------------+-----------------+------+
Table 3: Video Conferencing: Typical
Values of Resolutions, Frame Rates, and
PAMs3.4. Video Sharing 🍓
This is a service that lets people upload and share video data (live streaming or not) and watch those videos. 📺 It's also known as video hosting. ✨ A typical User-Generated Content (UGC) scenario for this one is capturing video with mobile cameras like GoPros or cameras built into smartphones (amateur video). 📱 The main requirements are:
- Random access to pictures for downloaded video data;
- Temporal (frame-rate) scalability; and
- Error robustness.
Support of resolution and quality (SNR) scalability is highly desirable. 💜 Typical values of resolutions, frame rates, and PAMs for this application are presented in Table 4. 🌈
Typical values of resolutions and frame rates in Table 4 are taken from [10]. 📚
+-----------------------+------------------------+-----+
| Resolution | Frame Rate, FPS | PAM |
+=======================+========================+=====+
| 2160p (4K), 3840x2160 | 24, 25, 30, 48, 50, 60 | RA |
+-----------------------+------------------------+-----+
| 1440p (2K), 2560x1440 | 24, 25, 30, 48, 50, 60 | RA |
+-----------------------+------------------------+-----+
| 1080p, 1920x1080 | 24, 25, 30, 48, 50, 60 | RA |
+-----------------------+------------------------+-----+
| 720p, 1280x720 | 24, 25, 30, 48, 50, 60 | RA |
+-----------------------+------------------------+-----+
| 480p, 854x480 | 24, 25, 30, 48, 50, 60 | RA |
+-----------------------+------------------------+-----+
| 360p, 640x360 | 24, 25, 30, 48, 50, 60 | RA |
+-----------------------+------------------------+-----+
Table 4: Video Sharing: Typical Values of
Resolutions, Frame Rates, and PAMs3.5. Screencasting 🖥️
This is a service that lets users record and distribute video data from a computer screen. 🎬 It needs efficient compression of computer-generated content with high visual quality, up to visually and mathematically (numerically) lossless [11]. 🧠 Right now this covers business presentations (PowerPoint, Word documents, email messages, etc.), animation (cartoons), gaming content, and data visualization. 🎨 This kind of content has fast motion, rotation, smooth shading, 3D effects, highly saturated colors at full resolution, clear textures, and sharp edges with distinct colors [11]. It also covers virtual desktop infrastructure (VDI), screen/desktop sharing and collaboration, supervisory control and data acquisition (SCADA) displays, automotive/navigation displays, cloud gaming, factory automation displays, wireless displays, display walls, digital operating rooms (DiOR), etc. 🏥 For this application, an important requirement is support of low-delay configurations with zero structural delay for a wide range of video formats (e.g., RGB) in addition to YCbCr 4:2:0 and YCbCr 4:4:4 [11]. 🔥 Typical values of resolutions, frame rates, and PAMs for this application are presented in Table 5. 💅
+-----------------------+-----------------+--------------+
| Resolution | Frame Rate, FPS | PAM |
+=======================+=================+==============+
| Input color format: RGB 4:4:4 |
+-----------------------+-----------------+--------------+
| 5k, 5120x2880 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| 4k, 3840x2160 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| WQXGA, 2560x1600 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| WUXGA, 1920x1200 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| WSXGA+, 1680x1050 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| WXGA, 1280x800 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| XGA, 1024x768 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| SVGA, 800x600 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| VGA, 640x480 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| Input color format: YCbCr 4:4:4 |
+-----------------------+-----------------+--------------+
| 5k, 5120x2880 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| 4k, 3840x2160 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| 1440p (2K), 2560x1440 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| 1080p, 1920x1080 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
| 720p, 1280x720 | 15, 30, 60 | AI, RA, FIZD |
+-----------------------+-----------------+--------------+
Table 5: Screencasting for RGB and YCbCr 4:4:4 Format:
Typical Values of Resolutions, Frame Rates, and PAMs3.6. Game Streaming 🎮
This is a service that provides game content over the Internet to different local devices, like notebooks and gaming tablets. 🧚 In this category, the server renders 3D games in a cloud server and streams the game to any device with a wired or wireless broadband connection [12]. There are low-latency requirements for sending user interactions and receiving game data, with a turnaround delay of less than 100 ms. ⚡ This lets anyone play (or resume) full-featured games from anywhere on the Internet [12]. An example is Nvidia Grid [12]. Another scenario in this category is broadcasting video games that people play over the Internet, either in real time or for later viewing [12]. Companies such as Twitch and YY in China enable game broadcasting [12]. 🙈 Games typically have a lot of sharp edges and large motion [12]. The main requirements are:
- Random access to pictures for game broadcasting;
- Temporal (frame-rate) scalability; and
- Error robustness.
Support of resolution and quality (SNR) scalability is highly desirable. 🫶 For this application, typical values of resolutions, frame rates, and PAMs are similar to the ones in Table 3. 🌸
3.7. Video Monitoring and Surveillance 📹
This is a type of live broadcasting over IP-based networks. 📡 Video streams go to many receivers at the same time. 👀 A new receiver may join the stream at any moment, so the random access period should be kept small enough (approximately, 1-5 seconds). ⏱️ Data is transmitted publicly in the case of video monitoring and privately in the case of video surveillance. 🔒 For IP cameras that have to capture, process, and encode video data, complexity (including computational and hardware complexity, as well as memory bandwidth) should be kept low to allow real-time processing. 🤖 Support of a high dynamic range and a monochrome mode (e.g., for infrared cameras), as well as resolution and quality (SNR) scalability, is an essential requirement for video surveillance. 🌙 In some use cases, high video signal fidelity is required even after lossy compression. 💎 Typical values of resolutions, frame rates, and PAMs for video monitoring and surveillance applications are presented in Table 6. 🦋
+-----------------------+-----------------+----------+
| Resolution | Frame Rate, FPS | PAM |
+=======================+=================+==========+
| 2160p (4K), 3840x2160 | 12, 25, 30 | RA, FIZD |
+-----------------------+-----------------+----------+
| 5Mpixels, 2560x1920 | 12, 25, 30 | RA, FIZD |
+-----------------------+-----------------+----------+
| 1080p, 1920x1080 | 25, 30 | RA, FIZD |
+-----------------------+-----------------+----------+
| 1.23Mpixels, 1280x960 | 25, 30 | RA, FIZD |
+-----------------------+-----------------+----------+
| 720p, 1280x720 | 25, 30 | RA, FIZD |
+-----------------------+-----------------+----------+
| SVGA, 800x600 | 25, 30 | RA, FIZD |
+-----------------------+-----------------+----------+
Table 6: Video Monitoring and Surveillance:
Typical Values of Resolutions, Frame Rates, and
PAMs4. Requirements 👑
Taking the requirements discussed above for specific video applications, this section proposes requirements for an Internet video codec. 🎀
4.1. General Requirements 🌈
4.1.1. Coding Efficiency 💖
The most fundamental requirement is coding efficiency, i.e., compression performance on both "easy" and "difficult" content for the applications and use cases in Section 3. 🧁 The codec should provide higher coding efficiency over state-of-the-art video codecs such as HEVC/H.265 and VP9, at least 25%, in accordance with the methodology described in Section 5 of this document. 📊 For higher resolutions, the improvements in coding efficiency are expected to be higher than for lower resolutions. 🚀
4.1.2. Profiles and Levels 💅
okay bestie, good-quality specs and well-defined profiles and levels are literally required so devices can actually talk to each other and decoder implementers don't cry 😭. a profile is a subset of the entire bitstream syntax elements, so it also defines the tools you need to decode a conforming bitstream of that profile 🎀. a level puts numerical limits on the values of some syntax elements. Table 7 shows an example of the codec levels to support ✨. a real level definition should include constraints on features that affect decoder complexity, like maximum bitrate, line buffer size, memory usage, etc. 💜
+-------+-----------------------------------------------------------+
| Level | Example picture resolution at highest frame rate |
+=======+===========================================================+
| 1 | 128x96(12,288*)@30.0 |
| | 176x144(25,344*)@15.0 |
+-------+-----------------------------------------------------------+
| 2 | 352x288(101,376*)@30.0 |
+-------+-----------------------------------------------------------+
| 3 | 352x288(101,376*)@60.0 |
| | 640x360(230,400*)@30.0 |
+-------+-----------------------------------------------------------+
| 4 | 640x360(230,400*)@60.0 |
| | 960x540(518,400*)@30.0 |
+-------+-----------------------------------------------------------+
| 5 | 720x576(414,720*)@75.0 |
| | 960x540(518,400*)@60.0 |
| | 1280x720(921,600*)@30.0 |
+-------+-----------------------------------------------------------+
| 6 | 1,280x720(921,600*)@68.0 |
| | 2,048x1,080(2,211,840*)@30.0 |
+-------+-----------------------------------------------------------+
| 7 | 1,280x720(921,600*)@120.0 |
+-------+-----------------------------------------------------------+
| 8 | 1,920x1,080(2,073,600*)@120.0 |
| | 3,840x2,160(8,294,400*)@30.0 |
| | 4,096x2,160(8,847,360*)@30.0 |
+-------+-----------------------------------------------------------+
| 9 | 1,920x1,080(2,073,600*)@250.0 |
| | 4,096x2,160(8,847,360*)@60.0 |
+-------+-----------------------------------------------------------+
| 10 | 1,920x1,080(2,073,600*)@300.0 |
| | 4,096x2,160(8,847,360*)@120.0 |
+-------+-----------------------------------------------------------+
| 11 | 3,840x2,160(8,294,400*)@120.0 |
| | 8,192x4,320(35,389,440*)@30.0 |
+-------+-----------------------------------------------------------+
| 12 | 3,840x2,160(8,294,400*)@250.0 |
| | 8,192x4,320(35,389,440*)@60.0 |
+-------+-----------------------------------------------------------+
| 13 | 3,840x2,160(8,294,400*)@300.0 |
| | 8,192x4,320(35,389,440*)@120.0 |
+-------+-----------------------------------------------------------+
Table 7: Codec Levels*Note: The quantities of pixels are presented for applications in which a picture can have an arbitrary size (e.g., screencasting).
4.1.3. Bitstream Syntax 🦋
bitstream syntax should allow extensibility and backward compatibility, no cap 🫶. new features can be supported easily by using metadata (such as SEI messages, VUI, and headers) without affecting bitstream compatibility with legacy decoders. a newer version of the decoder shall be able to play bitstreams of an older version of the same or lower profile and level 👑.
4.1.4. Parsing and Identification of Sample Components 🧚
a bitstream should have a model that allows easy parsing and identification of the sample components (such as Annex B of ISO/IEC 14496-10 [18] or ISO/IEC 14496-15 [19]) ✨. especially, info needed for packet handling (e.g., frame type) should not require parsing anything below the header level. it's giving efficiency 💅.
4.1.5. Perceptual Quality Tools 🌸
perceptual quality tools (such as adaptive QP and quantization matrices) should be supported by the codec bitstream 💖.
4.1.6. Buffer Model 🍓
the codec specification shall define a buffer model such as hypothetical reference decoder (HRD) 🔥.
4.1.7. Integration 🫶
specifications providing integration with system and delivery layers should be developed 🌈.
4.2. Basic Requirements 🎀
4.2.1. Input Source Formats 🌸
input pictures coded by a video codec should have one of the following formats 💅:
- Bit depth: 8 and 10 bits (up to 12 bits for a high profile) per color component.
- Color sampling formats:
- YCbCr 4:2:0
- YCbCr 4:4:4, YCbCr 4:2:2, and YCbCr 4:0:0 (preferably in different profile(s))
- For profiles with bit depth of 10 bits per sample or higher, support of high dynamic range and wide color gamut 🌈.
- Support of arbitrary resolution according to the level constraints for applications in which a picture can have an arbitrary size (e.g., in screencasting) 🦋.
exemplary input source formats for codec profiles are shown in Table 8 ✨.
+---------+--------------------------------+------------------------+
| Profile | Bit depths per color component | Color sampling |
| | | formats |
+=========+================================+========================+
| 1 | 8 and 10 | 4:0:0 and 4:2:0 |
+---------+--------------------------------+------------------------+
| 2 | 8 and 10 | 4:0:0, 4:2:0, |
| | | and 4:4:4 |
+---------+--------------------------------+------------------------+
| 3 | 8, 10, and 12 | 4:0:0, 4:2:0, |
| | | 4:2:2, and 4:4:4 |
+---------+--------------------------------+------------------------+
Table 8: Exemplary Input Source Formats for Codec Profiles4.2.2. Coding Delay ⏱️
to meet coding delay requirements, a video codec should support all of the following 🧁:
- Support of configurations with zero structural delay, also referred to as "low-delay" configurations.
- Note: End-to-end delay should be no more than 320 ms [2], but it is preferable for its value to be less than 100 ms [9].
- Support of efficient random access point encoding (such as intracoding and resetting of context variables), as well as efficient switching between multiple quality representations 💜.
- Support of configurations with nonzero structural delay (such as out-of-order or multipass encoding) for applications without low-delay requirements, if such configurations provide additional compression efficiency improvements ✨.
4.2.3. Complexity 🧠
encoding and decoding complexity considerations are as follows 💅:
- Feasible real-time implementation of both an encoder and a decoder supporting a chosen subset of tools for hardware and software implementation on a wide range of state-of-the-art platforms. The subset of real-time encoder tools should provide meaningful improvement in compression efficiency at reasonable complexity of hardware and software encoder implementations as compared to real-time implementations of state-of-the-art video compression technologies such as HEVC/H.265 and VP9 🔥.
- High-complexity software encoder implementations used by offline encoding applications can have a 10x or more complexity increase compared to state-of-the-art video compression technologies such as HEVC/H.265 and VP9 💀.
4.2.4. Scalability 🌈
the mandatory scalability requirement is as follows:
- Temporal (frame-rate) scalability should be supported 🎀.
4.2.5. Error Resilience 🛡️
to meet the error resilience requirement, a video codec should satisfy all of the following conditions 🫶:
- Tools that are complementary to the error-protection mechanisms implemented on the transport level should be supported.
- The codec should support mechanisms that facilitate packetization of a bitstream for common network protocols.
- Packetization mechanisms should enable frame-level error recovery by means of retransmission or error concealment.
- The codec should support effective mechanisms for allowing decoding and reconstruction of significant parts of pictures in the event that parts of the picture data are lost in transmission.
- The bitstream specification shall support independently decodable subframe units similar to slices or independent tiles. It shall be possible for the encoder to restrict the bitstream to allow parsing of the bitstream after a packet loss and to communicate it to the decoder 😭.
4.3. Optional Requirements ✨
4.3.1. Input Source Formats 🍓
it is a desired but not mandatory requirement for a video codec to support some of the following features 💫:
- Bit depth: up to 16 bits per color component.
- Color sampling formats: RGB 4:4:4.
- Auxiliary channel (e.g., alpha channel) support.
4.3.2. Scalability 🦋
desirable scalability requirements are as follows 💜:
- Resolution and quality (SNR) scalability that provides a low-compression efficiency penalty (increase of up to 5% of BD-rate [13] per layer with reasonable increase of both computational and hardware complexity) can be supported in the main profile of the codec being developed by the NETVC Working Group. Otherwise, a separate profile is needed to support these types of scalability.
- Computational complexity scalability (i.e., computational complexity is decreasing along with degrading picture quality) is desirable.
4.3.3. Complexity ⚡
tools that enable parallel processing (e.g., slices, tiles, and wave-front propagation processing) at both encoder and decoder sides are highly desirable for many applications 🧚.
- High-level multicore parallelism: encoder and decoder operation, especially entropy encoding and decoding, should allow multiple frames or subframe regions (e.g., 1D slices, 2D tiles, or partitions) to be processed concurrently, either independently or with deterministic dependencies that can be efficiently pipelined.
- Low-level instruction-set parallelism: favor algorithms that are SIMD/GPU friendly over inherently serial algorithms
4.3.4. Coding Efficiency 🎬
compression efficiency on noisy content, content with film grain, computer generated content, and low resolution materials is desirable 🌟.
5. Evaluation Methodology ✨
okay bestie, so the whole testing setup is basically a three-lane highway 🛣️🦋 compression performance gets checked across three ranges that all overlap a little, and together they cover ten different bitrate values. no cap, the overlaps are the whole point 💅
- Low bitrate range (LBR) 🌸 is the range with the four lowest of the ten bitrates (one of those four is shared with the neighboring range).
- Medium bitrate range (MBR) 💜 is the range with the four middle bitrates (two of those four are shared with the neighboring ranges).
- High bitrate range (HBR) 🔥 is the range with the four highest of the ten bitrates (one of those four is shared with the neighboring range).
so first up, the reference codec (like HEVC or VP9) gets a set of ten QP (quantization parameter) values, same as in [14], and we work out the matching quality values for each one 🎀 in Figure 1 these show up as "QP0"-"QP9" and "Q0"-"Q9". to make sure the quality levels actually overlap between the reference codec and the tested codec, we run a quality alignment procedure on the outermost quality levels of each range. that's the leftmost and rightmost ones of the reference codec (so Q0, Q3, Q6, and Q9) and their partners on the tested codec (Q'0, Q'3, Q'6, and Q'9). the tested codec's Q'k values, and the QP'k values that go with them (QP'0, QP'3, QP'6, and QP'9), should be picked with these formulas:
Q'k = min { abs(Q'i - Qk) },
i in R
QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) },
i in Rwhere R is the range of the QP indexes of the tested codec, i.e., the candidate Internet video codec 🧚
the inner quality levels (Q'1, Q'2, Q'4, Q'5, Q'7, and Q'8), plus their QP values (QP'1, QP'2, QP'4, QP'5, QP'7, and QP'8), should be spaced as equidistantly as possible between the leftmost and rightmost levels of each range. they don't get the explicit mapping treatment from the procedure above, just the even spacing ✨
here's the picture, it's giving very clear vibes:
QP'9 QP'8 QP'7 QP'6 QP'5 QP'4 QP'3 QP'2 QP'1 QP'0 <+-----
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | Tested
| | | | | | | | | | | codec
Q'0 Q'1 Q'2 Q'3 Q'4 Q'5 Q'6 Q'7 Q'8 Q'9 <+-----
^ ^ ^ ^
| | | |
Q0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 <+-----
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | Reference
| | | | | | | | | | | codec
QP9 QP8 QP7 QP6 QP5 QP4 QP3 QP2 QP1 QP0 <+-----
+----------------+--------------+--------------+--------->
^ ^ ^ ^ Bitrate
|-------LBR------| |-----HBR------|
^ ^
|------MBR-----|Figure 1: Quality/QP Alignment for Compression Performance Evaluation
bestie, the QP mapping can come out different for different sequences, so this alignment has to be done separately for each quality assessment index and each sequence used for codec performance evaluation. that's the only way to hit the requirements from above 💖
for judging the quality of decoded output, two indexes get computed separately: PSNR [3] and MS-SSIM [3] [15]. with YCbCr, PSNR gets calculated for each color plane, but MS-SSIM only does the luma channel. with RGB, both metrics get computed for R, G, and B. so for each sequence in YCbCr you get 30 RD-points for PSNR (three RD-curves, one per channel) and 10 RD-points for MS-SSIM (one RD-curve, luma only). if the content is encoded as RGB, you get 60 RD-points total, 30 for PSNR and 30 for MS-SSIM (three RD-curves for each, one per channel) 🌈
finally, for the big-picture number, BD-rate savings [13] get computed for each range and each quality index. averages over all three ranges also need to be given for both PSNR and MS-SSIM 👑 the list of test video sequences, plus the ten QP values for the reference codec, are defined in [14]. testing has to use the codec application info in this document. as the reference, state-of-the-art codecs like HEVC/H.265 [4][5] or VP9 must be used. the reference source code for HEVC/H.265 lives at [6], and the HEVC/H.265 codec must be configured according to [16] and Table 9 🍓
+----------------------+--------------------------------------------+
| Intra-period, second | HEVC/H.265 encoding |
| | mode according to [16] |
+======================+============================================+
| AI | Intra Main or Intra |
| | Main10 |
+----------------------+--------------------------------------------+
| RA | Random access Main or |
| | Random access Main10 |
+----------------------+--------------------------------------------+
| FIZD | Low delay Main or |
| | Low delay Main10 |
+----------------------+--------------------------------------------+
Table 9: Intraperiods for Different HEVC/H.265 Encoding Modes
According to [16]okay so the coding efficiency requirement from Section 4.1.1 says the BD-rate savings, calculated for each color plane and averaged over all the video sequences used to test the NETVC codec, should be at least:
- 25% if calculated over the whole bitrate range; and
- 15% if calculated for each bitrate subrange (LBR, MBR, HBR).
since both objective metrics (PSNR and MS-SSIM) are available for some color planes, each value has to meet these requirements. so the final BD-rate saving, S, gets calculated for a given color plane like this:
S = min { S_psnr, S_ms-ssim }where S_psnr and S_ms-ssim are the BD-rate savings for that color plane using PSNR and MS-SSIM, respectively 🦋
and on top of all the objective stuff, subjective evaluation also must be done for the final NETVC codec adoption. for subjective tests, the MOS-based evaluation procedure must be used, as described in Section 2.1 of [3]. for perception-oriented tools that mainly affect subjective quality, extra tests may also be assigned individually, even for intermediate evaluation, but that's up to a decision of the NETVC WG 💜
6. Security Considerations 🔐
this document itself doesn't address any security considerations. but it's worth knowing that a codec implementation (encoder and decoder both) should think about the worst-case computational complexity, memory bandwidth, and physical memory size needed to process potentially untrusted input (like the decoded pictures used as references). not me getting worried about this, but it's serious stuff, bestie 😭
7. IANA Considerations 🌸
this document has no IANA actions. easy ✨
8. References 📚
8.1. Normative References
- [1] ITU-R, "Parameter values for ultra-high definition television systems for production and international programme exchange", ITU-R Recommendation BT.2020-2, October 2015, <https://www.itu.int/rec/R-REC-BT.2020-2-201510-I/en>.
- [2] ITU-T, "Quality of Experience requirements for telepresence services", ITU-T Recommendation G.1091, October 2014, <https://www.itu.int/rec/T-REC-G.1091/en>.
- [3] ISO, "Information technology -- Advanced image coding and evaluation -- Part 1: Guidelines for image coding system evaluation", ISO/IEC TR 29170-1:2017, October 2017, <https://www.iso.org/standard/63637.html>.
- [4] ISO, "Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 2: High efficiency video coding", ISO/IEC 23008-2:2015, May 2018, <https://www.iso.org/standard/67660.html>.
- [5] ITU-T, "High efficiency video coding", ITU-T Recommendation H.265, November 2019, <https://www.itu.int/rec/T-REC-H.265>.
- [6] Fraunhofer Institute for Telecommunications, "High Efficiency Video Coding (HEVC) reference software (HEVC Test Model also known as HM)", <https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/>.
8.2. Informative References 📚
okay so these are the "for context, go read this" sources 🤓 bestie, they're not required to implement the thing, but they're the receipts behind the design choices 💅
- [7] Federal Agencies Digital Guidelines Initiative, "Term: High dynamic range imaging",
http://www.digitizationguidelines.gov/term.php?term=highdynamicrangeimaging🌈 - [8] Federal Agencies Digital Guidelines Initiative, "Term: Compression, visually lossless",
http://www.digitizationguidelines.gov/term.php?term=compressionvisuallylossless🎀 - [9] Wenger, S., "The case for scalability support in version 1 of Future Video Coding", SG 16 (Study Period 2013) Contribution 988, September 2015,
https://www.itu.int/md/T13-SG16-C-0988/en✨ - [10] YouTube, "Recommended upload encoding settings",
https://support.google.com/youtube/answer/1722171?hl=en🦋 - [11] Yu, H., Ed., McCann, K., Ed., Cohen, R., Ed., and P. Amon, Ed., "Requirements for an extension of HEVC for coding of screen content", ISO/IEC JTC 1/SC 29/WG 11 Moving Picture Experts Group MPEG2013/N14174, San Jose, USA, January 2014,
https://mpeg.chiariglione.org/standards/mpeg-h/high-efficiency-video-coding/requirements-extension-hevc-coding-screen-content🖥️ - [12] Parhy, M., "Game streaming requirement for Future Video Coding", ISO/IEC JTC 1/SC 29/WG 11 Moving Picture Experts Group N36771, Warsaw, Poland, June 2015. 🎮
- [13] Bjontegaard, G., "Calculation of average PSNR differences between RD-curves", SG 16 VCEG-M33, April 2001,
https://www.itu.int/wftp3/av-arch/video-site/0104_Aus/📐 - [14] Daede, T., Norkin, A., and I. Brailovskiy, "Video Codec Testing and Quality Measurement", Work in Progress, Internet-Draft, draft-ietf-netvc-testing-09, 31 January 2020,
https://tools.ietf.org/html/draft-ietf-netvc-testing-09🧪 - [15] Wang, Z., Simoncelli, E.P., and A.C. Bovik, "Multiscale structural similarity for image quality assessment", IEEE Thirty-Seventh Asilomar Conference on Signals, Systems and Computers, DOI 10.1109/ACSSC.2003.1292216, November 2003,
https://ieeexplore.ieee.org/document/1292216🔍 - [16] Bossen, F., "Common HM test conditions and software reference configurations", Joint Collaborative Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (ITU-T Q.6/SG 16) and ISO/IEC Moving Picture Experts Group (ISO/IEC JTC 1/SC 29/WG 11) , Document JCTVC-L1100, April 2013,
http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=7281🧾 - [17] ITU-R, "Studio encoding parameters of digital television for standard 4:3 and wide screen 16:9 aspect ratios", ITU-R Recommendation BT.601, March 2011,
https://www.itu.int/rec/R-REC-BT.601/📺 - [18] ISO/IEC, "Information technology -- Coding of audio-visual objects -- Part 10: Advanced video coding", ISO/IEC DIS 14496-10,
https://www.iso.org/standard/75400.html🎬 - [19] ISO/IEC, "Information technology -- Coding of audio-visual objects -- Part 15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format", ISO/IEC 14496-15,
https://www.iso.org/standard/74429.html📦 - [20] ITU-R, "Parameter values for the HDTV standards for production and international programme exchange", ITU-R Recommendation BT.709, June 2015,
https://www.itu.int/rec/R-REC-BT.709🌟
Acknowledgments 🫶
the authors are so grateful and we're sending them all the love 💖 big thanks to Mr. Paul Coverdale, Mr. Vasily Rufitskiy, and Dr. Jianle Chen for so many useful chats about this doc and for helping while they were writing it 🙌 and also a huge shoutout to Mr. Mo Zanaty, Dr. Minhua Zhou, Dr. Ali Begen, Mr. Thomas Daede, Mr. Adam Roach, Dr. Thomas Davies, Mr. Jonathan Lennox, Dr. Timothy Terriberry, Mr. Peter Thatcher, Dr. Jean-Marc Valin, Mr. Roman Danyliw, Mr. Jack Moffitt, Mr. Greg Coppa, and Mr. Andrew Krupiczka for their valuable comments on the different revisions 👑
Authors' Addresses 💌
- Alexey Filippov, Huawei Technologies, Email:
alexey.filippov@huawei.com🧚 - Andrey Norkin, Netflix, Email:
anorkin@netflix.com🍓 - Jose Roberto Alvarez, Huawei Technologies, Email:
j.alvarez@ieee.org🌸
📦 where this page comes from
- ietf: rfc8761 2020-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms