RFC 7867

RTP Control Protocol (RTCP) Extended Report (XR) Block for Loss Concealment Metrics for Video Applications

📅 published July 2016, from ietf

RFC 7867: RTCP XR Loss Concealment Metrics for Video 🎀

This one is from July 2016 and was written by R. Huang at Huawei. It's a Standards Track document, which means it's an official Internet standard, and it's published under ISSN 2070-1721. The header doesn't list anything it obsoletes or updates, so it's a fresh addition. It defines a brand-new RTCP XR block so you can report how well a video receiver hides packet loss. Basically, it's a way to measure whether your video is secretly covering up glitches 👀✨

Abstract 💖

This document defines a new RTP Control Protocol (RTCP) Extended Report (XR) block. It lets you report loss concealment metrics for RTP video apps.

Status of This Memo 📌

This is an Internet Standards Track document, so it's officially part of the Internet's protocol family 💅

It's a product of the Internet Engineering Task Force (IETF) and reflects what the IETF community agreed on. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on what Internet Standards are, check out Section 2 of RFC 7841.

You can find the current status, any errata, and ways to send feedback at http://www.rfc-editor.org/info/rfc7867 🦋

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can read at http://trustee.ietf.org/license-info. These rules are the ones in effect on the publication date, so read them carefully because they spell out your rights and restrictions 📜

If you pull Code Components out of this document, they have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They also come with no warranty, as described in the Simplified BSD License 🧚

Table of Contents 🍓

1. Introduction ....................................................2
   1.1. RTCP and RTCP XR Reports ...................................3
   1.2. Performance Metrics Framework ..............................3
   1.3. Applicability ..............................................3
2. Terminology .....................................................3
3. Video Loss Concealment Methods ..................................3
4. Video Loss Concealment Report Block .............................4
5. SDP Signaling ...................................................8
   5.1. SDP rtcp-xr-attrib Attribute Extension .....................8
   5.2. Offer/Answer Usage .........................................9
6. Security Considerations .........................................9
7. IANA Considerations .............................................9
   7.1. New RTCP XR Block Type Value ...............................9
   7.2. New RTCP XR SDP Parameter ..................................9
   7.3. Contact Information for Registrations .....................10
8. References .....................................................10
   8.1. Normative References ......................................10
   8.2. Informative References ....................................11
Appendix A. Metrics Represented Using the Template from RFC 6390 ..12
Acknowledgements ..................................................16
Authors' Addresses ................................................16

1. Introduction ✨

Okay so multimedia apps lowkey always deal with packet loss on IP networks 😭 To keep quality decent when packets go missing, the decoder needs loss concealment mechanisms. Video loss concealment is basically a bunch of tricks for hiding the damage that packet loss does to a video call or stream 🎀

Here's why this matters: in some apps, receivers that use video loss concealment can tell monitors or senders a lot about the Quality of Experience (QoE). A big example is no-reference video quality evaluation. The thing is, video probes sitting upstream from the endpoint or terminal might not see the loss that happens between the probe and the endpoint. They also might not know exactly which concealment methods the endpoint is applying on the fly. So evaluating error concealment is important here, because it helps estimate how much the impairments actually hurt the viewer's experience 👀💅

This document defines one new block type for video loss concealment. It sits alongside the ones in [RFC3611] and [RFC7294] and is meant for a range of RTP video apps 🦋 The metrics here belong to the class of transport-related terminal metrics defined in [RFC6792] 💜

1.1. RTCP and RTCP XR Reports 🌸

RTCP reporting is defined in [RFC3550]. [RFC3611] then gives you an extensible structure for reporting with an RTCP Extended Report (XR). This document adds a new Extended Report block, and it's used the way [RFC3550] and [RFC3611] describe 🧚

1.2. Performance Metrics Framework 🍓

The Performance Metrics Framework in [RFC6390] gives guidance on defining and specifying performance metrics, so it's the rulebook for how to write them. The RTP monitoring framework in [RFC6792] goes further and gives guidelines for the reporting block format when you use RTCP XR. The XR block type in this document follows the guidelines in both [RFC6390] and [RFC6792], so it's giving very organized 🌈

1.3. Applicability 🎀

These metrics are for video apps, meaning the video part of audio/video apps that use RTP. They work when the receiving endpoint runs packet loss concealment to soften the blow that network problems have on QoE ✨

Picture an IPTV system. Set-top boxes could use this RTCP XR block to report loss and loss concealment metrics to an IPTV management system. That way the service provider can keep an eye on the quality of IPTV service that end users are actually getting 📺💖

2. Terminology 💜

Here's the official key word list. These words are interpreted as described in [RFC2119], which is the BCP 14 reference:

"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🔥

3. Video Loss Concealment Methods 🦋

Video loss concealment mechanisms fall into 4 types. Let's go through them 👑

  • a) Frame freeze

The damaged video frame doesn't get displayed. The last frame that was displayed just stays frozen for the whole length of the loss event 🥶

  • b) Interframe extrapolation

If loss damages part of a video frame, the same area from the previous frame(s) can be used to guess what the missing pixels should have been. This works pretty well when nothing is moving in the scene, but it's super noticeable when there's a lot of movement between frames 😬 Simple decoders just reuse the pixels from the missing area. More complex decoders can use several frames to do a fancier extrapolation. A more sophisticated version of interframe repair estimates how the damaged region is moving, based on the motion around it. It then uses that motion to pick which part of the previous frame to use for the repair 🧠

Some important frames, like Instantaneous Decoding Refresh (IDR) frames, don't depend on any other frames and can show up with a scene change. Using interframe extrapolation to hide the loss of these frames might not look good at all 💀

  • c) Interpolation

The decoder uses the undamaged pixels in the frame to estimate what the missing block of pixels should have looked like 🌷

  • d) Error-resilient encoding

The sender encodes the message redundantly, so the receiver can fix errors using that extra information 🛡️ There are usually two kinds of error-resilient encoding. One kind embeds redundant data for error resiliency into the compressed image/video bitstream, which the decoder uses. The other kind works at the bitstream level, and Forward Error Correction (FEC) is an example of that 🫶

Methods b, c, and d usually get deployed together to give complex decoders full loss concealment. Method a is pretty independent, and simple decoders can use it on its own 🍓

Here's the tricky bit: frame freeze repairs video based on whole frames, but the other methods repair video based on finer-grained stuff like macroblocks or bitstreams. Because of that, the measurements for frame freeze and the other methods come out slightly different. So in this document, we split frame freeze apart from the other 3 loss concealment mechanisms 💅✨

4. Video Loss Concealment Report Block 🎀

okay bestie, this block is where video loss concealment metrics finally get their spotlight 🎬✨ it's the video sibling of the audio metrics from [RFC7294] 💖 audio's been hogging the stage for too long, fr.

The report block MUST be sent together with the info from the Measurement Information Block [RFC6776] 🫶. Instances of this metric block point to a source by synchronization source (SSRC), which links to that separate auxiliary Measurement Information Block [RFC6776]. This block relies on the measurement period in that Measurement Information Block to know the span of the report 🕰️. If the measurement period doesn't arrive in the same compound RTCP packet as this metric block, this metric block MUST be discarded at the receiving side 💀 no exceptions, periodt. The metrics here are usually measured at the moment a video frame gets decoded and rendered for playout 🎞️.

Here's the format of the Video Loss Concealment Report Block 👀

    0               1               2               3
    0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    BT=34      | I | V |  RSV  |       Block Length            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         SSRC of Source                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Impaired Duration                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Concealed Duration                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  Mean Frame Freeze Duration (optional)        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    MIFP       |    MCFP       |     FFSC      |     Reserved  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 1: Format for the Video Loss Concealment Report Block

Block Type (BT): 8 bits 🦋

A Video Loss Concealment Report Block is identified by the constant 34. That's its ID card 🪪

Interval Metric Flag (I): 2 bits 📏

This field tells you whether the reported metrics are interval, cumulative, or sampled metrics [RFC6792] 🧠

  • I=10: Interval Duration - the reported value applies to the most recent measurement interval duration between successive metrics reports.
  • I=11: Cumulative Duration - the reported value applies to the accumulation period characteristic of cumulative measurements.
  • I=01: Sampled Value - this value MUST NOT be used for this block type. absolutely not, bestie 🙅‍♀️
  • I=00: Reserved.

Video Loss Concealment Method Type (V): 2 bits 🎭

This field identifies the video loss concealment method type used at the receiver. Here's the lineup:

  • V=10: Frame-freeze
  • V=11: Other Loss Concealment Method
  • V=01 and V=00: Reserved

If frame-freeze and another loss concealment method are used together for the media stream, two report blocks (one with V=10 for frame freeze and one with V=11 for the other loss concealment method) SHOULD be compounded together to report complete concealment information. the full picture, no cropping 📸✨

RSV: 4 bits 🫧

These bits are reserved for future use. They MUST be set to zero by senders and ignored by receivers (see Section 4.2 of [RFC6709]).

Block Length: 16 bits 📐

This field follows the definition in [RFC3611]. In this report block, it MUST be set to 5 when V=10 and set to 4 when V=11. The block MUST be discarded if the block length is set to a different value. no vibes, no exceptions 🚫

SSRC of Source: 32 bits 🎙️

As defined in Section 4.1 of [RFC3611].

Impaired Duration: 32 bits 🌧️

This is the total duration, measured in RTP timestamp units from the sending side of the reporting block, of video that got impaired by transmission loss before any loss concealment methods were applied.

Two values are reserved: 0xFFFFFFFE means out of range (aka a measured value bigger than 0xFFFFFFFD), and 0xFFFFFFFF means the measurement is unavailable 🙈

Concealed Duration: 32 bits 🫥

This is the total duration, measured in RTP timestamp units from the sending side of the reporting block, of concealed damaged video pictures where the loss concealment method matching the Video Loss Concealment Method Type got applied.

Two values are reserved: 0xFFFFFFFE means out of range (aka a measured value bigger than 0xFFFFFFFD), and 0xFFFFFFFF means the measurement is unavailable 🙈

Mean Frame-Freeze Duration: 32 bits ❄️

Mean Frame-Freeze Duration is the average duration, in RTP timestamp units from the sending side of the reporting block, of the frame-freeze events. You get it by adding up the total duration of all the frame freeze events and dividing by how many events there were 🧮 This metric is optional. It only exists when Video Loss Concealment Method Type=10.

Mean Impaired Frame Proportion (MIFP): 8 bits 🧩

Mean Impaired Frame Proportion is the average proportion of each video frame that got impaired by loss before any loss concealment method was applied during the interval. It's written as a fixed-point number with the binary point at the left edge of the field 🤓

Here's how it works: add up the impaired proportion of each video frame and divide by the number of frames in this period. Each frame's impaired proportion comes from dividing the number of missing macroblocks in that frame by the frame's total macroblock number. That's equal to multiplying the result by 256, capping the max value at 255 (so nothing overflows), and keeping the integer part 💅

If a video frame is totally lost, a value of 0xFF SHOULD be used for that frame when calculating the MIFP. it's giving total wipeout 💀

Mean Concealed Frame Proportion (MCFP): 8 bits 🫧

Mean Concealed Frame Proportion is the average proportion of each video frame that had loss concealment applied to it (shown as "V" in the definition of "Video Loss Concealment Method Type") during the interval. It's also a fixed-point number with the binary point at the left edge of the field 🎯

To calculate it, add up the concealed proportion of each video frame and divide by the number of frames in this period. Each frame's concealed proportion comes from dividing the number of concealed macroblocks in that frame by the frame's total macroblock number. That's equal to multiplying the result by 256, capping the max value at 255 (to avoid overflow), and taking the integer part ✨

When calculating the MCFP, a value of 0xFF SHOULD be used for a lost frame that is totally concealed, and a value of 0 SHOULD be used for a frame if there are no concealed macroblocks in it. For Video Loss Concealment Method Type=10, each frame covered in the period of frame freeze counts as totally concealed, so a value of 0xFF MUST be assigned. frozen frames are fully concealed, no debate 🧊👑

Fraction of Frames Subject to Concealment (FFSC): 8 bits 🍰

Fraction of Frames Subject to Concealment is calculated by dividing the number of frames where loss concealment (using the Video Loss Concealment Method Type) got applied by the total number of frames. The result is expressed as a fixed-point number with the binary point at the left edge of the field. It's equal to multiplying the division result by 256, capping the max value at 255 (to avoid overflow), and taking the integer part 🧁

A value of 0 means there were no concealed frames. A value of 0xFF means every frame in the whole measurement interval got concealed 😭

Reserved: 8 bits 🫧

These bits are reserved for future use. They MUST be set to zero by senders and ignored by receivers (see Section 4.2 of [RFC6709]).

5. SDP Signaling 📜

[RFC3611] defines how to use the Session Description Protocol (SDP) to signal the use of RTCP XR blocks 💌

5.1. SDP rtcp-xr-attrib Attribute Extension 🪄

This section extends the SDP attribute "rtcp-xr" defined in Section 5.1 of [RFC3611] by adding a new xr-format value. That value signals the use of the report block defined in this document. Here's the ABNF [RFC5234] syntax:

xr-format =/ xr-vlc-block

xr-vlc-block = "vlc"

5.2. Offer/Answer Usage 🤝

When SDP is used in an offer/answer context, the SDP Offer/Answer usage defined in Section 5.2 of [RFC3611] for the unilateral "rtcp-xr" attribute parameters applies. For detailed usage of Offer/Answer for unilateral parameters, check Section 5.2 of [RFC3611] 📖

6. Security Considerations 🔐😭

It's believed that this RTCP XR block introduces no new security considerations beyond those described in [RFC3611]. Since this block doesn't provide per-packet statistics, the risk to confidentiality documented in paragraph 3 of Section 7 of [RFC3611] doesn't apply 🫶

BUT the next part is serious, bestie 🚨

An attacker is likely to put incorrect information in the Video Loss Concealment reports. That messes with the estimation of how well video loss concealment mechanisms are performing, and it can wreck the QoE of users 💀 Implementers SHOULD consider the guidance in [RFC7202] for using appropriate security mechanisms. In other words, where security is a concern, the implementation SHOULD apply encryption and authentication to the report block 🔒

For example, you can do this by using the AVPF profile together with the Secure RTP profile defined in [RFC3711]. An appropriate combination of the two profiles (an "SAVPF") is specified in [RFC5124]. Other mechanisms also exist (documented in [RFC7201]) and might be a better fit 🧚

7. IANA Considerations 📝

New block types for RTCP XR are subject to IANA registration. For general guidelines on IANA considerations for RTCP XR, check [RFC3611] 🗂️

7.1. New RTCP XR Block Type Value 🦋

This document assigns the block type value 34 to the Video Loss Concealment Metric Report Block in the IANA "RTP Control Protocol Extended Reports (RTCP XR) Block Type Registry" 🎀

7.2. New RTCP XR SDP Parameter 🌸

This document also registers a new parameter "video-loss-concealment" in the "RTP Control Protocol Extended Reports (RTCP XR) Session Description Protocol (SDP) Parameters Registry" 💅

7.3. Contact Information for Registrations 📬

The contact information for the registration is:

RAI Area Directors <rai-ads@ietf.org>

8. References ✨

8.1. Normative References 📚

These are the must-read docs, bestie 💅 Here's the full list:

8.2. Informative References 🌸

These ones are extra reading, so go deep if you're curious 🤓

Appendix A. Metrics Represented Using the Template from RFC 6390 🦋

Okay, this appendix is where the metrics get their official RFC 6390 makeover ✨ Each one gets the full template, so let's go!

a. Video Impaired Duration Metric 🎬

  • Metric Name: Video Impaired Duration Metric
  • Metric Description: Basically, how long the video was messed up by transmission loss, counted in total, before any loss concealment kicks in. 😬
  • Method of Measurement or Calculation: This is usually measured right when a video frame gets decoded and rendered for playout. 🎞️
  • Units of Measurement: This metric is expressed in units of RTP timestamp. ⏱️
  • Measurement Point(s) with Potential Measurement Domain: It's measured at the receiving end of the RTP stream. 📥
  • Measurement Timing: See paragraph 1 of Section 4. 👉
  • Use and Applications: This one fits video applications of RTP, plus the video part of audio/video apps, where packet loss concealment runs on the receiving endpoint to soften how network problems hit QoE. 💖

b. Video Concealed Duration Metric 🫥

  • Metric Name: Video Concealed Duration Metric
  • Metric Description: The total duration of damaged video pictures that got concealed, using the loss concealment method that matches Video Loss Concealment Method Type. 🩹
  • Method of Measurement or Calculation: Same vibe as before: it's based on measurements usually taken when a video frame is decoded and rendered for playout. 🎞️
  • Units of Measurement: This metric is expressed in units of RTP timestamp. ⏱️
  • Measurement Point(s) with Potential Measurement Domain: It's measured at the receiving end of the RTP stream. 📥
  • Measurement Timing: See paragraph 1 of Section 4. 👉
  • Use and Applications: These metrics fit video applications of RTP and the video component of audio/video apps, as long as packet loss concealment is built into the receiving endpoint to help with network impairments and QoE. 🌈

c. Mean Video Frame-Freeze Duration Metric 🧊

  • Metric Name: Mean Video Frame-Freeze Duration Metric
  • Metric Description: This is the average duration of the frame-freeze events. ⏸️
  • Method of Measurement or Calculation: It's based on measurements usually taken when a video frame is decoded and rendered for playout. To get the mean, you add up the total duration of all the frame-freeze events and divide by how many events there were. 🧮
  • Units of Measurement: This metric is expressed in units of RTP timestamp. ⏱️
  • Measurement Point(s) with Potential Measurement Domain: It's measured at the receiving end of the RTP stream. 📥
  • Measurement Timing: See paragraph 1 of Section 4. 👉
  • Use and Applications: Same squad as the others: video applications of RTP and the video part of audio/video apps, where packet loss concealment is built into the receiving endpoint to lessen network impact on QoE. 💜

d. Mean Impaired Video Frame Proportion Metric 🍓

  • Metric Name: Mean Impaired Video Frame Proportion Metric
  • Metric Description: This is the average proportion of each video frame that got impaired by loss, measured before any loss concealment method is applied, across the interval. 📉
  • Method of Measurement or Calculation: It's based on measurements usually taken when a video frame is decoded and rendered for playout. You add up the impaired proportion of each frame and divide by the number of frames in that period. Each frame's impaired proportion comes from dividing its missing macroblocks by its total macroblock count. That's the same as multiplying the result by 256, capping the max at 255 so it doesn't overflow, and keeping only the integer part. 🧮✨
  • Units of Measurement: This metric is expressed as a fixed-point number with the binary point at the left edge of the field. 🔢
  • Measurement Point(s) with Potential Measurement Domain: It's measured at the receiving end of the RTP stream. 📥
  • Measurement Timing: See paragraph 1 of Section 4. 👉
  • Use and Applications: These fit video applications of RTP and the video component of audio/video apps, where packet loss concealment is built into the receiving endpoint to reduce how network impairments hit QoE. 🌸

e. Mean Concealed Video Frame Proportion Metric 🩵

  • Metric Name: Mean Concealed Video Frame Proportion Metric
  • Metric Description: This is the average proportion of each video frame where loss concealment (using Video Loss Concealment Method Type) was applied, across the interval. 🫧
  • Method of Measurement or Calculation: It's based on measurements usually taken when a video frame is decoded and rendered for playout. You add up the concealed proportion of each frame and divide by the number of frames in that period. Each frame's concealed proportion comes from dividing its concealed macroblocks by its total macroblock count. That's the same as multiplying the result by 256, capping the max at 255 to avoid overflow, and taking the integer part. 🧮
  • Units of Measurement: This metric is expressed as a fixed-point number with the binary point at the left edge of the field. 🔢
  • Measurement Point(s) with Potential Measurement Domain: It's measured at the receiving end of the RTP stream. 📥
  • Measurement Timing: See paragraph 1 of Section 4. 👉
  • Use and Applications: These fit video applications of RTP and the video component of audio/video apps, as long as packet loss concealment is built into the receiving endpoint to help with network impairments and QoE. 🦋

f. Fraction of Video Frames Subject to Concealment Metric 🌷

  • Metric Name: Fraction of Video Frames Subject to Concealment Metric
  • Metric Description: This is the proportion of video frames where loss concealment (using the Video Loss Concealment Method Type) was applied, compared to the total number of frames during the interval. 🎯
  • Method of Measurement or Calculation: It's based on measurements usually taken when a video frame is decoded and rendered for playout. You get it by dividing the number of frames where loss concealment was applied (using Video Loss Concealment Method Type) by the total number of frames. That's equivalent to multiplying the result by 256, capping the max at 255 to avoid overflow, and taking the integer part. 🧁
  • Units of Measurement: This metric is expressed as a fixed-point number with the binary point at the left edge of the field. 🔢
  • Measurement Point(s) with Potential Measurement Domain: It's measured at the receiving end of the RTP stream. 📥
  • Measurement Timing: See paragraph 1 of Section 4. 👉
  • Use and Applications: These fit video applications of RTP and the video component of audio/video apps, where packet loss concealment is built into the receiving endpoint to help with network impairments and QoE. 💅

Acknowledgements 🫶

The author would like to thank Colin Perkins and Roni Even for their valuable comments. You two are seriously iconic 👑

Authors' Addresses 🎀

Rachel Huang
Huawei
101 Software Avenue, Yuhua District
Nanjing 210012
China

Email: rachel.huang@huawei.com

That's a wrap on RFC 7867! 🌈✨ Thanks for sticking with me, bestie 💖

📦 where this page comes from