RFC 8451

Considerations for Selecting RTP Control Protocol (RTCP) Extended Report (XR) Metrics for the WebRTC Statistics API

📅 published September 2018, from ietf

RFC 8451 is giving Considerations for Selecting RTP Control Protocol (RTCP) Extended Report (XR) Metrics for the WebRTC Statistics API 🎀 It's from the Internet Engineering Task Force (IETF) stream, dated September 2018, and its category is Informational. The ISSN is 2070-1721 ✨

The authors are V. Singh (callstats.io), R. Huang, R. Even (Huawei), D. Romascanu (Individual), and L. Deng (China Mobile) 💖 The title block doesn't list anything it obsoletes or updates, so there's no drama there 🫶

Abstract ✨

okay so this doc is all about monitoring media streams in WebRTC 🎥 It lists the RTCP Sender Report (SR), Receiver Report (RR), and Extended Report (XR) metrics that RTP implementations might need to support in some diverse environments 🌈 It also gives a set of identifiers for the WebRTC statistics API. Those identifiers are RTCP SR, RR, and XR metrics about how multimedia flows get transported 💜

Status of This Memo 🧚

This doc is not an Internet Standards Track specification. It's published for informational purposes, so no drama 💅

It's a product of the IETF, it represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🔥 Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you're curious 🤓

For info on the current status, any errata, and how to give feedback, go to https://www.rfc-editor.org/info/rfc8451 🦋

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License 🙈

Table of Contents 📑

1. Introduction ....................................................4
2. Terminology .....................................................4
3. RTP Statistics in WebRTC Implementations ........................5
4. Considerations for Impact of Measurement Interval ...............5
5. Candidate Metrics ...............................................6
   5.1. Network Impact Metrics .....................................6
        5.1.1. Loss and Discard Packet Count Metric ................6
        5.1.2. Burst/Gap Pattern Metrics for Loss and Discard ......7
        5.1.3. Run-Length Encoded Metrics for Loss and Discard .....8
   5.2. Application Impact Metrics .................................8
        5.2.1. Discarded Octets Metric .............................8
        5.2.2. Frame Impairment Summary Metrics ....................9
        5.2.3. Jitter Buffer Metrics ...............................9
   5.3. Recovery Metrics ..........................................10
        5.3.1. Post-Repair Packet Count Metrics ...................10
        5.3.2. Run-Length Encoded Metric for Post-Repair ..........10
6. Identifiers from Sender, Receiver, and Extended Report Blocks ..11
   6.1. Cumulative Number of Packets and Octets Sent ..............11
   6.2. Cumulative Number of Packets and Octets Received ..........11
   6.3. Cumulative Number of Packets Lost .........................11
   6.4. Interval Packet Loss and Jitter ...........................12
   6.5. Cumulative Number of Packets and Octets Discarded .........12
   6.6. Cumulative Number of Packets Repaired .....................12
   6.7. Burst Packet Loss and Burst Discards ......................12
   6.8. Burst/Gap Rates ...........................................13
   6.9. Frame Impairment Metrics ..................................13
7. Adding New Metrics to WebRTC Statistics API ....................13
8. Security Considerations ........................................14
9. IANA Considerations ............................................14
10. References ....................................................14
   10.1. Normative References .....................................14
   10.2. Informative References ...................................16
Acknowledgements ..................................................17
Authors' Addresses ................................................18

1. Introduction ✨

okay bestie, Web real-time communication (WebRTC) [WebRTC-Overview] deployments are popping off, and applications need to be able to estimate service quality 📈 If enough information (metrics or statistics) gets handed to the application, it can try to improve the media quality 💖 [RFC7478] lays down a requirement for statistics, and it goes like this:

F38   The browser must be able to collect statistics, related to the
      transport of audio and video between peers, needed to estimate
      quality of experience.

The WebRTC Stats API [W3C.webrtc-stats] currently lists metrics reported in the RTCP Sender Report and Receiver Report (SR/RR) [RFC3550] to fulfill this requirement. Lowkey though, the basic metrics from RTCP SR/RR aren't enough for precise quality monitoring or diagnosing potential issues 🥺

Standards such as "RTP Control Protocol Extended Reports (RTCP XR)" [RFC3611] are giving us more, along with other extensions standardized in the XRBLOCK Working Group. For example, there's burst/gap loss metric reporting [RFC6958] and burst/gap discard metric reporting [RFC7003] 🤩 These were made to collect and report performance metrics from RTP endpoint devices, so you can get end-to-end service visibility and measure delivery quality across various RTP services. These metrics are able to complement those in [RFC3550] 🫶

In this document, we give the rationale for picking additional RTP metrics for the WebRTC getStats() API [W3C.webrtc] ✨ All identifiers proposed here are recommended to be implemented by a WebRTC endpoint. An endpoint may choose not to expose an identifier if it doesn't implement the corresponding RTCP Report. This document only considers RTP-layer metrics. Other metrics, like IP-layer metrics, are out of scope 🙈

2. Terminology 🎀

In addition to the terminology from [RFC3550], [RFC3611], and [RFC7478], this document uses the following term:

ReportGroup: a set of metrics identified by a common synchronization source (SSRC) 🌸

3. RTP Statistics in WebRTC Implementations 💅

The RTCP Sender Reports (SRs) and Receiver Reports (RRs) [RFC3550] expose the basic metrics for the local and remote media streams. But these metrics only give partial or limited information, which might not be enough for diagnosing problems or monitoring quality 😭 For example, it can be useful to tell apart packets that were lost from packets discarded because they arrived late. Even though both hurt multimedia quality the same way, knowing which one happened helps you figure out what's broken 🔍 RTP Control Protocol Extended Reports (XRs) [RFC3611] and other extensions discussed in the XRBLOCK Working Group give more detailed statistics, which complement the basic metrics in the RTCP SR and RRs 💜

The WebRTC application pulls statistics out of the browser by querying the getStats() API [W3C.webrtc]. The browser can easily report the local variables, meaning the statistics for the outgoing and incoming RTP media streams 💁‍♀️ However, without RTCP XRs or some other signaling mechanism, the WebRTC application can't expose the remote endpoints' statistics 🥹 [WebRTC-RTP-USAGE] does not mandate the use of any RTCP XRs, so their usage is optional. If RTCP XRs get successfully negotiated between endpoints (via SDP), then the application has access to both local and remote statistics ✨ Another route: once the WebRTC application has the local information, it can report it to an application server or a third-party monitoring system, which gives quality estimates or diagnostic services to application developers 🧚 The exchange of statistics between endpoints, or between a monitoring server and an endpoint, is outside the scope of this document 🌈

4. Considerations for Impact of Measurement Interval ⏰

RTCP extensions like RTCP XR usually share the same timing interval with the RTCP SR/RR. That means they get sent as compound packets, together with the RTCP SR/RR 🫶 Alternatively, if the RTCP XR uses a different measurement interval, all XRs using that same measurement interval get compounded together, and the measurement interval is indicated in a specific measurement information block defined in [RFC6776] 🍓

When using WebRTC getStats() APIs (see "Statistics Model" in [W3C.webrtc]), applications can query this information at arbitrary intervals 🔁 For statistics reported by the remote endpoint, like the ones conveyed in an RTCP SR/RR/XR, these won't change until the next RTCP report is received 📭 Local statistics are a different story. They have no such restriction as long as the endpoint is sending and receiving media 🎙️ For example, an application might poll the stack for statistics every 1 second. In that case, the underlying local stack returns the current snapshot of the local statistics (for incoming and outgoing media streams) ✅ But it may hand back the same remote statistics as before, because no new RTCP reports may have arrived in that past 1 second 🙃 This can happen when the polling interval is shorter than the average RTCP reporting interval 💀

5. Candidate Metrics 👑

Since the following metrics are all defined in RTCP XR, which is not mandated in WebRTC, all of them are local. Plot twist though: if RTCP XR is supported by negotiation between two browsers, the following metrics can also be generated remotely and sent to the local endpoint (the one that generated the media) via RTCP XR packets 🦋

The metrics are sorted into 3 categories: network impact metrics, application impact metrics, and recovery metrics 🌷 Network impact metrics record information only about network transmission, which makes them useful for network problem diagnosis 🔌 Application impact metrics mainly collect information from the application's point of view, like bit rate, frame rate, or jitter buffers 🎬 Recovery metrics reflect how well the repair mechanisms perform, like loss concealment, retransmission, or Forward Error Correction (FEC) 🛟 All 3 types are useful for estimating the quality of services in WebRTC implementations. WebRTC applications can use these metrics to calculate estimated Mean Opinion Score (MOS) [ITU-T_P.800.1] values or Media Delivery Index (MDI) [RFC4445] for their services 🎀

5.1. Network Impact Metrics 🌐

5.1.1. Loss and Discard Packet Count Metric

okay so, in multimedia transport, packets that arrive badly get sorted into 3 types: lost, discarded, and duplicate packets 🙈 Packet loss can come from a network device breaking, bit-error corruption, or network congestion (aka packets getting dropped by an intermediate router queue). Duplicate packets can happen when network delays make the sender retransmit the original packets. Discarded packets are the ones that got delayed for so long (maybe they missed the playout time) that the receiver says "nah, useless" 💀 Lost and discarded packets are a big problem for multimedia services. Missing data and long delays can tank the service quality. For example, a big block of contiguous packets that are lost or discarded can make the audio choppy, and really long network transmission delays can make audio or video buffer 😭

The RTCP SR/RR defines a metric that counts the total number of RTP data packets lost since the beginning of reception. BUT (sorry, I had to) this statistic doesn't tell lost packets apart from discarded and duplicate ones. Packets that show up late get discarded and are not reported as lost, and duplicate packets get treated like a normally received packet 🫠 So the loss metric can be super misleading when lots of duplicates arrive or packets get discarded. The media transport can look totally fine on paper while the users are actually having a bad time 💔 In those cases it is better to use more accurate metrics on top of the ones defined in RTCP SR/RR ✨

The metrics for lost and duplicated packets in the Statistics Summary Report Block of [RFC3611] extend the loss info carried in standard RTCP SR/RR. They explicitly account for lost and duplicated packets. Lost packet counts are really useful for diagnosing network problems 🔍 It is better to use the packet loss metrics of [RFC3611] for the lost packet count, instead of the cumulative number of packets lost metric of [RFC3550]. Duplicated packets are usually rare and have little effect on QoS evaluation, so they may not be suitable for use in WebRTC 🌸

Using loss metrics without discard metrics can lead to inaccurate quality evaluation, because packet discard due to jitter is often more common than packet loss in modern IP networks 👀 The discarded metric specified in [RFC7002] counts the number of packets discarded due to jitter. It augments the loss statistics from standard RTCP SR/RR. For WebRTC services with jitter buffers that need precise quality evaluation and accurate troubleshooting, this metric is useful as a complement to the RTCP SR/RR metrics 🎀

5.1.2. Burst/Gap Pattern Metrics for Loss and Discard

Okay, the RTCP SR/RR loss stats are kinda coarse 🥺 They're all per-call stats, and they aren't detailed enough to catch the moody, on-and-off nature of impairments like bursty packet loss. Even when the average packet loss rate is low, the lost packets can bunch up in short dense periods, and that causes short periods of degraded quality. Bursts hurt the experience more than non-bursts at low packet loss rates, but at high packet loss rates it's the other way around 🙃 So capturing burst gap information is super helpful for quality evaluation and for locating impairments. If the WebRTC application needs to evaluate service quality, burst gap metrics give more accurate info than RTCP SR/RR 💅

[RFC3611] introduces burst gap metrics in the VoIP Report Block. These metrics record the density and duration of burst and gap periods, which helps isolate network problems. Bursts line up with periods where the packet loss/discard rate is high enough to cause noticeable degradation in audio or video quality 🔥 Metrics related to burst gaps are also introduced in [RFC7003] and [RFC6958], which define two new report blocks for use across a range of RTP applications beyond those in [RFC3611]. These metrics tell discarded packets apart from loss packets that occur in the burst period, and they give more info for diagnosing network problems 🧚 The block also reports how often burst events happen, which is useful for evaluating quality of experience ✨ So if WebRTC applications need to do quality evaluation and see when and why quality degrades, these metrics should be considered 🦋

5.1.3. Run-Length Encoded Metrics for Loss and Discard

Run-length encoding uses a bit vector to encode info about packets 🌈 Each bit in the vector represents a packet. Depending on the signaled metric, the bit says whether the packet was lost, duplicated, discarded, or repaired. An endpoint typically uses run-length encoding to accurately tell the other endpoint the status of each packet in the interval 💜 [RFC3611] and [RFC7097] define run-length encoding for lost and duplicate packets, and for discarded packets, respectively.

The WebRTC application could benefit from this extra info 🤩 If losses happen after discards, an endpoint may be able to correlate the two run-length vectors to spot congestion-related losses. For example, a router queue got overloaded, causing delays, and then overflowed. If the losses are independent, that may indicate bit-error corruption 👀 For the WebRTC Stats API [W3C.webrtc-stats], these types of metrics are not recommended for use, due to the large amount of data and the computation involved 💀

5.2. Application Impact Metrics

5.2.1. Discarded Octets Metric

This metric reports the cumulative size of the packets discarded in the interval 📦 It is complementary to the number of discarded packets. An application measures sent octets and received octets to calculate the sending rate and the receiving rate, respectively. The application can then work out the actual bit rate in a particular interval by subtracting the discarded octets from the received octets 🧮

For WebRTC, the discarded octets metric supplements the metrics on sent and received octets. It gives an accurate method for calculating the actual bit rate, which is an important parameter for reflecting the quality of the media 💖 The Bytes Discarded metric is defined in [RFC7243] 🎀

5.2.2. Frame Impairment Summary Metrics

RTP has different framing mechanisms for different payload types 🍓 For audio streams, a single RTP packet may contain one or multiple audio frames. In video streams, though, a single video frame may be sent across multiple RTP packets. The size of each packet is limited by the Maximum Transmission Unit (MTU) of the underlying network 📏 The statistics from standard SR/RR only collect info from the transport layer, so they may not fully reflect the quality the application actually observes 🙈

Video is typically encoded using two frame types: key frames and derived frames. Key frames are normally just spatially compressed, meaning they don't predict from other pictures. Derived frames are temporally compressed, meaning they depend on the key frame for decoding 🎞️ So key frames are way bigger than derived frames, and losing one of them causes a substantial drop in video quality 😭 Thus, it is reasonable to consider this application-layer info in WebRTC implementations. It can influence sender strategies to mitigate the problem, or it can be needed to accurately assess users' quality of experience 👑

The metrics in this category include: number of discarded key frames, number of lost key frames, number of discarded derived frames, and number of lost derived frames 📊 These metrics can be used to calculate the Media Loss Rate (MLR) of the MDI [RFC4445]. Details of how these metrics are defined are described in [RFC7003]. The metric also provides the rendered frame rate, which is an important parameter for quality estimation 🎬

5.2.3. Jitter Buffer Metrics

The size of the jitter buffer affects both the end-to-end delay on the network and the packet discard rate 🧃 When the buffer is too small, late-arriving packets don't get played out and get dropped. When the buffer is too large, packets are held longer than necessary, which reduces conversational quality 😤 Measurement of the jitter buffer should not be ignored when evaluating end-user perception of conversational quality. Metrics related to the jitter buffer, such as maximum and nominal jitter buffer, could be used to show how the jitter buffer behaves at the receiving endpoint 🫧 They're useful for providing better end-user quality of experience (QoE) when jitter buffer factors are used as inputs to calculate estimated MOS values 💜 So for those cases, jitter buffer metrics should be considered. The definition of these metrics is provided in [RFC7005] 🌸

5.3. Recovery Metrics

This document does not consider concealment metrics [RFC7294] as part of recovery metrics 🙅‍♀️

5.3.1. Post-Repair Packet Count Metrics

Web applications can support certain RTP error-resilience mechanisms by following the recommendations in [WebRTC-RTP-USAGE] 🛡️ For these web applications using repair mechanisms, providing some statistics about how well their repair mechanisms perform could help give a more accurate quality evaluation ✨

The unrepaired packet count and repaired loss count defined in [RFC7509] give recovery info about the error-resilience mechanisms to the monitoring application or the sending endpoint 📈 The endpoint can use these metrics to figure out the ratio of repaired packets to lost packets. Including post-repair packet count metrics helps the application evaluate how effective the applied repair mechanisms are 🫶

5.3.2. Run-Length Encoded Metric for Post-Repair

[RFC5725] defines run-length encoding for post-repair packets 🧩 When using error-resilience mechanisms, the endpoint can correlate the loss run length with this metric to figure out where the losses and repairs happened in the interval. This gives more accurate info for evaluating recovery mechanisms than the metrics in Section 5.3.1. However, when RTCP XR metrics are supported, using run-length encoded metrics is not suggested, because the per-packet info yields an enormous amount of data that isn't required in this case 🫠

For WebRTC, the application may benefit from the extra info 💫 If losses happen after discards, an endpoint may be able to correlate the two run-length vectors to identify congestion-related losses, like a router queue that got overloaded, caused delays, and then overflowed. If the losses are independent, that may indicate bit-error corruption. Lastly, when using error-resilience mechanisms, the endpoint can correlate the loss and post-repair run lengths to figure out where the losses and repairs happened in the interval. For example, consecutive losses are likely not to be repaired by a simple FEC scheme 🦋

6. Identifiers from Sender, Receiver, and Extended Report Blocks ✨

Okay bestie, this section is the metrics "menu" 🍓. Every metric gets an identifier (the name you'd query in the statistics API), and those identifiers are defined on a ReportGroup for each synchronization source (SSRC) 🎀. In real life your app wants to check stats on both the incoming (remote) and outgoing (local) stream, so that's the goal here 💅.

Sending and receiving SRs and RRs is mandatory, so the metrics in those are always there for you 🫶. XR metrics are trickier because two things decide if you can see them. First, is the metric measured at the endpoint? Second, does the endpoint actually report it in an XR block? If it's only measured and never reported, you only get it for the incoming (remote) stream 😭. But if it's measured and reported in an XR block, you get it for both the incoming (remote) and outgoing (local) streams 👑.

For a remote statistic, the timestamp is the one from the incoming SR, RR, or XR packet 🦋. For a local statistic, the timestamp is the current time from your local clock, which is usually the POSIX timestamp (milliseconds since January 1, 1970) ⏰.

As [RFC3550] says, octets metrics count the payload size only. So no header and no padding 🧚

6.1. Cumulative Number of Packets and Octets Sent

  • Name: packetsSent
  • Definition: Section 6.4.1 of [RFC3550].
  • Name: bytesSent
  • Definition: Section 6.4.1 of [RFC3550].

6.2. Cumulative Number of Packets and Octets Received

  • Name: packetsReceived
  • Definition: Section 6.4.1 of [RFC3550].
  • Name: bytesReceived
  • Definition: Section 6.4.1 of [RFC3550].

6.3. Cumulative Number of Packets Lost

  • Name: packetsLost
  • Definition: Section 6.4.1 of [RFC3550].

6.4. Interval Packet Loss and Jitter

  • Name: jitter
  • Definition: Section 6.4.1 of [RFC3550].
  • Name: fractionLost
  • Definition: Section 6.4.1 of [RFC3550].

6.5. Cumulative Number of Packets and Octets Discarded

  • Name: packetsDiscarded
  • Definition: The running total of RTP packets thrown out because they showed up too late or too early. See item a of Appendix A of [RFC7002].
  • Name: bytesDiscarded
  • Definition: The running total of octets thrown out because they showed up too late or too early. See Appendix A of [RFC7243].

6.6. Cumulative Number of Packets Repaired

  • Name: packetsRepaired
  • Definition: The running total of lost RTP packets that got fixed after an error-resilience mechanism did its thing. See item b of Appendix A of [RFC7509]. To be clear, this value is the upper bound on the cumulative number of lost packets 🩹.

6.7. Burst Packet Loss and Burst Discards

  • Name: burstPacketsLost
  • Definition: The running total of RTP packets lost during loss bursts. See item c of Appendix A of [RFC6958].
  • Name: burstLossCount
  • Definition: The running total of bursts of lost RTP packets. See item d of Appendix A of [RFC6958].
  • Name: burstPacketsDiscarded
  • Definition: The running total of RTP packets discarded during discard bursts. See item b of Appendix A of [RFC7003].
  • Name: burstDiscardCount
  • Definition: The running total of bursts of discarded RTP packets. See item e of Appendix A of [RFC8015].
[RFC3611] recommends a Gmin (threshold) value of 16 for classifying a packet loss or discard burst 🚨.

6.8. Burst/Gap Rates

  • Name: burstLossRate
  • Definition: The fraction of RTP packets lost during bursts. See item a of Appendix A of [RFC7004].
  • Name: gapLossRate
  • Definition: The fraction of RTP packets lost during gaps. See item b of Appendix A of [RFC7004].
  • Name: burstDiscardRate
  • Definition: The fraction of RTP packets discarded during bursts. See item e of Appendix A of [RFC7004].
  • Name: gapDiscardRate
  • Definition: The fraction of RTP packets discarded during gaps. See item f of Appendix A of [RFC7004].

6.9. Frame Impairment Metrics

  • Name: framesLost
  • Definition: The running total of full frames lost. See item i of Appendix A of [RFC7004].
  • Name: framesCorrupted
  • Definition: The running total of frames that were only partially lost. See item j of Appendix A of [RFC7004].
  • Name: framesDropped
  • Definition: The running total of full frames discarded. See item g of Appendix A of [RFC7004].
  • Name: framesSent
  • Definition: The running total of frames sent.
  • Name: framesReceived
  • Definition: The running total of partial or full frames received.

7. Adding New Metrics to WebRTC Statistics API 🌈

While this document was being drafted, the metrics defined here got added to the W3C WebRTC specification, iconic 🙌. If you want a new metric in the future, the move is to open an issue or a pull request on the W3C WebRTC specification repository (https://github.com/w3c/webrtc-stats) 🧚

8. Security Considerations 💀

Okay, serious moment 🫢. This document is all about listing the RTCP XR metrics from the corresponding RTCP reporting extensions, and it doesn't create any security vulnerabilities beyond what's already covered in [RFC3611] and [RFC6792]. Those are the ones to read.

The overall security considerations for RTP as used in WebRTC applications are described in [WebRTC-RTP-USAGE] and [WebRTC-Sec], and those apply to this memo too. Do not skip them, periodt 🔒.

9. IANA Considerations 🎀

This document has no IANA actions. Nothing to register, no drama 💅.

10. References 📚

10.1. Normative References

10.2. Informative References

Acknowledgements 🫶

okay bestie, the authors gave a huge shoutout to Bernard Aboba, Harald Alvestrand, Al Morton, Colin Perkins, and Shida Schubert 💖 they gave really valuable comments and suggestions on the earlier draft versions of this doc, and honestly the final version is better because of them ✨ main character energy for the reviewers fr 👑

Authors' Addresses 🎀

okay so here are the people who wrote this 🦋 they're all super talented, so you know where to find them 💅

Varun Singh 🌸

Varun Singh
CALLSTATS I/O Oy
Annankatu 31-33 C 42
Helsinki  00100
Finland

Email: varun@callstats.io
URI:   https://www.callstats.io/about

Rachel Huang 🌈

Rachel Huang
Huawei
101 Software Avenue, Yuhua District
Nanjing  210012
China

Email: rachel.huang@huawei.com

Roni Even 💜

Roni Even
Huawei
14 David Hamelech
Tel Aviv  64953
Israel

Email: roni.even@huawei.com

Dan Romascanu 🧚

Dan Romascanu

Email: dromasca@gmail.com

Lingli Deng ⭐

Lingli Deng
China Mobile

Email: denglingli@chinamobile.com

And that's the whole doc, bestie! 🌸✨ period 💅

📦 where this page comes from