RFC 8082

Using Codec Control Messages in the RTP Audio-Visual Profile with Feedback with Layered Codecs

📅 published March 2017, from ietf

RFC 8082 💅 (Part 1 of 3)

Title Block 🎀

Okay bestie, this one is called Using Codec Control Messages in the RTP Audio-Visual Profile with Feedback with Layered Codecs ✨ It's an IETF (Internet Engineering Task Force) doc, and it's a Standards Track document, so it's a real protocol spec. It was dated March 2017 and has ISSN 2070-1721.

The authors are S. Wenger and J. Lennox from Vidyo, Inc., and B. Burman and M. Westerlund from Ericsson. 🫶

It updates RFC 5104 🔥 and the title block doesn't say it obsoletes anything.

Abstract 🦋

Okay so this doc updates RFC 5104 by fixing a shortcoming in how the Codec Control Message Full Intra Request (FIR) gets described when you're using it with layered codecs 💖 The big deal is that a media sender needs to send a decoder refresh point whenever a FIR comes in on any layer of the layered bitstream, no matter if those layers are sent in a single RTP flow or in multiple RTP flows. 😭

The other payload-specific feedback messages from RFC 5104 and RFC 4585 (which was updated by RFC 5506) got analyzed too, and they were fine. No matching shortcomings found! 🌈

Status of This Memo 🌸

This is an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841.

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc8082. 🍓

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 💜

Table of Contents 💖

1.  Introduction and Problem Statement  . . . . . . . . . . . . .   3
2.  Requirements Language . . . . . . . . . . . . . . . . . . . .   4
3.  Updated Definition of Decoder Refresh Point . . . . . . . . .   4
4.  Full Intra Request for Layered Codecs . . . . . . . . . . . .   5
5.  Identifying the Use of Layered Bitstreams (Informative) . . .   6
6.  Layered Codecs and Non-FIR Codec Control Messages
    (Informative) . . . . . . . . . . . . . . . . . . . . . . . .   7
  6.1.  Picture Loss Indication (PLI) . . . . . . . . . . . . . .   7
  6.2.  Slice Loss Indication (SLI) . . . . . . . . . . . . . . .   7
  6.3.  Reference Picture Selection Indication (RPSI) . . . . . .   7
  6.4.  Temporal-Spatial Trade-Off Request and Notification
        (TSTR/TSTN) . . . . . . . . . . . . . . . . . . . . . . .   8
  6.5.  H.271 Video Back Channel Message (VBCM) . . . . . . . . .   8
7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
8.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
  9.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
  9.2.  Informative References  . . . . . . . . . . . . . . . . .   9
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  11
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  11

1. Introduction and Problem Statement ✨

Okay so the "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)" [RFC4585] and "Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)" [RFC5104] define a bunch of payload-specific feedback messages. A media receiver uses them to tell a media sender about certain conditions or to make certain requests 🎀 Those feedback messages get sent as RTCP receiver reports, and RFC 4585 has timing rules that make them practical for time-sensitive codec control. It's giving efficient! 💅

Since those RFCs were written, layered codecs have gotten super popular 🔥 Layered codecs use multiple sub-bitstreams called "layers" to represent the content at different fidelities. Depending on the media codec and its RTP payload format, there are a few options for how to transport those layers in RTP. Summarizing "A Taxonomy of Semantics and Mechanisms for Real-Time Transport Protocol (RTP) Sources" [RFC7656]):

Single layers or groups of layers may be sent in their own RTP streams in Multiple RTP streams on a Single media Transport (MRST) or Multiple RTP streams on Multiple media Transports (MRMT) mode. Using media-codec specific multiplexing mechanisms, multiple layers may be sent in a single RTP stream in Single RTP stream on a Single media Transport (SRST) mode.

The dependency relationship between layers in a truly layered, pyramid-shaped bitstream forms a directed graph, with the base layer at the root 🌳 Enhancement layers depend on the base layer and potentially on other enhancement layers. The target layer and all layers it depends on have to be decoded jointly to recreate the uncompressed media signal at the fidelity of the target layer. This layering structure is assumed from here on out. For more exotic layering structures, please see Section 5. 🧚

Implementation experience has shown that the Full Intra Request (FIR) command as defined in [RFC5104] is underspecified when used with layered codecs and when more than one RTP stream carries the layers of a layered bitstream at a given fidelity 😭 The problem is that from the [RFC5104] language, it's not clear whether a FIR received for just one of those RTP streams has to trigger a decoder refresh point (as defined in [RFC5104], Section 2.2) for all layers, or only for the layer carried in the RTP stream the FIR is associated with. Confusing, right?? 🙈

This document fixes that shortcoming by:

  • a. Updating the definition of the decoder refresh point (as defined in [RFC5104], Section 2.2) to cover layered codecs, in line with the corresponding definitions used in a popular layered codec format, namely H.264/SVC (Scalable Video Coding) [H.264]. Specifically, a decoder refresh point, in conjunction with layered codecs, resets the state of the whole decoder, which implies that it includes hard or gradual single-layer decoder refresh for all layers;
  • b. Requiring a media sender to send a decoder refresh point after the media sender has received a FIR over an RTCP stream associated with any of the RTP streams over which a part of the layered bitstream is transported;
  • c. Requiring that a media receiver send the FIR on the RTCP stream associated with the base layer. The option of receiving FIR on the enhancement-layer-associated RTCP stream as specified in point b) above is kept for backward compatibility; and
  • d. Providing guidance on how to detect that a layered bitstream is in use for which the above rules apply.

Soooo, the reaction to FIR for layered codecs in [RFC5104] and its companion docs is underspecified, but it seems like that's not the case for any of the other payload-specific codec control messages defined in [RFC4585] and [RFC5104] 💅 A brief summary of the analysis that led to that conclusion is also included in this document. 🦋

2. Requirements Language 💜

Okay, here's the key-word situation: the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 🎀

3. Updated Definition of Decoder Refresh Point 🌈

The rest of this section completely replaces the definition of decoder refresh point in Section 2.2 of [RFC5104]. ✨

Decoder Refresh Point: A bit string, packetized in one or more RTP packets, that completely resets the decoder to a known state. 🔥

Examples of "hard" single-layer decoder refresh points are Intra pictures in H.261 [H.261], H.263 [H.263], MPEG-1 [MPEG-1], MPEG-2 [MPEG-2], and MPEG-4 [MPEG-4]. Instantaneous Decoder Refresh (IDR) pictures in H.264 [H.264] and H.265 [H.265] count too, and so do keyframes in VP8 [RFC6386] and VP9 [VP9-BITSTREAM]. "Gradual" decoder refresh points can also be used; see, for example, H.264 [H.264]. Both "hard" and "gradual" decoder refresh points are acceptable in the scope of this specification, but in most cases the user experience will benefit from using a "hard" decoder refresh point. 🍓

A decoder refresh point also contains all header information above the syntactical level of the picture layer that is conveyed in-band. In [H.264], for example, a decoder refresh point contains those parameter set Network Adaptation Layer (NAL) units that generate parameter sets necessary for the decoding of the following slice/data partition NAL units. (That is, assuming the parameter sets have not been conveyed out of band.) 🧚

When a layered codec is in use, the above definition (in particular, the requirement to completely reset the decoder to a known state) implies that the decoder refresh point includes hard or gradual single-layer decoder refresh points for all layers. This is the key part, bestie 👑

4. Full Intra Request for Layered Codecs 🦋

A media receiver or middlebox may decide to send a FIR command based on the guidance in Section 4.3.1 of [RFC5104]. When sending the FIR command, it MUST target the RTP stream that carries the base layer of the layered bitstream. You do this by setting the Feedback Control Information (FCI) (and in particular, the synchronization source (SSRC) field in it) to refer to the SSRC of the forward RTP stream that carries the base layer. 💖

When a Full Intra Request command is received by the designated media sender in the RTCP stream associated with any of the RTP streams in which any layer of a layered bitstream is sent, the designated media sender MUST send a decoder refresh point (Section 3) as defined above at its earliest opportunity. The congestion control requirements on the forward RTP streams in Sections 3.5.1 and 5 of [RFC5104] apply to the RTP streams both in isolation and combined. 🌸

Note: the requirement to react to FIR commands associated with enhancement layers is included for robustness and backward-compatibility reasons. 🫶

5. Identifying the Use of Layered Bitstreams (Informative) 🦋

okay so the changes to RFC 5104 we just went over are super clear about how to handle FIR commands when layered bitstreams are in use 💅. But figuring out whether you're actually looking at a layered bitstream is surprisingly hard 😭. In the usual sense, meaning inter-layer prediction between pyramid-arranged layers, implementers are expected to know when it's happening and when it's not, so they can apply the updates to RFC 5104 correctly. The tricky part is that some layered codecs make non-pyramid-shaped bitstreams. That's kind of exotic today, but certain video coding syntaxes, like H.264/SVC, totally support it 🌈. If you blindly apply the rules above to those structures, you get suboptimal behavior. Nothing breaks and there's no interoperability failure, but the user experience can suffer because decoder refresh points get sent at times or on parts of the bitstream that don't really matter to the user 🙈. So this informative section is here to lay out the current understanding of when a layered bitstream is in use and when it isn't ✨.

The big key observation is this: the RTP payload format negotiated for the RTP streams, looked at on its own, isn't necessarily a sign that a layered bitstream is being used 👀. Some layered codecs, including H.264/SVC, can form decodable bitstreams that contain only (one or more) enhancement layers, without the base layer. That effectively creates simulcastable sub-bitstreams inside one scalable bitstream (as defined in the video coding standard), but with no inter-layer prediction 🧚. In that situation, sending a decoder refresh point on all layers for that payload format and media source might be counterproductive. It might be, but it's not guaranteed 🤷‍♀️. Optimized reactions to FIRs received on RTP streams carrying these exotic bitstreams are out of scope for this document, so we're not getting into that here 🙅‍♀️.

One good indication that you're probably looking at pyramid-shaped layering with inter-layer prediction is when the various RTP streams are "bound" together at the signaling level 💜. In an SDP environment, that means they're marked as dependent on each other using "The Session Description Protocol (SDP) Grouping Framework" [RFC5888] and layer dependency [RFC5583]. Iconic signaling moment 💅.

6. Layered Codecs and Non-FIR Codec Control Messages (Informative) 🎀

Between them, AVPF [RFC4585] and Codec Control Messages [RFC5104] define seven payload-specific feedback messages in total 📚. We already covered guidance for the FIR command message above, so this section gives some info on the remaining six codec control messages 🫶.

6.1. Picture Loss Indication (PLI) 🌸

PLI is defined in Section 6.3.1 of [RFC4585] 💖. The prudent response to a PLI received for an enhancement layer is to "repair" that enhancement layer and all dependent enhancement layers using appropriate source-coding-specific means ✨. The reference layer or layers that the enhancement layer uses don't need repairing though 🙌. The encoder can figure out on its own what a dependent enhancement layer is, so it doesn't need any help from the system stack. Honestly, there's nothing that needs to be specified here 😌.

6.2. Slice Loss Indication (SLI) 🍓

SLI is defined in Section 6.3.2 of [RFC4585] 🧚. The current understanding is that the prudent response to an SLI received for an enhancement layer is to "repair" the affected spatial area of that enhancement layer and all dependent enhancement layers using appropriate source-coding-specific means. Same as PLI, the reference layers used by that enhancement layer don't need repairing, and the encoder can work out by itself what counts as a dependent enhancement layer without any help from the system stack. So again, nothing needs to be specified here 💅. SLI has seen very little implementation, and as far as anyone knows, none of it has been used with layered systems 🥲.

6.3. Reference Picture Selection Indication (RPSI) 🦋

RPSI is defined in Section 6.3.3 of [RFC4585] 🌈. A technical equivalent of RPSI has been used with non-layered systems for many years, but no implementations are known to work with layered codecs 👀. The current understanding is that when an RPSI arrives on any layer and indicates a missing reference picture, the encoder has to handle that missing reference picture in the indicated layer and in all dependent layers. So RPSI should just work without needing any extra specification language 💜.

6.4. Temporal-Spatial Trade-Off Request and Notification (TSTR/TSTN) ⭐

TSTR/TSTN are defined in Sections 4.3.2 and 4.3.3 of [RFC5104], respectively 📚. A TSTR request tells the other side which trade-off between spatial quality and frame rate it prefers 🎬. A technical equivalent of TSTR/TSTN has been deployed for many years in non-scalable systems, so it's not brand new 🙃.

TSTR and TSTN messages include an SSRC target, which, just like FIR, may refer to an RTP stream carrying a base layer, an enhancement layer, or multiple layers 🫧. So the current understanding is that the semantics of the message apply to the layers present in the targeted RTP stream 🎀.

Here's the tea 🍵: per-layer TSTR/TSTN is, in some ways, counterproductive in a system using layered codecs. With a sufficiently complex layered bitstream layout, the sending system can adjust the spatio/temporal quality balance by adding or removing temporal, spatial, or quality enhancement layers 🌸. Right now it's unclear whether the allowed (or even recommended) response to a TSTR is to adjust the bit allocation within the layer(s) in the addressed RTP stream, or to adjust the layering structure instead. The second option can involve more than just the addressed RTP stream 😳.

Until there's a critical mass of implementation practice, it's probably prudent for implementers not to assume either option, or any middle ground between them 🙈. Instead, it's suggested that implementations be liberal in accepting TSTR messages and, when one arrives, respond with a TSTN indicating "no change" 🫶. It's also suggested that new implementations don't send TSTR messages unless they're operating in SRST mode as defined in [RFC7656] 🧚. Finally, implementers are encouraged to contribute to the IETF documentation of any implementation requirements that would make per-layer TSTR/TSTN useful 🔥.

6.5. H.271 Video Back Channel Message (VBCM) 🌈

VBCM is defined in Section 4.3.4 of [RFC5104] 💫. Everything said above for RPSI (Section 6.3) applies here too 💅.

7. IANA Considerations 🌸

This memo doesn't ask IANA for anything 🙅‍♀️. Nothing to see here, no registrations, no requests 🫶.

8. Security Considerations 🔥

The security considerations of AVPF [RFC4585] (as updated by "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences" [RFC5506]) and Codec Control Messages [RFC5104] apply here, and they are serious 😭🚨. The clarified response to FIR does not introduce any additional security considerations, but the inherited ones still absolutely count 🛡️. Please don't skip them, bestie 💖.

9. References 📚

9.1. Normative References 💜

9.2. Informative References 🦋

Acknowledgements 💖

The authors want to thank Mo Zanaty for useful discussions 🫶🌷.

Authors' Addresses 🎀

okay bestie, these are the people who wrote it, so here's where to find them 💖

  • Stephan Wenger 🦋
    • Vidyo, Inc.
    • Email: stewe@stewe.org
  • Jonathan Lennox 💅
    • Vidyo, Inc.
    • Email: jonathan@vidyo.com
  • Bo Burman 🌸
    • Ericsson
    • Kistavagen 25
    • SE - 164 80 Kista
    • Sweden
    • Email: bo.burman@ericsson.com
  • Magnus Westerlund 🧚
    • Ericsson
    • Farogatan 2
    • SE - 164 80 Kista
    • Sweden
    • Phone: +46107148287
    • Email: magnus.westerlund@ericsson.com

that's the whole cast, the end of the document, and four authors who clearly knew their stuff 👑✨

📦 where this page comes from