RFC 5104
Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)
RFC 5104: Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF) 🎀
- Series: Network Working Group, Request for Comments: 5104
- Category: Standards Track
- Date: February 2008
- Authors: S. Wenger and U. Chandra (Nokia), M. Westerlund and B. Burman (Ericsson)
- Obsoletes / Updates: the title block doesn't list anything it obsoletes or updates 💖
Status of This Memo ✨
okay so this doc is an Internet standards track protocol for the Internet community 🫶 It also asks for discussion and suggestions for improvements. If you want the current standardization state and status of this protocol, check the latest edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🌈
Abstract 🦋
this doc adds a few extensions to the messages defined in the Audio-Visual Profile with Feedback (AVPF) 💅 They're super helpful mainly in conversational multimedia setups that use centralized multipoint functions. Some of them also work in smaller multicast environments and point-to-point calls 📞
The extensions covered are messages related to the ITU-T Rec. H.271 Video Back Channel, Full Intra Request, Temporary Maximum Media Stream Bit Rate, and Temporal-Spatial Trade-off 🔥
Table of Contents 📋
1. Introduction ....................................................4
2. Definitions .....................................................5
2.1. Glossary ...................................................5
2.2. Terminology ................................................5
2.3. Topologies .................................................8
3. Motivation ......................................................8
3.1. Use Cases ..................................................9
3.2. Using the Media Path ......................................11
3.3. Using AVPF ................................................11
3.3.1. Reliability ........................................12
3.4. Multicast .................................................12
3.5. Feedback Messages .........................................12
3.5.1. Full Intra Request Command .........................12
3.5.1.1. Reliability ...............................13
3.5.2. Temporal-Spatial Trade-off Request and
Notification .......................................14
3.5.2.1. Point-to-Point ............................15
3.5.2.2. Point-to-Multipoint Using
Multicast or Translators ..................15
3.5.2.3. Point-to-Multipoint Using RTP Mixer .......15
3.5.2.4. Reliability ...............................16
3.5.3. H.271 Video Back Channel Message ...................16
3.5.3.1. Reliability ...............................19
3.5.4. Temporary Maximum Media Stream Bit Rate
Request and Notification ...........................19
3.5.4.1. Behavior for Media Receivers Using TMMBR ..21
3.5.4.2. Algorithm for Establishing Current
Limitations ...............................23
3.5.4.3. Use of TMMBR in a Mixer-Based
Multipoint Operation ......................29
3.5.4.4. Use of TMMBR in Point-to-Multipoint Using
Multicast or Translators ..................30
3.5.4.5. Use of TMMBR in Point-to-Point Operation ..31
3.5.4.6. Reliability ...............................31
4. RTCP Receiver Report Extensions ................................32
4.1. Design Principles of the Extension Mechanism ..............32
4.2. Transport Layer Feedback Messages .........................33
4.2.1. Temporary Maximum Media Stream Bit Rate
Request (TMMBR) ....................................34
4.2.1.1. Message Format ............................34
4.2.1.2. Semantics .................................35
4.2.1.3. Timing Rules ..............................39
4.2.1.4. Handling in Translators and Mixers ........39
4.2.2. Temporary Maximum Media Stream Bit Rate
Notification (TMMBN) ...............................39
4.2.2.1. Message Format ............................39
4.2.2.2. Semantics .................................40
4.2.2.3. Timing Rules ..............................41
4.2.2.4. Handling by Translators and Mixers ........41
4.3. Payload-Specific Feedback Messages ........................41
4.3.1. Full Intra Request (FIR) ...........................42
4.3.1.1. Message Format ............................42
4.3.1.2. Semantics .................................43
4.3.1.3. Timing Rules ..............................44
4.3.1.4. Handling of FIR Message in Mixers and
Translators ...............................44
4.3.1.5. Remarks ...................................44
4.3.2. Temporal-Spatial Trade-off Request (TSTR) ..........45
4.3.2.1. Message Format ............................46
4.3.2.2. Semantics .................................46
4.3.2.3. Timing Rules ..............................47
4.3.2.4. Handling of Message in Mixers and
Translators ...............................47
4.3.2.5. Remarks ...................................47
4.3.3. Temporal-Spatial Trade-off Notification (TSTN) .....48
4.3.3.1. Message Format ............................48
4.3.3.2. Semantics .................................49
4.3.3.3. Timing Rules ..............................49
4.3.3.4. Handling of TSTN in Mixers and
Translators ...............................49
4.3.3.5. Remarks ...................................49
4.3.4. H.271 Video Back Channel Message (VBCM) ............50
4.3.4.1. Message Format ............................50
4.3.4.2. Semantics .................................51
4.3.4.3. Timing Rules ..............................52
4.3.4.4. Handling of Message in Mixers or
Translators ...............................52
4.3.4.5. Remarks ...................................52
5. Congestion Control .............................................52
6. Security Considerations ........................................53
7. SDP Definitions ................................................54
7.1. Extension of the rtcp-fb Attribute ........................54
7.2. Offer-Answer ..............................................55
7.3. Examples ..................................................56
8. IANA Considerations ............................................58
9. Contributors ...................................................60
10. Acknowledgements ..............................................60
11. References ....................................................60
11.1. Normative References .....................................60
11.2. Informative References ...................................611. Introduction 💖
so AVPF [RFC4585] was built mainly for efficient point-to-point and small multipoint calls with no centralized multipoint control 💅 But in real life, lots of small multipoint conferences run on devices called Multipoint Control Units (MCUs) 🏢 The conversational video conferencing industry has been around for a long time, and it's clear it needs a few extra feedback messages to handle centralized multipoint conferencing efficiently ✨ Some of these messages are useful beyond centralized multipoint too, and the description of each message says so. This is especially true for the message meant to carry ITU-T Rec. H.271 [H.271] bit strings for Video Back Channel messages 🦋
In Real-time Transport Protocol (RTP) [RFC3550] terms, MCUs are made of mixers and translators 🧚 Most MCUs also include signaling support. While working on this memo, people noticed there's a lot of confusion around words like mixer, translator, and MCU 😭 To help, a number of topologies with real industry relevance were identified, but [RFC3550] doesn't document them in enough detail. They're documented in [RFC5117], and you need to have read (or be reading alongside) [RFC5117] to get this memo 📚
Some of the messages here are forward only. They don't require an explicit notification back to the sender that they arrived, or saying what the receiver did about them 👀 Other messages need a response, which makes it a two-way communication model that could be useful for control purposes. BUT the point of this memo is NOT to turn RTP Control Protocol (RTCP) into a generalized control protocol 🙅♀️ All the messages mentioned have pretty strict real-time constraints, meaning their value drops the longer they take to arrive ⏰ So more traditional control protocol options, like Session Initiation Protocol (SIP) [RFC3261], are undesirable for the same purpose. That's why this solution is recommended instead of "XML Schema for Media Control" [XML-MC], which uses SIP Info to carry XML messages with similar semantics to the ones defined in this memo 💜 Also, all the messages have a super simple format that an RTP/RTCP sender/receiver can process easily 🍓 Last but definitely not least, every message only relates to the RTP stream it's attached to, and not to any other property of a communication system. Specifically, none of them relate to the properties of the access links the session travels over 🌸
2. Definitions 🎀
2.1. Glossary 📖
AIMD - Additive Increase Multiplicative Decrease
AVPF - The extended RTP profile for RTCP-based feedback
FCI - Feedback Control Information [RFC4585]
FEC - Forward Error Correction
FIR - Full Intra Request
MCU - Multipoint Control Unit
MPEG - Moving Picture Experts Group
PLI - Picture Loss Indication
PR - Packet rate
QP - Quantizer Parameter
RTT - Round trip time
SSRC - Synchronization Source
TMMBN - Temporary Maximum Media Stream Bit Rate Notification
TMMBR - Temporary Maximum Media Stream Bit Rate Request
TSTN - Temporal-Spatial Trade-off Notification
TSTR - Temporal-Spatial Trade-off Request
VBCM - Video Back Channel Message2.2. Terminology 📚✨
Okay bestie, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 🎀
Message: So a Message is an RTCP feedback message [RFC4585] defined by this specification, and it comes in one of these flavors 💅:
- Request: Message that requires acknowledgement 🙋♀️
- Command: Message that forces the receiver to an action 🫡
- Indication: Message that reports a situation 👀
- Notification: Message that provides a notification that an event has occurred. Notifications are commonly generated in response to a Request. 🔔
Note that, with the exception of "Notification", this terminology is in alignment with ITU-T Rec. H.245 [H245]. It's giving consistency 🤝
Decoder Refresh Point: A bit string, packetized in one or more RTP packets, that completely resets the decoder to a known state. 🔄🌸
Examples for "hard" decoder refresh points are Intra pictures in H.261, H.263, MPEG-1, MPEG-2, and MPEG-4 part 2, and Instantaneous Decoder Refresh (IDR) pictures in H.264. "Gradual" decoder refresh points may also be used; see for example [AVC]. While both "hard" and "gradual" decoder refresh points are acceptable in the scope of this specification, in most cases the user experience will benefit from using a "hard" decoder refresh point. Hard ones are the iconic choice here 👑
A decoder refresh point also contains all header information above the picture layer (or equivalent, depending on the video compression standard) that is conveyed in-band. In H.264, for example, a decoder refresh point contains parameter set Network Adaptation Layer (NAL) units that generate parameter sets necessary for the decoding of the following slice/data partition NAL units (and that are not conveyed out of band). So it packs its own parameter sets, no cap 🧚
Decoding: The operation of reconstructing the media stream. 🧩
Rendering: The operation of presenting (parts of) the reconstructed media stream to the user. 🖼️✨
Stream thinning: The operation of removing some of the packets from a media stream. Stream thinning, preferably, is media-aware, implying that media packets are removed in the order of increasing relevance to the reproductive quality. However, even when employing media-aware stream thinning, most media streams quickly lose quality when subjected to increasing levels of thinning. Media-unaware stream thinning leads to even worse quality degradation 😭. In contrast to transcoding, stream thinning is typically seen as a computationally lightweight operation. Lowkey the lightweight glow-up of the bunch 🦋
Media: Often used (sometimes in conjunction with terms like bit rate, stream, sender, etc.) to identify the content of the forward RTP packet stream (carrying the codec data), to which the codec control message applies. 🎬
Media Stream: The stream of RTP packets labeled with a single Synchronization Source (SSRC) carrying the media (and also in some cases repair information such as retransmission or Forward Error Correction (FEC) information). 📦💜
Total media bit rate: The total bits per second transferred in a media stream, measured at an observer-selected protocol layer and averaged over a reasonable timescale, the length of which depends on the application. In general, a media sender and a media receiver will observe different total media bit rates for the same stream, first because they may have selected different reference protocol layers, and second, because of changes in per-packet overhead along the transmission path. The goal with bit rate averaging is to be able to ignore any burstiness on very short timescales (e.g., below 100 ms) introduced by scheduling or link layer packetization effects. Averaging is how we stay chill about tiny bursts 🌊
Maximum total media bit rate: The upper limit on total media bit rate for a given media stream at a particular receiver and for its selected protocol layer. Note that this value cannot be measured on the received media stream. Instead, it needs to be calculated or determined through other means, such as quality of service (QoS) negotiations or local resource limitations. Also note that this value is an average (on a timescale that is reasonable for the application) and that it may be different from the instantaneous bit rate seen by packets in the media stream. It's the ceiling, not the vibe check 🚧
Overhead: All protocol header information required to convey a packet with media data from sender to receiver, from the application layer down to a pre-defined protocol level (for example, down to, and including, the IP header). Overhead may include, for example, IP, UDP, and RTP headers, any layer 2 headers, any Contributing Sources (CSRCs), RTP padding, and RTP header extensions. Overhead excludes any RTP payload headers and the payload itself. The extra baggage that travels with the snacks 🎒🍓
Net media bit rate: The bit rate carried by a media stream, net of overhead. That is, the bits per second accounted for by encoded media, any applicable payload headers, and any directly associated meta payload information placed in the RTP packet. A typical example of the latter is redundancy data provided by the use of RFC 2198 [RFC2198]. Note that, unlike the total media bit rate, the net media bit rate will have the same value at the media sender and at the media receiver unless any mixing or translating of the media has occurred. The pure, no-filler version of the bit rate 💖
For a given observer, the total media bit rate for a media stream is equal to the sum of the net media bit rate and the per-packet overhead as defined above multiplied by the packet rate. Math check, periodt 🧮✨
Feasible region: The set of all combinations of packet rate and net media bit rate that do not exceed the restrictions in maximum media bit rate placed on a given media sender by the Temporary Maximum Media Stream Bit Rate Request (TMMBR) messages it has received. The feasible region will change as new TMMBR messages are received. It's the safe zone, and it keeps shifting as new TMMBR messages roll in 🗺️
Bounding set: The set of TMMBR tuples, selected from all those received at a given media sender, that define the feasible region for that media sender. The media sender uses an algorithm such as that in section 3.5.4.2 to determine or iteratively approximate the current bounding set, and reports that set back to the media receivers in a Temporary Maximum Media Stream Bit Rate Notification (TMMBN) message. The sender basically hands the receivers the receipts 🧾💅
2.3. Topologies 🗺️🌈
Please refer to [RFC5117] for an in-depth discussion. The topologies referred to throughout this memo are labeled (consistently with [RFC5117]) as follows:
Topo-Point-to-Point . . . . . Point-to-point communication
Topo-Multicast . . . . . . . Multicast communication
Topo-Translator . . . . . . . Translator based
Topo-Mixer . . . . . . . . . Mixer based
Topo-RTP-switch-MCU . . . . . RTP stream switching MCU
Topo-RTCP-terminating-MCU . . Mixer but terminating RTCP3. Motivation 🎀💭
This section discusses the motivation and usage of the different video and media control messages. The video control messages have been under discussion for a long time, and a requirement document was drawn up [Basso]. That document has expired; however, we quote relevant sections of it to provide motivation and requirements. So the backstory is long, but the receipts are still worth reading 📜🫶
3.1. Use Cases
Okay so there are a bunch of ways these feedback messages could be used 💅 We're starting with the use cases Basso et al. [Basso] came up with. Some got reformulated and we added comments, bestie 🫶
- An RTP video mixer composes multiple encoded video sources into a single encoded video stream 🎀 Every time a new video source gets added, the RTP mixer needs to ask that source for a decoder refresh point. That way it can start an uncorrupted prediction chain on the part of the mixed picture where the new source's data lives 🧚
- An RTP video mixer gets multiple encoded RTP video streams from conference participants and dynamically picks one of them to go in its output RTP stream 👑 When it's time to switch (decided by things like voice activation or the user interface), the mixer asks the remote source for a decoder refresh point, so it doesn't use unrelated content as reference data for inter picture prediction 🙈 After that request, the video mixer stops sending the current RTP stream and watches the RTP stream from the new source until it spots data belonging to the decoder refresh point. Only then does the RTP mixer start forwarding the newly selected stream to the receiver(s) ✨
- An application needs to tell the remote encoder that the desired trade-off between temporal and spatial resolution has changed 🌈 For example, one user may want a higher frame rate and lower spatial quality, while another user wants the opposite. This choice also depends a lot on the content 💀 Lots of current video conferencing systems offer a way to make this choice in the user interface, usually as a slider. The mechanism is helpful in point-to-point, centralized multipoint and non-centralized multipoint uses 🦋
- Use case 4 of the Basso document applies only to Picture Loss Indication (PLI) as defined in AVPF [RFC4585] and is not reproduced here 🌸
- Use case 5 of the Basso document relates to a mechanism known as "freeze picture request" 🙃 Sending freeze picture requests over a non-reliable forward RTCP channel has been identified as problematic, so no freeze picture request has been included in this memo, and the use case discussion is not reproduced here 🚫
- A video mixer dynamically selects one of the received video streams to send out to participants and tries to give all participants the highest bit rate possible, while minimizing stream trans-rating 🍓 One way to do this is to set up sessions with endpoints using the maximum bit rate accepted by each endpoint, and accepted by the call admission method used by the mixer. By means of commands that reduce the maximum media stream bit rate below what was negotiated during session set up, the mixer can reduce the maximum bit rate sent by endpoints to the lowest of all the accepted bit rates. As the lowest accepted bit rate changes due to endpoints joining and leaving, or due to network congestion, the mixer can adjust the limits at which endpoints can send their streams to match the new value. The mixer then requests a new maximum bit rate, which is equal to or less than the maximum bit rate negotiated at session setup for a specific media stream, and the remote endpoint can respond with the actual bit rate that it can support 📊
The picture Basso, et al., draw up covers most applications we foresee 💖 However, we would like to extend the list with two additional use cases:
- Currently deployed congestion control algorithms (AIMD and TCP Friendly Rate Control (TFRC) [RFC3448]) probe for additional available capacity as long as there is something to send 😭 With congestion control algorithms that use packet loss as the indication for congestion, this probing generally results in reduced media quality (often to a point where the distortion is large enough to make the media unusable), due to packet loss and increased delay.
In a number of deployment scenarios, especially cellular ones, the bottleneck link is often the last hop link 📱 That cellular link also commonly has some type of QoS negotiation, enabling the cellular device to learn the maximal bit rate available over this last hop. A media receiver behind this link can, in most (if not all) cases, calculate at least an upper bound for the bit rate available for each media stream it currently receives. How this is done is an implementation detail and is not discussed herein. Indicating the maximum available bit rate to the transmitting party for the various media streams can be beneficial, since it prevents that party from probing for bandwidth for this stream in excess of a known hard limit. For cellular or other mobile devices, the known available bit rate for each stream (deduced from the link bit rate) can change quickly, due to handover to another transmission technology, QoS renegotiation due to congestion, and so on. To enable minimal disruption of service, quick convergence is necessary, and therefore media path signaling is desirable 🔥
- The use of reference picture selection (RPS) as an error resilience tool was introduced in 1997 as NEWPRED [NEWPRED], and is now widely deployed 🌟 When RPS is in use, simplistically put, the receiver can send a feedback message to the sender, indicating a reference picture that should be used for future prediction. ([NEWPRED] mentions other forms of feedback as well.) AVPF contains a mechanism for conveying such a message, but did not specify for which codec, or according to which syntax, the message should conform. Recently, the ITU-T finalized Rec. H.271, which (among other message types) also includes a feedback message. It is expected that this feedback message will fairly quickly enjoy wide support 🧠 Therefore, a mechanism to convey feedback messages according to H.271 appears to be desirable 🎀
3.2. Using the Media Path
There are two reasons why we use the media path for the codec control messages 🛤️
First, systems employing MCUs often separate the control and media processing parts 🏢 As these messages are intended for, or generated by, the media part rather than the signaling part of the MCU, having them on the media path avoids transmission across interfaces and unnecessary control traffic between signaling and processing. If the MCU is physically decomposed, the use of the media path avoids the need for media control protocol extensions (e.g., in media gateway control (MEGACO) [RFC3525]) 🧩
Secondly, the signaling path quite commonly contains several signaling entities, e.g., SIP proxies and application servers 🚦 Avoiding going through signaling entities avoids delay, for several reasons. Proxies have less stringent delay requirements than media processing, and due to their complex and more generic nature, they may result in significant processing delay. The topological locations of the signaling entities are also commonly not optimized for minimal delay, but rather for other architectural goals. Thus, the signaling path can be significantly longer, both geographically and in a delay sense 🐢
3.3. Using AVPF
The AVPF feedback message framework [RFC4585] provides the appropriate framework to implement the new messages 🧁 AVPF implements rules controlling the timing of feedback messages, to avoid congestion through network flooding by RTCP traffic. We re-use these rules by referencing AVPF 🔁
The signaling setup for AVPF allows each individual type of function to be configured or negotiated on an RTP session basis 🎛️
3.3.1. Reliability
The use of RTCP messages implies that each message transfer is unreliable, unless the lower layer transport provides reliability 🫠 The different messages proposed in this specification have different requirements in terms of reliability. However, in all cases, the reaction to an (occasional) loss of a feedback message is specified 🛟
3.4. Multicast
The codec control messages might be used with multicast 📡 The RTCP timing rules specified in [RFC3550] and [RFC4585] ensure that the messages do not cause overload of the RTCP connection. The use of multicast may result in the reception of messages with inconsistent semantics 👀 The reaction to inconsistencies depends on the message type, and is discussed for each message type separately.
3.5. Feedback Messages
This section describes the semantics of the different feedback messages and how they apply to the different use cases 📝
3.5.1. Full Intra Request Command
Okay, the Full Intra Request (FIR) Command is the one that's giving main character energy 👑 When the designated media sender receives it, it requires that the media sender sends a Decoder Refresh Point (see section 2.2) at the earliest opportunity 🎬 The evaluation of such an opportunity includes the current encoder coding strategy and the current available network resources.
FIR is also known as an "instantaneous decoder refresh request", "fast video update request" or "video fast update request" 🎀
Using a decoder refresh point means refraining from using any picture sent prior to that point as a reference for the encoding process of any subsequent picture sent in the stream 🚫 For predictive media types that are not video, the analogue applies. For example, if in MPEG-4 systems scene updates are used, the decoder refresh point consists of the full representation of the scene and is not delta-coded relative to previous updates 🎞️
Decoder refresh points, especially Intra or IDR pictures, are in general several times larger in size than predicted pictures 🐘 Thus, in scenarios in which the available bit rate is small, the use of a decoder refresh point implies a delay that is significantly longer than the typical picture duration 🥲
Usage in multicast is possible; however, aggregation of the commands is recommended 🫶 A receiver that receives a request closely after sending a decoder refresh point -- within 2 times the longest round trip time (RTT) known, plus any AVPF-induced RTCP packet sending delays -- should await a second request message, to ensure that the media receiver has not been served by the previously delivered decoder refresh point ⏳ The reason for the specified delay is to avoid sending unnecessary decoder refresh points. A session participant may have sent its own request while another participant's request was in-flight to them. Suppressing those requests that may have been sent without knowledge about the other request avoids this issue 🙌
Using the FIR command to recover from errors is explicitly disallowed, and instead the PLI message defined in AVPF [RFC4585] should be used 🚨 The PLI message reports lost pictures and has been included in AVPF for precisely that purpose 💅
Full Intra Request is applicable in use-cases 1 and 2 🌷
3.5.1.1. Reliability ✨
okay so the FIR message results in a decoder refresh point getting delivered 🎀, unless the message gets lost 💀. Decoder refresh points are super easy to spot in the bit stream 👀, so there's no need for protocol-level notification. A simple command repetition mechanism is enough for the level of reliability we need 💅. BUT repeating stuff means the recipient needs a way to not respond to messages it already got and answered 🙈, so that part is non-negotiable.
To get the best possible reliability, a FIR sender may repeat the FIR until the desired content has been received 🔁. The repetition interval is set by the RTCP timing rules that apply to the session. Once the sender gets a complete decoder refresh point, or detects an attempt to send one that got damaged by packet loss, the repetition of the FIR has to stop 🛑. If another FIR is needed, the request sequence number must be increased. A FIR sender shall not have more than one FIR (different request sequence number) outstanding at any time per media sender in the session 🫶.
The receiver of FIR (i.e., the media sender) is the complementary side, so it helps make sure a decoder refresh point actually gets delivered 🧚. If it receives repetitions of the FIR more than 2*RTT after it has sent a decoder refresh point, it shall send a new decoder refresh point. Why 2*RTT? Two round trip times give the decoder refresh point time to get back to the requestor, and give the end of the FIR repetitions time to reach and be detected by the media sender. It's giving very well-timed 🕰️✨.
An RTP mixer or RTP switching MCU that receives a FIR from a media receiver is responsible for making sure a decoder refresh point is delivered to the requesting receiver 👑. It may be necessary for the mixer/MCU to generate FIR commands too. From a reliability perspective, the two legs (FIR-requesting endpoint to mixer/MCU, and mixer/MCU to decoder refresh point generating endpoint) are handled independently from each other. Two separate legs, zero drama 🌈.
3.5.2. Temporal-Spatial Trade-off Request and Notification
The Temporal-Spatial Trade-off Request (TSTR) tells the video encoder to change its trade-off between temporal and spatial resolution 🦋. Index values from 0 to 31 go up in order to show a growing desire for higher frame rate. So a requester asking for an index of 0 wants high quality and is fine with a low frame rate, whereas a requester asking for 31 wants a high frame rate, potentially at the cost of low spatial quality 🍓.
In general, the encoder reaction time may be significantly longer than the typical picture duration. See use case 3 for an example ⏳. The encoder decides whether and to what extent the request results in a change of the trade-off. It then returns a Temporal-Spatial Trade-off Notification (TSTN) message to indicate the trade-off it will use henceforth 🔥.
TSTR and TSTN were introduced mainly because control protocol mechanisms, e.g., a SIP re-invite, are believed to be too heavyweight and too slow for a reasonable user experience 😭. Picture a user interface where the remote user picks the temporal/spatial trade-off with a slider 🎚️. Immediate feedback to any slider movement is needed for that to feel good. A SIP re-INVITE [RFC3261] would need at least two more round-trips than the TSTR/TSTN mechanism, and it may involve proxies and other complex mechanisms 🙄. Even in a well-designed system, it could take a second or so until the new trade-off is finally selected. Also, using RTCP solves the multicast use case very efficiently 💜.
The use of TSTR and TSTN in multipoint scenarios is a non-trivial subject, and it can be achieved in many implementation-specific ways 🧩.
The problem is that TSTRs will typically arrive unsynchronized, and may request different trade-off values for the same stream and/or endpoint encoder 😵💫. This memo does not specify how a translator, mixer, or endpoint reacts to receiving a suggested trade-off in a TSTR. We only require the receiver of a TSTR message to reply by sending a TSTN, carrying the new trade-off it chose by its own criteria (which may or may not be based on the trade-off in the TSTR). In other words, the trade-off sent in a TSTR is a non-binding recommendation, nothing more 🫧.
Three TSTR/TSTN scenarios need to be distinguished, based on the topologies described in [RFC5117]. The scenarios are described in the following subsections 📚.
3.5.2.1. Point-to-Point
In this most trivial case (Topo-Point-to-Point), the media sender typically adjusts its temporal/spatial trade-off based on the requested value in TSTR, subject to its own capabilities 🥰. The TSTN message conveys back the new trade-off value. That value may be identical to the old one if, for example, the sender is not capable of adjusting its trade-off 🤷♀️.
3.5.2.2. Point-to-Multipoint Using Multicast or Translators
RTCP Multicast is used either with media multicast according to Topo-Multicast, or following RFC 3550's translator model according to Topo-Translator 📡. In these cases, unsynchronized TSTR messages from different receivers may arrive, possibly with different requested trade-offs (because of different user preferences). This memo does not specify how the media sender tunes its trade-off. Possible strategies include selecting the mean or median of all trade-off requests received, giving priority to certain participants, or continuing to use the previously selected trade-off (e.g., when the sender is not capable of adjusting it) 🧠. Again, all TSTR messages need to be acknowledged by TSTN, and the value conveyed back has to reflect the decision made ✅.
3.5.2.3. Point-to-Multipoint Using RTP Mixer
In this scenario (Topo-Mixer), the RTP mixer receives all TSTR messages and has the opportunity to act on them based on its own criteria 🎛️. In most cases, the mixer should form a "consensus" of potentially conflicting TSTR messages arriving from different participants, and then initiate its own TSTR message(s) to the media sender(s) 🤝. As in the previous scenario, the strategy for forming this "consensus" is up to the implementation. It can, for example, encompass averaging the participants' request values, giving priority to certain participants, or using session default values 🌸.
Even if a mixer or translator performs transcoding, it is very difficult to deliver media with the requested trade-off, unless the content the mixer or translator receives is already close to that trade-off 😬. So if the mixer changes its trade-off, it needs to request that the media sender(s) use the new value, by creating a TSTR of its own. Once it has decided on the trade-off used, it includes that value in the acknowledgement to the downstream requestors 📨. Only in cases where the original source has substantially higher quality (and bit rate) is it likely that transcoding alone can result in the requested trade-off 💎.
3.5.2.4. Reliability
A request and reception acknowledgement mechanism is specified 📬. The Temporal-Spatial Trade-off Notification (TSTN) message tells the requester that its request has been received, and what trade-off is used henceforth. This acknowledgement mechanism is desirable for at least the following reasons 💖:
- A change in the trade-off cannot be directly identified from the media bit stream.
- User feedback cannot be implemented without knowing the chosen trade-off value, according to the media sender's constraints.
- Repetitive sending of messages requesting an unimplementable trade-off can be avoided.
3.5.3. H.271 Video Back Channel Message ✨
okay so ITU-T Rec. H.271 is the one that defines the syntax, the semantics, and the suggested encoder reaction to a Video Back Channel Message 🎀 this memo uses its own structure to transparently pass that message along from the media receiver to the media sender. we're not going to dive deep into every H.271 code point here bestie, you can read the spec text [H.271] for all that 📚
BUT here's the tea 🍵 some H.271 messages look kinda similar to the native AVPF and CCM messages in this memo, and some H.271 messages need extra caution in multicast environments, or are just straight up not usable in multicast or multipoint setups 💀 so Table 1 gives you a quick, simplified glow-up-level overview: the messages currently in H.271, their rough AVPF or Codec Control Message (CCM) lookalikes (the CCMs are the ones specified in this memo), and how multicast-safe we currently think each one is 💜
H.271 msg type AVPF/CCM msg type multicast-safe
--------------------------------------------------------------------
0 (when used for
reference picture
selection) AVPF RPSI No (positive ACK of pictures)
1 picture loss AVPF PLI Yes
2 partial loss AVPF SLI Yes
3 one parameter CRC N/A Yes (no required sender action)
4 all parameter CRC N/A Yes (no required sender action)
5 refresh point CCM FIR Yes
Table 1: H.271 messages and their AVPF/CCM equivalents💅 Note: H.271 message type 0 is NOT a strict equivalent to AVPF's Reference Picture Selection Indication (RPSI). it's really an indication of known-as-correct reference picture(s) at the decoder 🧠 it doesn't command an encoder to use a defined reference picture, which is the kind of control info RPSI is meant to carry. however, it is believed and intended that H.271 message type 0 will be used for the same purpose as AVPF's RPSI, although other use forms are also possible 🦋
okay now because the H.271 messages are kinda opaque (especially when it comes to multicast safety), the following guidelines MUST be followed whenever an implementation wants to use the H.271 video back channel message 🫶
- Implementations using the H.271 feedback message MUST stay in compliance with congestion control principles, as outlined in section 5. no skipping this one bestie 🚦
- An implementation SHOULD utilize the IETF-native messages as defined in [RFC4585] and in this memo instead of similar messages defined in [H.271] 🌸 our current understanding of the similar messages is written up in Table 1 above. one good reason to diverge from that SHOULD statement would be if it's clearly understood that, for a given application and video compression standard, the "similarity" we just talked about isn't actually there, unlike what the table suggests 🙈
- some of the H.271 code points that exist right now are not multicast-safe 😭 so the sensible move is to NOT use the H.271 feedback message type in multicast environments. it MAY be used only when every one of the issues listed later is fully understood by the implementer and properly taken into account by all endpoints. in all other cases, the H.271 message type MUST NOT be used in conjunction with multicast 🚫
- even in centralized multipoint environments, where the mixer should theoretically be able to sort out the issues documented below, building a mixer and cooperative endpoints that actually do this is a very difficult and tedious task 🥴 so H.271 messages MUST NOT be used in centralized multipoint scenarios, unless all the issues mentioned below are fully understood by the implementer and properly taken into account by both the mixer and the endpoints 🧚
Issues to be taken into account when considering the use of H.271 in multipoint environments, bestie, these are the drama points 🎭:
- Different state on different receivers. in lots of environments, you can't guarantee that the decoder state of all media receivers is identical at any given moment 😵 the most obvious reason for that misalignment is a loss on the path to just one of many media receivers. but there are less obvious reasons too, like recent joins to the multipoint conference (whether by joining the multicast group or through extra mixer output). different states can make the media receivers send messages that contradict each other, or make one receiver's H.271 message unhelpful when the media sender sees it for the other receivers. if the media sender reacts naively to these contradicting messages, things can get unpredictable and annoying real quick 🙃
- Combining messages from different media receivers in a media sender is a non-trivial task. these messages might contradict each other, and their transport is unreliable (there may well be other reasons too 🤷♀️). for many H.271 messages (types 0, 2, 3, and 4), the combining algorithm has to be aware of both the network/protocol environment (with respect to congestion) and the media codec being used, since H.271 messages of a given type can mean different things for different media codecs 🧩
- The suppression of requests may need to go beyond the basic mechanisms described in AVPF. those basic mechanisms are driven exclusively by timing and transport considerations at the protocol level 🕰️ but a receiver is often required to hold back (or delay) generating requests based on information it gets from the media stream. for example, it makes no sense for a receiver to issue a FIR while an Intra/IDR picture is already being transmitted 🙄
- when you use the non-multicast-safe messages (for example, H.271 type 0 positive ACKs of received pictures/slices) in larger multicast groups, the media receiver will likely be forced to delay or even skip sending them 😩 from the media sender's point of view, it looks like data wasn't properly received (even though it actually was), and a naively implemented media sender will react to these fake problems when it really shouldn't 🌀
3.5.3.1. Reliability 💌
H.271 Video Back Channel Messages don't require reliable transmission, and confirmation of reception can be derived from the forward video bit stream 🎞️ so no specific reception acknowledgement is specified.
as for re-sending rules, section 3.5.1.1 applies 🔁
3.5.4. Temporary Maximum Media Stream Bit Rate Request and Notification 💖
Okay bestie, so a receiver, translator, or mixer uses the Temporary Maximum Media Stream Bit Rate Request (TMMBR, "timber") 💅 to ask a sender to limit the maximum bit rate for a media stream (see section 2.2) to, or below, the value it provides. The Temporary Maximum Media Stream Bit Rate Notification (TMMBN) carries the media sender's current take on the most limiting subset of the TMMBR limits it has received 🫶. That way, participants can skip sending TMMBRs that wouldn't restrict the sender any further. The main home for TMMBR/TMMBN is an MCU or mixer setup (use case 6), matching Topo-Translator or Topo-Mixer, but it also works for Topo-Point-to-Point 🦋.
Each temporary limit on a media stream is written as a tuple 🎀. The first part is the maximum total media bit rate (as defined in section 2.2) that the media receiver is currently ready to accept for this stream. The second part is the per-packet overhead that the media receiver has observed for this stream at its chosen reference protocol layer ✨.
Here's where it gets spicy 🌶️. As section 2.2 explains, the overhead the TMMBR sender (the media receiver) sees can be different from the overhead the TMMBR receiver (the media sender) sees. This happens when the two ends use different reference protocol layers, or when translators or mixers in the middle change the per-packet overhead. For example, a gateway between the two that converts IPv4 to IPv6 changes the per-packet overhead by 20 bytes 😭. Tunnels can change it too. The problem of varying overhead is also discussed in [RFC3890]. You'll see in the algorithm for using TMMBR that the difference in perceived overhead between the two ends is totally fine, because the calculations use variables that have the same value at both ends, like packet rate and net media rate 💜.
Reporting both the maximum total media bit rate and the per-packet overhead lets different receivers give bit rate and overhead values for different protocol layers. Those could be the IP level, the outer part of a tunnel protocol, or the link layer 🌈. Which layer a peer reports on depends on how integrated that peer is, because it has to be able to pull the info from that layer. For example, an app that has no idea which IP version it's running over can't really figure out the overhead of the IP header, so it won't want to include IP overhead in its overhead or maximum total media bit rate calculation 🙈.
Most peers should be able to report values at least for the IP layer 💅. In some implementations, it may also be helpful to include link layer information, which allows for a more precise overhead calculation and better use of connectivity resources ⭐.
The Temporary Maximum Media Stream Bit Rate messages are generic, so they can be used on any RTP packet stream 🧚. That's what makes them different from the other codec control messages in this specification, which only apply to specific media types or payload formats. TMMBR works at the transport level, along with the requirements the transport puts on the media encoding 🌸.
The reasoning below assumes that the participants have negotiated a session maximum bit rate using a signaling protocol 🍓. That value can be global, for example in point-to-point, multicast, or translator cases. It can also be local, between the participant and its peer or mixer. Either way, the bit rate negotiated in signaling is the one the participant guarantees it can handle (depacketize and decode). In practice, the participant's connectivity also influences the negotiated value. It doesn't make much sense to negotiate a total media bit rate that your network interface can't support 💀.
It's also a good idea to have negotiated a maximum packet rate for the session or sender ✨. [RFC3890] provides an SDP [RFC4566] attribute for this purpose. However, that attribute can't be used in RTP sessions established using offer/answer [RFC3264]. So this memo specifies an optional maximum packet rate signaling parameter instead 🦋.
An established maximum total media bit rate can be changed at any time, as long as the timing rules for sending feedback messages are followed 👑. The limit can change to any value between zero and the session maximum that was negotiated during session establishment signaling. Even if a sender gets a TMMBR message that allows an increase in bit rate, all increases must be governed by a congestion control mechanism 🔥. TMMBR only indicates known limitations, usually in the local environment, and doesn't guarantee anything about the full path 🫣. Also, any increase in a TMMBR-established bit rate limit is only carried out after a certain delay from when the TMMBN message announcing the increase was sent. That delay is at least twice the longest RTT known by the media sender, plus the media sender's calculated wait time before it can send another TMMBR message for this session, based on AVPF timing rules ⏳. The delay exists so other session participants have time to make their own bit rate requirements known, which may be lower 🎀.
If the new value indicated by TMMBR is likely to stay valid for the rest of the session, the TMMBR sender is expected to renegotiate the session upper limit using the session signaling protocol 💅.
3.5.4.1. Behavior for Media Receivers Using TMMBR ✨
This section is an informal description of behaviour that is described more precisely in section 4.2 🎀.
A media sender starts the session limited by the maximum media bit rate and maximum packet rate negotiated in session signaling, if there are any 🌸. Note that this value may have been negotiated for a different protocol layer than the one the participant uses in its TMMBR messages 🧐. Each media receiver picks a reference protocol layer, forms an estimate of the overhead it's observing at that level (or estimates it if no packets have been seen yet), and decides the maximum total media bit rate it can accept. That decision takes into account its own limitations and any transport path limitations it might know about 🦋. If the current limitations are more restrictive than what was agreed in session signaling, the media receiver reports its initial estimate of these two quantities to the media sender using a TMMBR message. Overall message traffic goes down because one TMMBR message can include tuples for multiple media senders 🫶.
The media sender uses an algorithm like the one in section 3.5.4.2 to pick which tuples it received are the most limiting (the bounding set, as defined in section 2.2) 🔥. It adjusts its operation to stay within the feasible region (as defined in section 2.2), and it also sends a TMMBN to the media receivers with the chosen bounding set 💖. That notification also says who was responsible for each tuple in the bounding set, meaning the "owner"(s) of the limitation. A session participant that owns no tuple in the bounding set is called a "non-owner" 🙈.
If a media receiver doesn't own any tuple in the bounding set reported by the TMMBN, it runs the same algorithm as the media sender to check whether its current estimated (maximum total media bit rate, overhead) tuple would make it into the bounding set if the media sender knew it. If so, it sends a TMMBR reporting that tuple's value to the sender. Otherwise, it takes no action for the moment 😌. Its estimated tuple values may change over time, or it may receive a new TMMBN. When either happens, it runs the algorithm again to decide whether it needs to send a TMMBR 🌈.
If, on the other hand, a media receiver owns a tuple in the reported bounding set, it takes no action until its estimate of its own tuple values changes. At that point, it sends a TMMBR to the media sender to report the changed values 🍓.
A media receiver can switch between being an owner and a non-owner of a bounding tuple from one TMMBN message to the next 🔄. So it must check the contents of each TMMBN to figure out what to do next 👀.
Implementations can use other algorithms of their choosing, as long as the bit rate limitations that come from the exchange of TMMBR and TMMBN messages are at least as strict (at least as low, in the bit rate dimension) as the ones from the algorithm above 💅.
Obviously, in point-to-point cases with only one media receiver, that receiver becomes the "owner" once it gets the first TMMBN in response to its own TMMBR, and it stays the "owner" for the rest of the session 👑. So when it's known there will always be only one media receiver, the algorithm above isn't required. Media receivers that know they're the only ones in a session can send TMMBR messages with bit rate limits both higher and lower than the previously notified limit, at any time (subject to the AVPF [RFC4585] RTCP RR send timing rules) 🦋. However, it can be hard for a session participant to tell if it's the only receiver in the session 🫣. Because of that, any implementation of TMMBR is required to include the algorithm described in the next section or a stricter equivalent 💜.
3.5.4.2. Algorithm for Establishing Current Limitations 🧮
This section introduces an example algorithm for calculating a session limit. Other algorithms can be used, as long as the result of the calculation is at least as restrictive as the result this algorithm gives 💖.
First, let's think about what it means to use a tuple to limit the media sender's behavior 🤔. The bit rate and the overhead value make a two-dimensional solution space for calculating the bit rate of media streams. Luckily, the two variables are linked. Specifically, the bit rate available for RTP payloads equals the TMMBR reported bit rate minus the packet rate used, multiplied by the TMMBR reported overhead converted to bits 🌸. So when different bit rate/overhead combinations need to be considered, the packet rate determines the correct limit. An example makes this clearer 🎀:
Example:
Receiver A: TMMBR_max total BR = 35 kbps, TMMBR_OH = 40 bytes
Receiver B: TMMBR_max total BR = 40 kbps, TMMBR_OH = 60 bytesFor a given packet rate (PR), the bit rate available for media payloads in RTP will be:
Max_net media_BR_A =
TMMBR_max total BR_A - PR * TMMBR_OH_A * 8 ... (1)
Max_net media_BR_B =
TMMBR_max total BR_B - PR * TMMBR_OH_B * 8 ... (2)For a PR = 20, these calculations give Max_net media_BR_A = 28600 bps and Max_net media_BR_B = 30400 bps, which suggests receiver A is the limiting one for this packet rate 👀. However, at a certain PR there's a switchover point where receiver B becomes the limiting one 🔥. You can find the switchover point by setting Max_media_BR_A equal to Max_media_BR_B and solving for PR:
TMMBR_max total BR_A - TMMBR_max total BR_B
PR = ------------------------------------------- ... (3)
8*(TMMBR_OH_A - TMMBR_OH_B)okay so plug in the numbers from before and equation (3) gives you 31.25 as the switchover point between the two limits 🎀. For packet rates below 31.25 per second, receiver A is the limiting one. Above that, receiver B is more limiting 💅. Anyone building something that controls media encoding and packetization has to account for this, bestie 🫶. Multiple TMMBR limits can apply to the trade-off between net media bit rate and packet rate, and which one wins depends on the packet rate you're looking at ✨.
This also changes how TMMBR has to work 🦋. First, a media sender might get several TMMBR tuples limiting it at once. Second, every session participant (the media sender and the receivers) needs to be able to tell whether a given tuple will become a limitation on the media sender, or whether the limits already in place are stricter than the new values 👀. If nobody can make that call, TMMBR suppression just doesn't work, and that's a full-on disaster 💀.
The core idea is kinda iconic 🌈. Each TMMBR tuple is basically the equation of a straight line (see equations (1) and (2)) in a graph where packet rate runs along the X-axis and net bit rate runs along the Y-axis 📈. The lower envelope of all those lines, together with the X and Y axes, makes a polygon. Any point inside that polygon is a combo of packet rate and bit rate that satisfies every TMMBR constraint 🫧. The highest feasible packet rate is the smaller of two things: the point where the bounding polygon hits the X-axis, or the session maximum packet rate (SMAXPR, measured in packets per second) from signaling, if there is one 💜. Usually a media sender would rather run below that theoretical max, so the actual media content reaches the receivers faster 🚀. So the whole point of the algorithm is to figure out which TMMBR tuples form the bounding set, so the feasible region is clearly outlined and the sender can pick its favourite spot inside it 🍓
Figure 1 shows a bounding polygon made by TMMBR tuples A and B 🎨. Tuple C sits outside that polygon, so it doesn't matter for the feasible trade-offs between media rate and packet rate 🙅♀️. The line labeled ss..s is the packet rate limit set by SMAXPR from session setup. In Figure 1, the limit from tuple B is stricter than SMAXPR. It could totally be the other way around, though, in which case the polygon gets cut off on the right by the vertical SMAXPR line 🔥
Net ^
Media|a c b s
Bit | a c b s
Rate | a c b s
| a cb s
| a c s
| a bc s
| a b c s
| ab c s
| Feasible b c s
| region ba s
| b a s c
| b s c
| b s a
| bs
+------------------------------>
Packet rate
Figure 1 - Geometric Interpretation of TMMBR TuplesNotice that the slopes of the lines making up the bounding polygon get more and more negative as packet rate goes up 📉. Also, if you rearrange equations (1) and (2) a little, they end up in the canonical form:
y = mx + bwhere m is the slope and has value equal to the negative of the tuple overhead (in bits), and b is the y-intercept and has value equal to the tuple maximum total media bit rate.
So here's the vibe check 🧠: when you're processing TMMBR tuples to pick the initial bounding set, sort them and process them by increasing overhead. Once a tuple joins the bounding set, every tuple that hasn't been selected yet and has lower overhead can be tossed, because the next side of the polygon has to be steeper (so the matching TMMBR needs higher overhead) than the one you just added 🙌
Line cc..c in Figure 1 shows another principle 💖. It's parallel to line aa..a but has a higher Y-intercept, meaning its TMMBR tuple has a higher maximum total media bit rate. Since cc..c sits outside the bounding polygon, if two TMMBR tuples have the same overhead, the one with the higher maximum total media bit rate can't be part of the bounding set and can get set aside 🗑️✨
Two more observations finish the algorithm 🦋. Moving from the left, the corners of the bounding polygon (the intersection points between neighbouring sides) sit at higher and higher packet rates. Also moving from the left, each line in the bounding set crosses the X-axis at a lower packet rate than the one before it 📉
Okay, here's the full algorithm, no cap 💅. It works with two lists of TMMBR tuples: the candidate list X and the selected list Y, both sorted by increasing overhead value. The algorithm finishes once every member of X has been discarded or removed for processing 🏁. Membership in the selected list Y is probationary until the whole algorithm is done 🫧. Each member of the selected list gets an intersection value, which is the packet rate where its line meets the line of the previous tuple in the selected list 💜. Each member also gets a maximum packet rate value, which is the smaller of SMAXPR (if there is one) and the packet rate where that tuple's line crosses the X-axis 🌸
When the algorithm finishes, the selected list is exactly the bounding set from section 2.2 🎀
Initial Algorithm
This one's for the media sender 💖. It runs once the sender has received one or more TMMBRs, but before it has figured out a bounding set for the first time ✨
- Sort the TMMBR tuples by increasing overhead. This is the initial candidate list X.
- When multiple tuples in the candidate list have the same overhead value, discard all of them except the one with the lowest maximum total media bit rate value 🙈
- Select and remove from the candidate list the TMMBR tuple with the lowest maximum total media bit rate value. If more than one tuple has that value, pick the one with the highest overhead value. This is the first member of the selected list Y. Set its intersection value to zero. Then calculate its maximum packet rate as the minimum of SMAXPR (if available) and the value from the formula below, which is the packet rate where the corresponding line crosses the X-axis 👑
Max PR = TMMBR max total BR / (8 * TMMBR OH) ... (4)- Discard from the candidate list all tuples with a lower overhead value than the selected tuple 🧹
- Remove the first remaining tuple from the candidate list so it can be processed. Call this the current candidate 🫶
- Calculate the packet rate PR at the intersection of the line from the current candidate with the line from the last tuple in the selected list Y, using equation (3) 📐
- If the calculated PR is equal to or lower than the intersection value stored for the last tuple in the selected list, discard the last tuple of the selected list and go back to step 6, keeping the same current candidate 😤
Note that the choice of the initial member of the selected list Y in step 3 guarantees the selected list never gets emptied by this process, so the algorithm will eventually (if not immediately) fall through to step 8 ✨
- (This step is reached when the calculated PR of the current candidate is greater than the intersection value of the current last member of the selected list Y.) If the calculated PR of the current candidate is lower than the maximum packet rate associated with the last tuple in the selected list, add the current candidate to the end of the selected list. Store PR as its intersection value. Then calculate its maximum packet rate as the lesser of SMAXPR (if available) and the maximum packet rate from equation (4) 🌈
- If any tuples remain in the candidate list, go back to step 5 🔁
Incremental Algorithm
The previous algorithm covered the initial case, where no selected list had been created yet 🌸. It also only applied to the media sender. When a selected list already exists at either the media sender or a media receiver, there are two other cases to think about:
- when a TMMBR tuple that isn't currently in the selected list is a candidate for addition;
- when the values change in a TMMBR tuple that is currently in the selected list.
At the media receiver, these cases match the non-owner and owner of a tuple in the TMMBN-reported bounding set 🦋. In either case, you can update the selected list to account for the new or changed tuple using the basic algorithm above, with one tweak: the initial candidate set consists only of the existing selected list plus the new or changed tuple 🫧. There's some extra optimization on top of starting with a smaller candidate set, using the following observations 🧠
The first observation: if the new or changed candidate ends up in the new selected list, it might cause zero or more other tuples to get dropped. If more than one other tuple gets dropped, the dropped tuples will be consecutive 💅. You can see this geometrically by imagining a new line that cuts off a series of segments from the existing bounding polygon. The cut-off segments are connected one to the next, which is the geometric version of consecutive tuples in a list sorted by overhead value 📈. Beyond the dropped set in either direction, every tuple from the earlier selected list will still be in the updated one 💜. The second observation: apart from the new candidate, the order of the tuples left in the updated selected list doesn't change, because their overhead values haven't changed 🎀
So the consequence is this: once you've figured out where the new candidate goes and how far the dropped set extends (if at all), you can copy the remaining tuples straight from the candidate list into the selected list, keeping their order 🙌. That leads to this modified algorithm:
- Run steps 1-4 of the basic algorithm.
bestie, the selected list is where the magic happens 🦋
- If the new candidate survived steps 2 and 4 and became the new first member of the selected list, run steps 5-9 on subsequent candidates until another candidate is added to the selected list. Then move all remaining candidates to the selected list, preserving their order. 💅
- If the new candidate survived steps 2 and 4 but did not become the new first member, start by moving all tuples in the candidate list with lower overhead values than the new candidate to the selected list, preserving their order. Run steps 5-9 for the new candidate, except the intersection values and maximum packet rates for the tuples on the selected list have to be calculated on the fly, because they weren't stored before. Keep going only until a later tuple gets added to the selected list, then move all remaining candidates to the selected list, preserving their order. 🫶
Ngl, the new candidate could get added to the selected list and then get dropped again when the next tuple is processed 💀. It's easy to see that in this case the new candidate doesn't displace any of the earlier tuples in the selected list. ASCII art is kinda bad at showing this, sorry 🙈. Line cc..c in Figure 1 would be an example if it had a steeper slope (tuple C had a higher overhead value) but still crossed line aa..a beyond where line aa..a crosses line bb..b. 🌈
This algorithm is approximate, because it ignores tuples outside the selected list. Here's how those sneaky tuples can become relevant 👀. Look at Figure 1 and imagine the maximum total media bit rate in tuple A goes up enough that line aa..a moves outside line cc..c. Tuple A stays in the bounding set the media sender calculated. But once it sends a new TMMBN, media receiver C runs the algorithm and realizes its tuple C should now enter the bounding set. It sends a TMMBR to the media sender, which redoes its calculation and comes to the right conclusion ✨.
The rules of section 4.2 require that the media sender hold off on raising its sending rate until media receivers have had a chance to respond to the TMMBN. In the example I just gave, this delay means the relaxation of tuple A doesn't actually lead to an attempt to send media faster than the capacity at C. Crisis averted 🎀
3.5.4.3. Use of TMMBR in a Mixer-Based Multipoint Operation
Picture a small mixer-based multiparty conference, as shown in Topo-Mixer of [RFC5117] 🌸. Everyone has agreed on a common maximum bit rate this session can use. The conference runs over a bunch of unicast paths between the participants and the mixer. Each participant can keep an eye on the congestion situation on their own path, and so can the mixer, using things like RTCP receiver reports (RRs) or the transport protocol, e.g., Datagram Congestion Control Protocol (DCCP) [RFC4340]. But any one participant has zero knowledge of the congestion on the connections to everyone else 😭. Worse, without mechanisms like the ones in this document, the mixer (which does know the congestion on every connection it manages) has no standardized way to tell media senders to slow down, unless it forges its own receiver reports, which is undesirable 🙅♀️. In principle, a mixer facing this situation is obliged to thin or transcode streams intended for connections that detected congestion.
In practice, media-aware stream thinning is a very difficult and cumbersome operation, and it adds undesirable delay 💀. If the mixer is media-unaware, the reproduced media quality drops to unacceptable levels really fast. So a way to slow senders down even when there's no congestion on their connections to the mixer is desirable ✨.
To let the mixer throttle traffic on individual links without transcoding, there's a need for a mechanism that lets the mixer ask a participant's media encoders to limit the media stream bit rate they're currently generating. TMMBR provides the required mechanism 👑. When the mixer detects congestion between itself and a given participant, it runs the following procedure:
- It starts thinning the media traffic to the congested participant down to the supported bit rate.
- It uses TMMBR to ask the media sender(s) to reduce the total media bit rate they send to the mixer, to a value that complies with congestion control principles for the slowest link. Slow here refers to the available bandwidth / bit rate / capacity and packet rate after congestion control.
- As soon as the sending part has reduced the bit rate, the mixer stops stream thinning implicitly, because once the stream complies with congestion control there's no need for it anymore 🦋.
This combo of stream thinning as an immediate reaction, followed up by a quick control mechanism, seems like a reasonable compromise between media quality and fighting congestion. Iconic balance, honestly 🫶.
3.5.4.4. Use of TMMBR in Point-to-Multipoint Using Multicast or Translators
In these topologies, which match Topo-Multicast or Topo-Translator, RTCP RRs are sent globally 🌍. That means all participants can spot transmission problems like congestion on a medium timescale. Since every media sender knows the congestion situation of every media receiver, the reason for using TMMBR in the previous section doesn't apply here 🙈. Still, the congestion control response can get better when the unicast links use congestion controlled transport protocols (such as TCP or DCCP). A peer can also report local limitations to the media sender, just saying 💅.
3.5.4.5. Use of TMMBR in Point-to-Point Operation
In use case 7, TMMBR can help performance when the known upper limit of the bit rate changes 🎀. In this use case, the signaling protocol has already set an upper limit for the session and total media bit rates. But when the transport link bit rate drops, a receiver can avoid serious congestion by sending a TMMBR to the sending side. So TMMBR is useful for putting restrictions on the application and placing congestion control in the right ballpark. However, TMMBR usually can't give the continuously quick feedback loop that real congestion control needs. Its semantics don't match congestion control either, given its different purpose 😤. For these reasons, TMMBR SHALL NOT be used as a substitute for congestion control. Not a replacement, bestie, periodt 💜.
3.5.4.6. Reliability
You can't tell how a media sender reacted to a TMMBR message just by looking at the media stream 👀. So a more explicit mechanism is needed to avoid unnecessary re-sending of TMMBR messages. A statistically based retransmission scheme would only give statistical guarantees that the request was received. It also wouldn't stop retransmitting messages that were already received. And it wouldn't make it easy to suppress other participants' requests either. So a mechanism based on explicit notification is used instead 🦋.
When a media sender gets a TMMBR, it sends back a TMMBN containing the current bounding set, and says which session participants own that limit. In multicast scenarios, this lets everyone else suppress any request they might have, as long as their limitations are less strict than the current ones (i.e., they define lines lying outside the feasible region as defined in section 2.2) 🙌. Keeping and notifying only the bounding set of tuples keeps message sizes and media sender state small 🍓. A media sender only keeps state for the SSRCs of the current owners of the bounding set of tuples. All other requests and their sources aren't saved. Once the bounding set is established, new TMMBR messages should be generated only by owners of the bounding tuples and by other entities that determine (by applying the algorithm of section 3.5.4.2 or its equivalent) that their limitations should now be part of the bounding set 🧚.
4. RTCP Receiver Report Extensions ✨
This memo specifies six new feedback messages 💖. The Full Intra Request (FIR), Temporal-Spatial Trade-off Request (TSTR), Temporal-Spatial Trade-off Notification (TSTN), and Video Back Channel Message (VBCM) are "Payload Specific Feedback Messages" as defined in section 6.3 of AVPF [RFC4585]. The Temporary Maximum Media Stream Bit Rate Request (TMMBR) and Temporary Maximum Media Stream Bit Rate Notification (TMMBN) are "Transport Layer Feedback Messages" as defined in section 6.2 of AVPF.
The new feedback messages are defined in the following subsections, following a similar structure to sections 6.2 and 6.3 of the AVPF specification [RFC4585] 🎀.
4.1. Design Principles of the Extension Mechanism
RTCP was originally introduced as a channel to convey presence, reception quality statistics, and hints on the desired media coding 🌸. A limited set of media control mechanisms came in early RTP payload formats for video, for example in RFC 2032 [RFC2032] (which was obsoleted by RFC 4587 [RFC4587]). However, this specification, for the first time, suggests a two-way handshake for some of its messages. There's a danger this could be misunderstood as a precedent for using RTCP as an RTP session control protocol 😱. To prevent that misunderstanding, this subsection clarifies the scope of the extensions in this memo, and it strongly suggests that future extensions follow the rationale spelled out here, or compellingly explain why they diverge from it 💅.
In this memo, and in AVPF [RFC4585], only messages that:
a) have comparatively strict real-time constraints, which prevent the use of mechanisms such as a SIP re-invite in most application scenarios (the real-time constraints are explained separately for each message where necessary);
b) are multicast-safe, in that the reaction to potentially contradicting feedback messages is specified, as necessary for each message; and
c) are directly related to activities of a certain media codec, class of media codecs (e.g., video codecs), or a given RTP packet stream
have been included.
In this memo, a two-way handshake is introduced only for messages where:
a) a notification or acknowledgement is required due to their nature. An analysis to determine whether this requirement exists has been performed separately for each message.
b) the notification or acknowledgement cannot easily be derived from the media bit stream.
All messages in AVPF [RFC4585] and in this memo present their contents in a simple, fixed binary format. This keeps things friendly for media receivers that haven't implemented higher control protocol functionality (SDP, XML parsers, and such) in their media path 🧸.
Messages that don't follow the design principles just described aren't an appropriate use of RTCP or of the Codec Control Framework defined in this document. Big no-no, bestie 🚫💜
4.2. Transport Layer Feedback Messages 🦋
okay so, as laid out in section 6.1 of RFC 4585 [RFC4585], transport layer feedback messages get identified by the RTCP packet type value RTPFB (205) 💌
In AVPF, only one message of this category had been defined. This memo adds two more, and they're told apart by the feedback message type (FMT) parameter, like this 👑
Assigned in AVPF [RFC4585]:
1: Generic NACK
31: reserved for future expansion of the identifier number space
Assigned in this memo:
2: reserved (see note below)
3: Temporary Maximum Media Stream Bit Rate Request (TMMBR)
4: Temporary Maximum Media Stream Bit Rate Notification (TMMBN)
Available for assignment:
0: unassigned
5-30: unassigned🙈 Note on FMT=2: early versions of AVPF [RFC4585] reserved FMT=2 for a code point that got removed later. Someone pointed out that there might still be implementations out in the wild using that value, because they followed the expired document. There's plenty of numbering space, so we mark FMT=2 as reserved. That way we avoid possible interoperability problems with any of those early implementations 🫶
The next subsection defines the formats of the Feedback Control Information (FCI) entries for the TMMBR and TMMBN messages, and it also covers the behaviour expected at the media sender and receiver ✨
4.2.1. Temporary Maximum Media Stream Bit Rate Request (TMMBR)
The Temporary Maximum Media Stream Bit Rate Request is identified by the RTCP packet type value PT=RTPFB and FMT=3 🎀
The FCI field of a Temporary Maximum Media Stream Bit Rate Request (TMMBR) message SHALL contain one or more FCI entries 💅
4.2.1.1. Message Format
The Feedback Control Information (FCI) consists of one or more TMMBR FCI entries with the following syntax 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MxTBR Exp | MxTBR Mantissa |Measured Overhead|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2 - Syntax of an FCI Entry in the TMMBR MessageHere's what each field does 👀
SSRC (32 bits): the SSRC value of the media sender that's being asked to obey the new maximum bit rate 🌸MxTBR Exp (6 bits): the exponential scaling of the mantissa for the maximum total media bit rate value. It's an unsigned integer[0..63]💜MxTBR Mantissa (17 bits): the mantissa of the maximum total media bit rate value, as an unsigned integer 🍓Measured Overhead (9 bits): the measured average packet overhead value in bytes. The measurement SHALL be done according to the description in section 4.2.1.2. It's an unsigned integer[0..511]🔥
Now the math, which is honestly so clean 😭 The maximum total media bit rate (MxTBR) value in bits per second gets calculated from the MxTBR exponent (exp) and mantissa like this:
MxTBR = mantissa * 2^expThis gives you 17 bits of resolution across the range 0 to 131072*2^63 (approximately 1.2*10^24) 🌈
And last but not least, the length of the TMMBR feedback message SHALL be set to 2+2*N, where N is the number of TMMBR FCI entries ⭐
4.2.1.2. Semantics ✨
Behaviour at the Media Receiver (Sender of the TMMBR) 🎀
okay bestie, so TMMBR is how a media receiver says "hey, I've got a transport-related limit over here" 💅 It's a tuple with two components. The first value is the highest bit rate per sender of a media stream, available at a receiver-chosen protocol layer, which the receiver currently supports in this RTP session 🦋 The second value is the measured header overhead in bytes, as defined in section 2.2 and measured at the chosen protocol layer in the packets received for the stream 🧚 That overhead measurement is a running average (exponentially smoothed, so it's giving smooth vibes 🌈) that gets updated for every packet received for that particular media source (SSRC):
avg_OH (new) = 15/16*avg_OH (old) + 1/16*pckt_OH,where avg_OH is the running (exponentially smoothed) average and pckt_OH is the overhead observed in the latest packet.
If a maximum bit rate has been negotiated through signaling, then the maximum total media bit rate that the receiver reports in a TMMBR message MUST NOT exceed the negotiated value converted to a common basis (i.e., with overheads adjusted to bring it to the same reference protocol layer). No going over the negotiated number, periodt 💀
Within the common packet header for feedback messages (as defined in section 6.1 of [RFC4585]), the SSRC of packet sender field tells you who the request came from 🫶 The SSRC of media source field is not used and SHALL be set to 0 🙈 Inside a particular TMMBR FCI entry, though, the SSRC of media source in the FCI field tells you which media sender that tuple applies to. This is super useful in multicast or translator topologies, where the reporting entity can address all of the media senders in a single TMMBR message using multiple FCI entries. Multitasking queen behaviour 👑
The media receiver SHALL save the contents of the latest TMMBN message received from each media sender 📝
The media receiver MAY send a TMMBR FCI entry to a particular media sender under the following circumstances:
- before any TMMBN message has been received from that media sender;
- when the media receiver has been identified as the source of a bounding tuple within the latest TMMBN message received from that media sender, and the value of the maximum total media bit rate or the overhead relating to that media sender has changed;
- when the media receiver has not been identified as the source of a bounding tuple within the latest TMMBN message received from that media sender, and, after the media receiver applies the incremental algorithm from section 3.5.4.2 or a stricter equivalent, the media receiver's tuple relating to that media sender is determined to belong to the bounding set.
A TMMBR FCI entry MAY be repeated in subsequent TMMBR messages if no Temporary Maximum Media Stream Bit Rate Notification (TMMBN) FCI has been received from the media sender at the time of transmission of the next RTCP packet. The bit rate value of a TMMBR FCI entry MAY be changed from one TMMBR message to the next. The overhead measurement SHALL be updated to the current value of avg_OH each time the entry is sent. Keeping it fresh 🍓
If the value set by a TMMBR message is expected to be permanent, the TMMBR setting party SHOULD renegotiate the session parameters to reflect that using session setup signaling, e.g., a SIP re-invite. Make it official in the setup phase 💖
Behaviour at the Media Sender (Receiver of the TMMBR) 🌸
When it receives a TMMBR message containing an FCI entry relating to it, the media sender SHALL use an initial or incremental algorithm, as applicable, to determine the bounding set of tuples based on the new information. The algorithm used SHALL be at least as strict as the corresponding algorithm defined in section 3.5.4.2. The media sender MAY accumulate TMMBRs over a small interval (relative to the RTCP sending interval) before making this calculation. Batching the vibes ✨
Once it has determined the bounding set of tuples, the media sender MAY use any combination of packet rate and net media bit rate within the feasible region that these tuples describe, to produce a lower total media stream bit rate, as it may need to address a congestion situation or other limiting factors. See section 5 (congestion control) for more discussion.
If the media sender concludes that it can increase the maximum total media bit rate value, it SHALL wait before actually doing so, for a period long enough to allow a media receiver to respond to the TMMBN if it determines that its tuple belongs in the bounding set. This delay period is estimated by the formula:
2 * RTT + T_Dither_Max,where RTT is the longest round trip time known to the media sender and T_Dither_Max is defined in section 3.4 of [RFC4585]. Even in point-to-point sessions, a media sender MUST obey the aforementioned rule 🛑, as it is not guaranteed that a participant is able to determine correctly whether all the sources are co-located in a single node, and are coordinated. Nobody gets to skip the wait, not even in 1:1 calls 😭
A TMMBN message SHALL be sent by the media sender at the earliest possible point in time, in response to any TMMBR messages received since the last sending of TMMBN. The TMMBN message indicates the calculated set of bounding tuples and the owners of those tuples at the time of the transmission of the message 📣
An SSRC may time out according to the default rules for RTP session participants, i.e., the media sender has not received any RTP or RTCP packets from the owner for the last five regular reporting intervals. An SSRC may also explicitly leave the session, with the participant indicating this through the transmission of an RTCP BYE packet or using an external signaling channel. If the media sender determines that the owner of a tuple in the bounding set has left the session, the media sender SHALL transmit a new TMMBN containing the previously determined set of bounding tuples, but with the tuple belonging to the departed owner removed. Ghosted, so we update the list 👻
A media sender MAY proactively initiate the equivalent to a TMMBR message to itself, when it is aware that its transmission path is more restrictive than the current limitations. As a result, a TMMBN indicating the media source itself as the owner of a tuple is being sent, thereby avoiding unnecessary TMMBR messages from other participants. However, like any other participant, when the media sender becomes aware of changed limitations, it is required to change the tuple, and to send a corresponding TMMBN. Self-awareness is a whole vibe 🧘
Discussion 💜
Due to the unreliable nature of transport of TMMBR and TMMBN, the above rules may lead to the sending of TMMBR messages that appear to disobey those rules. Furthermore, in multicast scenarios it can happen that more than one "non-owning" session participant may determine, rightly or wrongly, that its tuple belongs in the bounding set. This is not critical for a number of reasons:
- a) If a TMMBR message is lost in transmission, either the media sender sends a new TMMBN message in response to some other media receiver or it does not send a new TMMBN message at all. In the first case, the media receiver applies the incremental algorithm and, if it determines that its tuple should be part of the bounding set, sends out another TMMBR. In the second case, it repeats the sending of a TMMBR unconditionally. Either way, the media sender eventually gets the information it needs. 🙌
- b) Similarly, if a TMMBN message gets lost, the media receiver that has sent the corresponding TMMBR does not receive the notification and is expected to re-send the request and trigger the transmission of another TMMBN. Retry energy 🔁
- c) If multiple competing TMMBR messages are sent by different session participants, then the algorithm can be applied taking all of these messages into account, and the resulting TMMBN provides the participants with an updated view of how their tuples compare with the bounded set. Everyone gets the group chat update 💬
- d) If more than one session participant happens to send TMMBR messages at the same time and with the same tuple component values, it does not matter which of those tuples is taken into the bounding set. The losing session participant will determine, after applying the algorithm, that its tuple does not enter the bounding set, and will therefore stop sending its TMMBR. Graceful exit, no drama 🫡
It is important to consider the security risks involved with faked TMMBRs. See the security considerations in section 6. This one is serious, so no bestie left behind 🚨
As indicated already, the feedback messages may be used in both multicast and unicast sessions in any of the specified topologies. However, for sessions with a large number of participants, using the lowest common denominator, as required by this mechanism, may not be the most suitable course of action. Large sessions may need to consider other ways to adapt the bit rate to participants' capabilities, such as partitioning the session into different quality tiers or using some other method of achieving bit rate scalability. Big rooms need bigger plans 🌈
4.2.1.3. Timing Rules ⏰
The first transmission of the TMMBR message MAY use early or immediate feedback in cases when timeliness is desirable. Any repetition of a request message SHOULD use regular RTCP mode for its transmission timing. Fast first hello, then chill repeats 🦋
4.2.1.4. Handling in Translators and Mixers 🧚
Media translators and mixers will need to receive and respond to TMMBR messages, as they are part of the chain that provides a certain media stream to the receiver. The mixer or translator may act locally on the TMMBR and thus generate a TMMBN to indicate that it has done so. Alternatively, in the case of a media translator, it can forward the request, or in the case of a mixer, generate one of its own and pass it forward. In the latter case, the mixer will need to send a TMMBN back to the original requestor to indicate that it is handling the request. Middle people, but make it communicative 💅
4.2.2. Temporary Maximum Media Stream Bit Rate Notification (TMMBN) 🌟
The Temporary Maximum Media Stream Bit Rate Notification is identified by RTCP packet type value PT=RTPFB and FMT=4. That's the TMMBN's ID card 🪪
The FCI field of the TMMBN feedback message may contain zero, one, or more TMMBN FCI entries. Zero, one, or a whole bunch, it's giving flexible 🎀
4.2.2.1. Message Format
okay so the FCI (Feedback Control Information) 💖 holds zero, one, or more TMMBN FCI entries, and each entry is laid out like this 🎀:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MxTBR Exp | MxTBR Mantissa |Measured Overhead|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3 - Syntax of an FCI Entry in the TMMBN Message
Here's what each field does ✨:
SSRC(32 bits): the SSRC value of the "owner" of this tuple.MxTBR Exp(6 bits): the exponential scaling of the mantissa for the maximum total media bit rate value. The value is an unsigned integer[0..63].MxTBR Mantissa(17 bits): the mantissa of the maximum total media bit rate value, as an unsigned integer.Measured Overhead(9 bits): the measured average packet overhead value in bytes, as an unsigned integer[0..511].
So the TMMBN FCI carries entries for the bounding tuples 🦋. Each entry names the owner by SSRC, then gives the applicable maximum total media bit rate and overhead value. Lowkey the whole message is just a list of these little tuples, iykyk 🌸.
The length of the TMMBN message SHALL be set to 2+2*N, where N is the number of TMMBN FCI entries.
4.2.2.2. Semantics
okay this feedback message is how the senders hear back about their TMMBR messages 📣. It tells them that one or more TMMBR messages came in, or that an owner has left the session. It also shows everyone the current set of bounding tuples and who the "owners" of those tuples are 👑.
In the common packet header for feedback messages (as defined in section 6.1 of [RFC4585]), the SSRC of packet sender field tells you who sent the notification. The SSRC of media source is not used and SHALL be set to 0 💅.
A TMMBN message SHALL be scheduled for transmission after a TMMBR message with an FCI entry identifying this media sender is received. Only a single TMMBN SHALL be sent, even if more than one TMMBR message comes in between scheduling the transmission and actually sending the TMMBN. The TMMBN message shows the bounding tuples and their owners at the moment it's transmitted. The bounding tuples included SHALL be the set reached by applying the applicable algorithm of section 3.5.4.2 (or an equivalent one) to the previous bounding set, if any, plus the tuples received in TMMBR messages since the last TMMBN was transmitted 🧚.
The reception of a TMMBR message SHALL still result in the transmission of a TMMBN message, even if the newly reported TMMBR tuple is not accepted into the bounding set after the algorithm runs. In that case the bounding tuples and their owners do not change, unless the TMMBR came from an owner of a tuple within the previously calculated bounding set. This is the bit that keeps everyone in the loop 🫶: it lets session participants who missed the last TMMBN still get a correct view of this media sender's state.
As indicated in section 4.2.1.2, when a media sender determines that an "owner" of a bounding tuple has left the session, that tuple is removed from the bounding set, and the media sender SHALL send a TMMBN message indicating the remaining bounding tuples. If there are no remaining bounding tuples, a TMMBN without any FCI SHALL be sent to indicate this. Without a remaining bounding tuple, the maximum media bit rate and maximum packet rate negotiated in session signaling, if any, apply 🌈.
Note: if any media receivers remain in the session, this last will be a temporary situation. The empty TMMBN will cause every remaining media receiver to determine that its limitation belongs in the bounding set and send a TMMBR in consequence.
In unicast scenarios (that is, where a single sender talks to a single receiver), the ownership algorithm above basically collapses 💀: the media receiver becomes the "owner" of the one bounding tuple as soon as it has issued its first TMMBR message. Simple, no drama 🍓.
4.2.2.3. Timing Rules
The TMMBN acknowledgement SHOULD be sent as soon as the timing rules for the session allow. Immediate or early feedback mode SHOULD be used for these messages ⭐.
4.2.2.4. Handling by Translators and Mixers
As discussed in section 4.2.1.4, mixers or translators may need to issue TMMBN messages as responses to TMMBR messages for SSRCs handled by them 🌸.
4.3. Payload-Specific Feedback Messages
okay now we're into the payload-specific feedback (PSFB) side of things 💜. As specified by section 6.1 of RFC 4585 [RFC4585], Payload-Specific FB messages are identified by the RTCP packet type value PSFB (206).
AVPF [RFC4585] defines three payload-specific feedback messages and one application layer feedback message. This memo adds four more payload-specific feedback messages 🎀. All of them are identified by the FMT parameter, like so:
Assigned in [RFC4585]:
1: Picture Loss Indication (PLI)
2: Slice Lost Indication (SLI)
3: Reference Picture Selection Indication (RPSI)
15: Application layer FB message
31: reserved for future expansion of the number space
Assigned in this memo:
4: Full Intra Request (FIR) Command
5: Temporal-Spatial Trade-off Request (TSTR)
6: Temporal-Spatial Trade-off Notification (TSTN)
7: Video Back Channel Message (VBCM)
Unassigned:
0: unassigned
8-14: unassigned
16-30: unassignedThe following subsections define the new FCI formats for the payload-specific feedback messages 🦋.
4.3.1. Full Intra Request (FIR)
The FIR message is identified by RTCP packet type value PT=PSFB and FMT=4 ✨.
The FCI field MUST contain one or more FIR entries. Each entry applies to a different media sender, identified by its SSRC 💅.
4.3.1.1. Message Format
The Feedback Control Information (FCI) for the Full Intra Request consists of one or more FCI entries, whose contents are shown in Figure 4. The length of the FIR feedback message MUST be set to 2+2*N, where N is the number of FCI entries.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seq nr. | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4 - Syntax of an FCI Entry in the FIR Message
SSRC(32 bits): the SSRC value of the media sender that is asked to send a decoder refresh point.Seq nr.(8 bits): command sequence number 🔢. The sequence number space is unique for each pairing of the SSRC of the command source and the SSRC of the command target. The sequence number SHALL be increased by 1 modulo 256 for each new command. A repetition SHALL NOT increase the sequence number. The initial value is arbitrary.Reserved(24 bits): all bits SHALL be set to 0 by the sender and SHALL be ignored on reception.
The semantics of this feedback message are independent of the RTP payload type 🌈.
4.3.1.2. Semantics
In the common packet header for feedback messages (as defined in section 6.1 of [RFC4585]), the SSRC of packet sender field tells you who sent the request, and the SSRC of media source is not used and SHALL be set to 0. The SSRCs of the media senders the FIR command applies to are in the corresponding FCI entries. A FIR message MAY contain requests to multiple media senders, using one FCI entry per target media sender 👑.
Upon reception of FIR, the encoder MUST send a decoder refresh point (see section 2.2) as soon as possible.
The sender MUST consider congestion control as outlined in section 5, which MAY restrict its ability to send a decoder refresh point quickly 🙈.
FIR SHALL NOT be sent as a reaction to picture losses -- it is RECOMMENDED to use PLI [RFC4585] instead. FIR SHOULD be used only in situations where not sending a decoder refresh point would render the video unusable for the users. Not me sending FIR every time a packet drops 😭.
A typical example where sending FIR is appropriate is when, in a multipoint conference, a new user joins the session and no regular decoder refresh point interval is established. Another example is a video switching MCU that changes streams. Here, normally, the MCU issues a FIR to the new sender so it emits a decoder refresh point. The decoder refresh point normally includes a Freeze Picture Release (defined outside this specification), which restarts the rendering process of the receivers. Both techniques mentioned are commonly used in MCU-based multipoint conferences 🎀.
Other RTP payload specifications, such as RFC 2032 [RFC2032], already define a feedback mechanism for certain codecs. An application supporting both schemes MUST use the feedback mechanism defined in this specification when sending feedback. For backward-compatibility reasons, such an application SHOULD also be capable of receiving and reacting to the feedback scheme defined in the respective RTP payload format, if that payload format requires it 🫶.
4.3.1.3. Timing Rules
The timing follows the rules outlined in section 3 of [RFC4585]. FIR commands MAY be used with early or immediate feedback. The FIR feedback message MAY be repeated. If using immediate feedback mode, the repetition SHOULD wait at least one RTT before being sent. In early or regular RTCP mode, the repetition is sent in the next regular RTCP packet ⏰.
4.3.1.4. Handling of FIR Message in Mixers and Translators
A media translator or a mixer that performs media encoding of the content for which the session participant issued a FIR is responsible for acting on it. A mixer acting on a FIR SHOULD NOT forward the message unaltered; instead, it SHOULD issue a FIR itself 🧚♀️.
4.3.1.5. Remarks ✨
okay so, video is basically the ONLY app where FIR is really useful rn 🎬 because it's the only RTP payload that's widely deployed and leans super hard on predicting media across RTP packet boundaries. BUT FIR could also make sense for other media types with the same vibe, meaning cross-frame prediction (whatever a "frame" even is for that media type) 🧚 One possible example is dynamic updates of MPEG-4 scene descriptions. The suggestion is that payload formats for those types should point to FIR and the other message types defined here and in AVPF [RFC4585], instead of inventing their own similar mechanisms in the payload specs. The payload specs might have to explain how their own terminology maps onto the video-centric terms used in this doc 💅
In conjunction with video codecs, FIR messages typically trigger the sending of full intra or IDR pictures. Both are several times larger than predicted (inter) pictures 🍓 Their size is independent of the time they are generated. In most environments, especially when employing bandwidth-limited links, using an intra picture means accepting a delay that is a significant multiple of the typical frame duration. Example: if the sending frame rate is 10 fps, and an intra picture is assumed to be 10 times as big as an inter picture, then a full second of latency has to be accepted 🙈 In that kind of environment there's no need for a super short delay in sending the FIR message. So waiting for the next possible time slot allowed by RTCP timing rules as per [RFC4585] should not have an overly negative impact on system performance. Lowkey very chill 🌸
Mandating a maximum delay for completing the sending of a decoder refresh point would be desirable from an application viewpoint, but it's problematic from a congestion control point of view 😭 "As soon as possible" (as mentioned above) appears to be a reasonable compromise 🫶
In environments where the sender has no control over the codec (e.g., when streaming pre-recorded and pre-coded content), the reaction to this command cannot be specified. One suitable reaction of a sender would be to skip forward in the video bit stream to the next decoder refresh point ⭐ In other scenarios, it may be preferable not to react to the command at all, e.g., when streaming to a large multicast group. Other reactions may also be possible 🦋 When deciding on a strategy, a sender could take into account factors such as the size of the receiving group, the "importance" of the sender of the FIR message (however "importance" may be defined in this specific application), the frequency of decoder refresh points in the content, and so on. However, a session that predominantly handles pre-coded content is not expected to use FIR at all 💜
The relationship between the Picture Loss Indication and FIR is as follows. As discussed in section 6.3.1 of AVPF [RFC4585], a Picture Loss Indication informs the decoder about the loss of a picture and hence the likelihood of misalignment of the reference pictures between the encoder and decoder. That scenario is normally related to losses in an ongoing connection. In point-to-point scenarios, and without advanced error resilience tools, one possible option for an encoder is sending a decoder refresh point. However, there are other options 🧠 One example is that the media sender ignores the PLI, because the embedded stream redundancy is likely to clean up the reproduced picture within a reasonable amount of time. The FIR, in contrast, leaves a (real-time) encoder no choice but to send a decoder refresh point. It doesn't let the encoder take into account any of the considerations mentioned above 🔥
4.3.2. Temporal-Spatial Trade-off Request (TSTR) 🎀
The TSTR feedback message is identified by RTCP packet type value PT=PSFB and FMT=5.
The FCI field MUST contain one or more TSTR FCI entries.
4.3.2.1. Message Format 🧁
The content of the FCI entry for the Temporal-Spatial Trade-off Request is depicted in Figure 5. The length of the feedback message MUST be set to 2+2*N, where N is the number of FCI entries included.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seq nr. | Reserved | Index |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5 - Syntax of an FCI Entry in the TSTR Message- SSRC (32 bits): The SSRC of the media sender that is requested to apply the trade-off value given in Index.
- Seq nr. (8 bits): Request sequence number. The sequence number space is unique for the pairing of the SSRC of request source and the SSRC of the request target. The sequence number SHALL be increased by 1 modulo 256 for each new command. A repetition SHALL NOT increase the sequence number. The initial value is arbitrary.
- Reserved (19 bits): All bits SHALL be set to 0 by the sender and SHALL be ignored on reception.
- Index (5 bits): An integer value between 0 and 31 that indicates the relative trade-off that is requested. An index value of 0 indicates the highest possible spatial quality, while 31 indicates the highest possible temporal resolution.
4.3.2.2. Semantics 💭
A decoder can suggest a temporal-spatial trade-off level by sending a TSTR message to an encoder. If the encoder is capable of adjusting its temporal-spatial trade-off, it SHOULD take the received TSTR message into account for future coding of pictures. A value of 0 suggests high spatial quality and a value of 31 suggests a high frame rate. The progression of values from 0 to 31 monotonically indicates a desire for a higher frame rate. The index values do not correspond to precise values of spatial quality or frame rate 🌈
The reaction to the reception of more than one TSTR message by a media sender from different media receivers is left open to the implementation. The selected trade-off SHALL be communicated to the media receivers by means of the TSTN message.
Within the common packet header for feedback messages (as defined in section 6.1 of [RFC4585]), the "SSRC of packet sender" field indicates the source of the request, and the "SSRC of media source" is not used and SHALL be set to 0. The SSRCs of the media senders to which the TSTR applies are in the corresponding FCI entries.
A TSTR message MAY contain requests to multiple media senders, using one FCI entry per target media sender 👑
4.3.2.3. Timing Rules ⏰
The timing follows the rules outlined in section 3 of [RFC4585]. This request message is not time critical and SHOULD be sent using regular RTCP timing. Only if it is known that the user interface requires quick feedback, the message MAY be sent with early or immediate feedback timing.
4.3.2.4. Handling of Message in Mixers and Translators 🦋
A mixer or media translator that encodes content sent to the session participant issuing the TSTR SHALL consider the request to determine whether it can fulfill it by changing its own encoding parameters. A media translator unable to fulfill the request MAY forward the request unaltered towards the media sender. A mixer encoding for multiple session participants will need to consider the joint needs of these participants before generating a TSTR on its own behalf towards the media sender. See also the discussion in section 3.5.2 ✨
4.3.2.5. Remarks 🤓
The term "spatial quality" does not necessarily refer to the resolution as measured by the number of pixels the reconstructed video is using. In fact, in most scenarios the video resolution stays constant during the lifetime of a session. However, all video compression standards have means to adjust the spatial quality at a given resolution, often influenced by the Quantizer Parameter or QP. A numerically low QP results in good reconstructed picture quality, whereas a numerically high QP yields a coarse picture. The typical reaction of an encoder to this request is to change its rate control parameters to use a lower frame rate and a numerically lower (on average) QP, or vice versa. The precise mapping of Index value to frame rate and QP is intentionally left open here, as it depends on factors such as the compression standard employed, spatial resolution, content, bit rate, and so on 💖
4.3.3. Temporal-Spatial Trade-off Notification (TSTN) 📣
The TSTN message is identified by RTCP packet type value PT=PSFB and FMT=6.
The FCI field SHALL contain one or more TSTN FCI entries.
4.3.3.1. Message Format 📦
The content of an FCI entry for the Temporal-Spatial Trade-off Notification is depicted in Figure 6. The length of the TSTN message MUST be set to 2+2*N, where N is the number of FCI entries.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seq nr. | Reserved | Index |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6 - Syntax of the TSTN- SSRC (32 bits): The SSRC of the source of the TSTR that resulted in this Notification.
- Seq nr. (8 bits): The sequence number value from the TSTR that is being acknowledged.
- Reserved (19 bits): All bits SHALL be set to 0 by the sender and SHALL be ignored on reception.
- Index (5 bits): The trade-off value the media sender is using henceforth.
Informative note: The returned trade-off value (Index) may differ from the requested one, for example, in cases where a media encoder cannot tune its trade-off, or when pre-recorded content is used. 🙈
4.3.3.2. Semantics ✨
okay so this one is the acknowledgement 🫶 It's the message that tells the requester "yes, I got your TSTR, we're good." For each TSTR that's aimed at you, a TSTN FCI entry SHALL go out in a TSTN feedback message. One TSTN message MAY acknowledge several requests by carrying several FCI entries. Every FCI entry in the TSTN SHALL have the same index value 💅 Putting one FCI in for each requestor is what lets each of them confirm that the media sender actually got their request, which is honestly so thoughtful.
The Notification SHALL also be sent when you get TSTR repetitions. Now here's the tea 🍵 If one requestor has sent TSTRs with several different sequence numbers, the request receiver SHALL only respond to the one with the highest sequence number (modulo 256). Heads up, the highest sequence number can look like a smaller integer because the field wraps around 🌀 For tracking the highest sequence number, [RFC3550] has an algorithm in Appendix A.1 for RTP packets, and you could adapt it for this 🧠
The TSTN SHALL include the Temporal-Spatial Trade-off index that's going to be used because of the request. Bestie, this is not always the index that was asked for 🙈 The media sender might need to combine requests from a bunch of session participants, and it could also have other policies or rules that limit what it picks ✨
Inside the common packet header for feedback messages (from section 6.1 of [RFC4585]), the "SSRC of packet sender" field tells you who the Notification came from. The "SSRC of media source" field isn't used, and it SHALL be set to 0 🌸 The SSRCs of the requesting entities that the Notification is for live in the matching FCI entries.
4.3.3.3. Timing Rules ⏰
The timing follows the rules in section 3 of [RFC4585] 🦋 This acknowledgement isn't super time critical, so it SHOULD be sent using regular RTCP timing. Chill vibes.
4.3.3.4. Handling of TSTN in Mixers and Translators 🔀
If a mixer or translator acts on a TSTR, it SHALL also send the matching TSTN 💖 If it has to forward the TSTR itself, the notification message MAY need to wait until the TSTR has been responded to. Patience is a virtue, babe 🧚
4.3.3.5. Remarks 💤
None.
4.3.4. H.271 Video Back Channel Message (VBCM) 🎀
The VBCM is identified by RTCP packet type value PT=PSFB and FMT=7 💜
The FCI field MUST contain one or more VBCM FCI entries. No empty FCI, we don't do that here 💅
4.3.4.1. Message Format 🧾
Here's what an FCI entry inside the VBCM indication looks like, shown in Figure 7 ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seq nr. |0| Payload Type| Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| VBCM Octet String.... | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7 - Syntax of an FCI Entry in the VBCM
Here's the breakdown of each field 🔍
- SSRC (32 bits): This is the SSRC of the media sender that's being asked to tell its encoder to react to the VBCM 🎯
- Seq nr. (8 bits): This is the command sequence number. The sequence number space is unique to each pairing of the command source's SSRC and the command target's SSRC. The sequence number SHALL go up by 1 (modulo 256) for each new command. A repetition SHALL NOT increase the sequence number. The starting value is arbitrary 🌈
- 0: This must be set to 0 by the sender, and it should not be acted upon by the receiver. (lowkey just leave it alone 🙈)
- Payload Type (7 bits): This is the RTP payload type that the VBCM bit stream must be interpreted under 🍓
- Length (16 bits): This is the length of the VBCM octet string in octets, not counting any padding octets 📏
- VBCM Octet String (variable length): This is the octet string the decoder makes, and it carries one specific feedback sub-message 💌
- Padding (variable length): These are bits set to 0 so everything lines up on a 32-bit boundary 🧱
4.3.4.2. Semantics 🧠
The "payload" of the VBCM indication carries different kinds of codec-specific feedback information. This can be a 'status report' (like saying a bit stream arrived with no errors, or that a partial or complete picture or block got lost), or 'update requests' (like asking for a full refresh of the bit stream) 🔄
Note: There's some overlap between the VBCM sub-messages and the CCM/AVPF feedback messages, like FIR. Check out section 3.5.3 for more on that, it's a whole thing 💀
The different types of feedback sub-messages in the VBCM are marked by the "payloadType", as defined in [H.271]. Here they are again so you don't have to go hunting 🗂️ The "payloadType" is an ITU-T Rec. H.271 term for the sub-type of the H.271 message, so please don't mix it up with an RTP payload type 🙅♀️
Payload Message Content
Type
---------------------------------------------------------------------
0 One or more pictures without detected bit stream error
mismatch
1 One or more pictures that are entirely or partially lost
2 A set of blocks of one picture that is entirely or partially
lost
3 CRC for one parameter set
4 CRC for all parameter sets of a certain type
5 A "reset" request indicating that the sender should completely
refresh the video bit stream as if no prior bit stream data
had been received
> 5 Reserved for future use by ITU-TTable 2: H.271 message types ("payloadTypes")
The bit string, or "payload", of a VBCM is variable length. It's self-contained and coded in a variable-length binary format. The media sender has to be able to parse this optimized binary format if it wants to use VBCMs 🤓
Each sub-message type (identified by payloadType) can mean different things depending on which codec is in use 🎞️
Inside the common packet header for feedback messages (from section 6.1 of [RFC4585]), the "SSRC of packet sender" field tells you where the request came from. The "SSRC of media source" field isn't used, and it SHALL be set to 0 🌸 The SSRCs of the media senders that the VBCM applies to are in the matching FCI entries. A VBCM sender MAY send H.271 messages to multiple media senders, and it MAY send more than one H.271 message to the same media sender inside the same VBCM 🫧
4.3.4.3. Timing Rules ⏰
The timing follows the rules in section 3 of [RFC4585] 💫 Different sub-message types can have different timing needs. If several types end up in the same feedback packet, the requirements of the sub-message type that's the most strict should be followed. Strictest one wins, always 👑
4.3.4.4. Handling of Message in Mixers or Translators 🔀
How a mixer or translator handles a VBCM depends on the sub-message type 🧩
4.3.4.5. Remarks 📝
Check out section 3.5.3 for a discussion of how H.271 messages and the messages defined in AVPF [RFC4585] (and this memo) with similar functionality get used 🍰
Note: People have wondered whether the RTP payload type field in this message is actually needed. It is needed if there could be more than one VBCM-capable RTP payload type in the same session, and the meaning of a given VBCM changes between payload types. For example, the picture identification mechanism in H.271 type 0 messages is fundamentally different between H.263 and H.264 (even though both use the same syntax). So the payload field is justified here. Someone also pointed out that TSTR and FIR don't need this, because their semantics are either loosely defined enough or generic enough to apply to every video payload that exists or is planned 🌟
5. Congestion Control 🌊
The correct use of the AVPF [RFC4585] timing rules stops the network from getting flooded with feedback messages. So, as long as the implementation and configuration are correct, the RTCP channel can't break its bit rate commitment and cause congestion 🙌
Some feedback messages change how media senders behave, or more specifically, how the media encoders behave 🎬
Because of that, modified behaviour MUST respect the bandwidth limits that congestion control sets. For example, when a media sender is responding to a FIR, the unusually large number of packets that make up the decoder refresh point have to be paced to match the congestion control algorithm, even if the user experience takes a hit from a slowly sent decoder refresh point 🐢
A change to the Temporary Maximum Media Stream Bit Rate value can only reduce congestion, not cause it, as long as congestion control is also in use. An increase in the value by a request REQUIRES the media sender to use congestion control when it raises its transmission rate to that value. A reduction in the value means a lower transmission bit rate, which lowers the risk of congestion 📉
6. Security Considerations 🔐
The messages defined here have certain security properties, and users of this protocol have to address and account for them. No skipping this part, it's serious 🚨
The setup signaling mechanism that's defined is open to modification attacks. Those can end up creating a session with a sub-optimal configuration, and in the worst case, the session gets rejected. To stop this kind of attack, the setup signaling requires authentication and integrity protection 🛡️
Spoofed or maliciously made feedback messages of the type defined in this specification can cause these problems 😱
- a. Severely reduced media bit rate, because false TMMBR messages set the maximum to a very low value.
- b. Assigning ownership of a bounding tuple to the wrong participant inside a TMMBN message. That can cause unnecessary oscillation in the bounding set, since the mistakenly identified owner reports a change in its tuple while the true owner possibly holds back on changes until a correct TMMBN message reaches the participants.
- c. Sending TSTRs that result in a video quality different from what the user wants, which makes the session less useful.
- d. Sending multiple FIR commands to lower the frame rate and make the video jerky, because of the frequent use of decoder refresh points.
To prevent these attacks, the feedback messages need authentication and integrity protection. This can be done against threats from outside the current RTP session by using the RTP profile that combines Secure RTP [SRTP] and AVPF into SAVPF [SAVPF] 🔒 In mixer cases, separate security contexts and filtering can be applied between the mixer and the participants, which protects the other users on the mixer from a misbehaving participant 🫶
7. SDP Definitions ✨
okay so [RFC4585] Section 4 gives us a brand new SDP [RFC4566] attribute called rtcp-fb 🎀 It's how you negotiate whether you can handle specific AVPF commands and indications, like Reference Picture Selection, Picture Loss Indication, and so on. The ABNF for rtcp-fb lives in section 4.2 of [RFC4585]. In this part we extend rtcp-fb so it can carry the codec control commands and indications from this doc, and we also talk through the Offer/Answer implications for them. main character energy 💅
7.1. Extension of the rtcp-fb Attribute
As described in AVPF [RFC4585], the rtcp-fb attribute tells everyone you can use RTCP feedback 🦋 AVPF says the rtcp-fb attribute must only be used as a media level attribute and must not be provided at session level. Every rule in [RFC4585] about the rtcp-fb attribute, including how it relates to payload types and to multiple rtcp-fb attributes in one session description, also applies to the new feedback messages we define here 🧚
The ABNF [RFC4234] for rtcp-fb from [RFC4585] looks like this 👀
"a=rtcp-fb: " rtcp-fb-pt SP rtcp-fb-val CRLFrtcp-fb-pt is the payload type, and rtcp-fb-val says what kind of feedback message it is, like ack, nack, trr-int, and rtcp-fb-id. Here's a quick example. To say you support Picture Loss Indication feedback, the sender puts this in their SDP 🌸
v=0
o=alice 3203093520 3203093520 IN IP4 host.example.com
s=Media with feedback
t=0 0
c=IN IP4 host.example.com
m=audio 49170 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 nack pliIn this document we add a new feedback value called "ccm" 🍓 It means "yep, I support codec control using RTCP feedback messages." The "ccm" feedback value SHOULD be used along with parameters that say which specific codec control commands are supported. We define four of those parameters in this doc:
"fir"means support for the Full Intra Request (FIR) 🔥"tmmbr"means support for the Temporary Maximum Media Stream Bit Rate Request/Notification (TMMBR/TMMBN). It has an optional sub-parameter for the session maximum packet rate (measured in packets per second) that's going to be used. If that isn't included, it defaults to infinity ♾️"tstr"means support for the Temporal-Spatial Trade-off Request/Notification (TSTR/TSTN) 💜"vbcm"means support for H.271 Video Back Channel Messages (VBCMs). It can have zero or more subparameters that name the supported H.271 "payloadType" values 🌈
In the ABNF for rtcp-fb-val from [RFC4585], there's a placeholder called rtcp-fb-id that's meant for new feedback types. "ccm" is a new feedback type defined in this document, so the ABNF for its parameters is defined right here. (Go check section 4.2 of [RFC4585] for the full ABNF syntax, it's the whole picture 🫶)
rtcp-fb-val =/ "ccm" rtcp-fb-ccm-param
rtcp-fb-ccm-param = SP "fir" ; Full Intra Request
/ SP "tmmbr" [SP "smaxpr=" MaxPacketRateValue]
; Temporary max media bit rate
/ SP "tstr" ; Temporal-Spatial Trade-Off
/ SP "vbcm" *(SP subMessageType) ; H.271 VBCMs
/ SP token [SP byte-string]
; for future commands/indications
subMessageType = 1*8DIGIT
byte-string = <as defined in section 4.2 of [RFC4585] >
MaxPacketRateValue = 1*15DIGIT7.2. Offer-Answer
The Offer/Answer [RFC3264] implications for codec control protocol feedback messages are pretty similar to what [RFC4585] describes 💖 The offerer MAY indicate that it can support selected codec commands and indications. The answerer MUST remove every CCM parameter for a CCM it doesn't want to support in this particular media session. That could be because it doesn't implement the message, or because its app logic decides the message isn't worth it. The answerer MUST NOT add new ccm parameters beyond what was offered 🙅♀️
The answer is binding for the media session. Both the offerer and the answerer MUST NOT use any feedback messages other than the ones both sides explicitly said they support. In other words, only the joint subset of CCM parameters from the offer and the answer may be used ✨
Heads up: including a CCM parameter in an offer or answer means that party is at least capable of receiving those CCMs and acting on them. If getting a negotiated CCM means the party has to reply with another CCM, it must also have that capability too 👀 It isn't mandated to start CCMs of any negotiated type, but it's generally expected that a party will start CCMs when it makes sense 🔥
The session maximum packet rate parameter that comes with the TMMBR indication is declarative, and the highest value from the offer and answer SHALL be used 🏆 If the session maximum packet rate parameter isn't in an offer, the answerer SHALL NOT include it 🙈
7.3. Examples
Okay the examples are where it all clicks 🧚 Example 1 is a point-to-point video call using H.263. The person who started the call says they can handle the FIR and TSTR/TSTN codec control messages. The SDP rides along on a high-level signaling protocol like SIP 📞
v=0
o=alice 3203093520 3203093520 IN IP4 host.example.com
s=Point-to-Point call
c=IN IP4 192.0.2.124
m=audio 49170 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 51372 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 ccm tstr
a=rtcp-fb:98 ccm firIn that example, when the sender gets a TSTR message from the remote party, it can adjust the trade-off the way the RTCP TSTN feedback message says to 🦋
Example 2 is a SIP endpoint joining a video mixer that's hosting a multiparty video conference 🎀 This participant only supports the FIR (Full Intra Request) codec control command, and it says so in its session description.
v=0
o=alice 3203093520 3203093520 IN IP4 host.example.com
s=Multiparty Video Call
c=IN IP4 192.0.2.124
m=audio 49170 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 51372 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 ccm firWhen the video MCU decides to route this participant's video, it sends an RTCP FIR feedback message. Once the endpoint gets that feedback message, it is required to generate a full intra request 📸
Example 3 shows the Offer/Answer implications for the codec control messages 🌸 The offerer wants to support "tstr", "fir" and "tmmbr". Here's the offered SDP:
-------------> Offer
v=0
o=alice 3203093520 3203093520 IN IP4 host.example.com
s=Offer/Answer
c=IN IP4 192.0.2.124
m=audio 49170 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 51372 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 ccm tstr
a=rtcp-fb:98 ccm fir
a=rtcp-fb:* ccm tmmbr smaxpr=120The answerer only wants to support the FIR and TSTR/TSTN messages, so its answer SDP is:
<---------------- Answer
v=0
o=alice 3203093520 3203093524 IN IP4 otherhost.example.com
s=Offer/Answer
c=IN IP4 192.0.2.37
m=audio 47190 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 53273 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 ccm tstr
a=rtcp-fb:98 ccm firExample 4 covers the Offer/Answer implications for H.271 Video Back Channel Messages (VBCMs) 🦋 The offerer wants VBCM support plus the sub-messages for payloadType 1 (one or more pictures that are entirely or partially lost) and payloadType 2 (a set of blocks of one picture that are entirely or partially lost).
-------------> Offer
v=0
o=alice 3203093520 3203093520 IN IP4 host.example.com
s=Offer/Answer
c=IN IP4 192.0.2.124
m=audio 49170 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 51372 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 ccm vbcm 1 2The answerer only wants sub-messages of type 1, so here's its answer:
<---------------- Answer
v=0
o=alice 3203093520 3203093524 IN IP4 otherhost.example.com
s=Offer/Answer
c=IN IP4 192.0.2.37
m=audio 47190 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 53273 RTP/AVPF 98
a=rtpmap:98 H263-1998/90000
a=rtcp-fb:98 ccm vbcm 1So in that last example, only VBCM indications with "payloadType" 1 get supported. Bye bye payloadType 2 🙈💅
8. IANA Considerations 🌈
The new value "ccm" has been registered with IANA in the "rtcp-fb" Attribute Values registry, which at the time of publication lived at http://www.iana.org/assignments/sdp-parameters. Here's the entry 🍓
Value name: ccm
Long Name: Codec Control Commands and Indications
Reference: RFC 5104A new registry called "Codec Control Messages" was also created to hold "ccm" parameters. At the time of publication it lived at http://www.iana.org/assignments/sdp-parameters. New registrations in this registry follow the "Specification required" policy defined by [RFC2434]. On top of that, they are required to indicate any additional RTCP feedback types, like "nack" and "ack" 🫶
The starting content of the registry is these values 💜
Value name: fir
Long name: Full Intra Request Command
Usable with: ccm
Reference: RFC 5104
Value name: tmmbr
Long name: Temporary Maximum Media Stream Bit Rate
Usable with: ccm
Reference: RFC 5104
Value name: tstr
Long name: Temporal Spatial Trade Off
Usable with: ccm
Reference: RFC 5104
Value name: vbcm
Long name: H.271 video back channel messages
Usable with: ccm
Reference: RFC 5104The following values have been registered as FMT values in the "FMT Values for RTPFB Payload Types" registry, which at the time of publication lived at http://www.iana.org/assignments/rtp-parameters 🌸
RTPFB range
Name Long Name Value Reference
-------------- --------------------------------- ----- ---------
Reserved 2 [RFC5104]
TMMBR Temporary Maximum Media Stream Bit 3 [RFC5104]
Rate Request
TMMBN Temporary Maximum Media Stream Bit 4 [RFC5104]
Rate NotificationAnd the following values have been registered as FMT values in the "FMT Values for PSFB Payload Types" registry, same publication-time location at http://www.iana.org/assignments/rtp-parameters ✨
PSFB range
Name Long Name Value Reference
-------------- --------------------------------- ----- ---------
FIR Full Intra Request Command 4 [RFC5104]
TSTR Temporal-Spatial Trade-off Request 5 [RFC5104]
TSTN Temporal-Spatial Trade-off Notification 6 [RFC5104]
VBCM Video Back Channel Message 7 [RFC5104]9. Contributors 💖
Okay bestie, Tom Taylor totally carried this spec 🫶 He helped rewrite big chunks of it, and the authors are so grateful. The parts about the algorithm for working out bounding sets for TMMBR especially glow up because of him ✨
10. Acknowledgements 🌸
The authors wanna thank Andrea Basso, Orit Levin, and Nermeen Ismail for their work on the requirements and discussion document [Basso]. Literally iconic 👑
Versions of this memo got reviewed and extensively commented on by Roni Even, Colin Perkins, Randell Jesup, Keith Lantz, Harikishan Desineni, Guido Franceschini, and others. The authors appreciate every single one of you 💜
11. References 📚
11.1. Normative References
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J.
Rey, "Extended RTP Profile for Real-Time Transport
Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC
4585, July 2006.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, July 2006.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model
with Session Description Protocol (SDP)", RFC 3264, June
2002.
[RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 2434,
October 1998.
[RFC4234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 4234, October 2005.11.2. Informative References
[Basso] Basso, A., Levin, O., and N. Ismail, "Requirements for
transport of video control commands", Work in Progress,
October 2004.
[AVC] Joint Video Team of ITU-T and ISO/IEC JTC 1, Draft ITU-T
Recommendation and Final Draft International Standard of
Joint Video Specification (ITU-T Rec. H.264 | ISO/IEC
14496-10 AVC), Joint Video Team (JVT) of ISO/IEC MPEG and
ITU-T VCEG, JVT-G050, March 2003.
[H245] ITU-T Rec. H.245, "Control protocol for multimedia
communication", May 2006.
[NEWPRED] S. Fukunaga, T. Nakai, and H. Inoue, "Error Resilient
Video Coding by Dynamic Replacing of Reference Pictures",
in Proc. Globcom'96, vol. 3, pp. 1503 - 1508, 1996.
[SRTP] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol
(SRTP)", RFC 3711, March 2004.
[RFC2032] Turletti, T. and C. Huitema, "RTP Payload Format for
H.261 Video Streams", RFC 2032, October 1996.
[SAVPF] Ott, J. and E. Carrara, "Extended Secure RTP Profile for
RTCP-based Feedback (RTP/SAVPF)", Work in Progress,
November 2007.
[RFC3525] Groves, C., Pantaleo, M., Anderson, T., and T. Taylor,
"Gateway Control Protocol Version 1", RFC 3525, June
2003.
[RFC3448] Handley, M., Floyd, S., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification",
RFC 3448, January 2003.
[H.271] ITU-T Rec. H.271, "Video Back Channel Messages", June
2006.
[RFC3890] Westerlund, M., "A Transport Independent Bandwidth
Modifier for the Session Description Protocol (SDP)", RFC
3890, September 2004.
[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram
Congestion Control Protocol (DCCP)", RFC 4340, March
2006.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002.
[RFC2198] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V.,
Handley, M., Bolot, J., Vega-Garcia, A., and S. Fosse-
Parisis, "RTP Payload for Redundant Audio Data", RFC
2198, September 1997.
[RFC4587] Even, R., "RTP Payload Format for H.261 Video Streams",
RFC 4587, August 2006.
[RFC5117] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117,
January 2008.
[XML-MC] Levin, O., Even, R., and P. Hagendorf, "XML Schema for
Media Control", Work in Progress, November 2007.Authors' Addresses 🎀
Stephan Wenger
Nokia Corporation
975, Page Mill Road,
Palo Alto,CA 94304
USA
Phone: +1-650-862-7368
EMail: stewe@stewe.org
Umesh Chandra
Nokia Research Center
975, Page Mill Road,
Palo Alto,CA 94304
USA
Phone: +1-650-796-7502
Email: Umesh.1.Chandra@nokia.com
Magnus Westerlund
Ericsson Research
Ericsson AB
SE-164 80 Stockholm, SWEDEN
Phone: +46 8 7190000
EMail: magnus.westerlund@ericsson.com
Bo Burman
Ericsson Research
Ericsson AB
SE-164 80 Stockholm, SWEDEN
Phone: +46 8 7190000
EMail: bo.burman@ericsson.comFull Copyright Statement 💅
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🦋
The IETF takes no position on whether any Intellectual Property Rights or other rights that someone might claim to cover the implementation or use of the technology in this document are valid or in scope, or on whether any license under those rights might be available. It also doesn't say it has independently looked for any such rights. Info about how rights work for RFC documents is in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat, plus any assurances of licenses that will be available, or the result of an attempt to get a general license or permission to use those proprietary rights for implementers or users of this spec, can be found in the IETF online IPR repository at http://www.ietf.org/ipr.
The IETF invites anyone to bring to its attention any copyrights, patents, patent applications, or other proprietary rights that may cover technology needed to implement this standard. Send that info to the IETF at ietf-ipr@ietf.org. Thanks, besties 🌈
📦 where this page comes from
- ietf: rfc5104 2008-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms