RFC 8853
Using Simulcast in Session Description Protocol (SDP) and RTP Sessions
RFC 8853: Using Simulcast in Session Description Protocol (SDP) and RTP Sessions 🎀
Okay bestie, this is part 1 of 8, so more is coming 💖. RFC 8853 is titled Using Simulcast in Session Description Protocol (SDP) and RTP Sessions. It's a Standards Track document from the Internet Engineering Task Force (IETF), with ISSN 2070-1721, dated January 2021. The authors are B. Burman and M. Westerlund from Ericsson, and S. Nandakumar and M. Zanaty from Cisco. The title block doesn't list anything it obsoletes or updates. 🦋
Abstract
Sometimes it's super useful to send several differently encoded versions of the same media source in different RTP streams. That's called simulcast ✨. This document explains how to do simulcast in RTP and how to signal it in the Session Description Protocol (SDP). The solution uses an RTP/RTCP identification method to figure out which RTP streams belong to the same media source, then adds an SDP extension that says those streams are different simulcast formats of that source. The extension is a new media-level SDP attribute that expresses the capability to send and/or receive simulcast RTP streams. 💅
Status of This Memo
This is an Internet Standards Track document. 🌸
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. ✨
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8853. 🧚
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 🫶
Table of Contents
1. Introduction
2. Definitions
2.1. Terminology
2.2. Requirements Language
3. Use Cases
3.1. Reaching a Diverse Set of Receivers
3.2. Application-Specific Media Source Handling
3.3. Receiver Media-Source Preferences
4. Overview
5. Detailed Description
5.1. Simulcast Attribute
5.2. Simulcast Capability
5.3. Offer/Answer Use
5.3.1. Generating the Initial SDP Offer
5.3.2. Creating the SDP Answer
5.3.3. Offerer Processing the SDP Answer
5.3.4. Modifying the Session
5.4. Use with Declarative SDP
5.5. Relating Simulcast Streams
5.6. Signaling Examples
5.6.1. Single-Source Client
5.6.2. Multisource Client
5.6.3. Simulcast and Redundancy
6. RTP Aspects
6.1. Outgoing from Endpoint with Media Source
6.2. RTP Middlebox to Receiver
6.2.1. Media-Switching Mixer
6.2.2. Selective Forwarding Middlebox
6.3. RTP Middlebox to RTP Middlebox
7. Network Aspects
7.1. Bitrate Adaptation
8. Limitation
9. IANA Considerations
10. Security Considerations
11. References
11.1. Normative References
11.2. Informative References
Appendix A. Requirements
Acknowledgements
Contributors
Authors' Addresses1. Introduction ✨
Most of today's multiparty video-conference solutions use centralized servers to cut down on bandwidth and CPU consumption in the endpoints. 🎥 Those servers receive RTP streams from each participant and send some suitable set of possibly modified RTP streams to the rest of the participants, who usually have heterogeneous capabilities (screen size, CPU, bandwidth, codec, etc.). One of the biggest issues is how to perform RTP stream adaptation to different participants' constraints with the minimum possible impact on both video quality and server performance. 😭
Simulcast is defined in this memo as the act of simultaneously sending multiple different encoded streams of the same media source -- e.g., the same video source encoded with different video-encoder types or image resolutions. 🌈 This can be done in several ways and for different purposes. This document focuses on the case where it is desirable to provide a media source as multiple encoded streams over RTP [RFC3550] towards an intermediary, so that the intermediary can provide the wanted functionality by selecting which RTP stream(s) to forward to other participants in the session. It also gets into more specifically how the identification and grouping of the involved RTP streams are done. 👑
The intended scope of the defined mechanism is to support negotiation and usage of simulcast when using SDP offer/answer and media transport over RTP. The media transport topologies considered are point-to-point RTP sessions, as well as centralized multiparty RTP sessions. In those, a media sender will provide the simulcasted streams to an RTP middlebox or endpoint, and middleboxes may further distribute the simulcast streams to other middleboxes or endpoints. Simulcast could be used point to point between middleboxes as part of a distributed multiparty scenario. Usage of multicast or broadcast transport is out of scope and left for future extensions. 🦋
This document describes a few scenarios that motivate the use of simulcast and also defines the needed RTP/RTCP and SDP signaling for it. 💖
2. Definitions 🎀
2.1. Terminology
This document makes use of the terminology defined in "A Taxonomy of Semantics and Mechanisms for Real-Time Transport Protocol (RTP) Sources" [RFC7656] and "RTP Topologies" [RFC7667]. The following terms are especially noted or here defined: 📚
RTP mixer: An RTP middlebox, in the wide sense of the term, encompassing Sections 3.6 to 3.9 of [RFC7667].
RTP session: An association among a group of participants communicating with RTP, as defined in [RFC3550] and amended by [RFC7656].
RTP stream: A stream of RTP packets containing media data, as defined in [RFC7656].
RTP switch: A common short term for the terms "switching RTP mixer", "source projecting middlebox", and "video switching Multipoint Control Unit (MCU)", as discussed in [RFC7667].
Simulcast stream: One encoded stream or dependent stream from a set of concurrently transmitted encoded streams and optional dependent streams, all sharing a common media source, as defined in [RFC7656]. For example, HD and thumbnail video simulcast versions of a single media source sent concurrently as separate RTP streams. 🎬
Simulcast format: Different formats of a simulcast stream serve the same purpose as alternative RTP payload types in nonsimulcast SDP: to allow multiple alternative media formats for a given RTP stream. As for multiple RTP payload types on the "m=" line in offer/answer [RFC3264], any one of the negotiated alternative formats can be used in a single RTP stream at a given point in time, but not more than one (based on RTP timestamp). What format is used can change dynamically from one RTP packet to another. 🔥
2.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 🙈
3. Use Cases 🌸
The use cases of simulcast described in this document relate to a multiparty communication session where one or more central nodes are used to adapt the view of the communication session towards individual participants and facilitate the media transport between participants. Thus, these cases target the RTP mixer type of topology. 🧚
There are two principal approaches for an RTP mixer to provide this adapted view of the communication session to each receiving participant:
- Transcoding (decoding and re-encoding) received RTP streams with characteristics adapted to each receiving participant. This often includes mixing or composition of media sources from multiple participants into a mixed media source originated by the RTP mixer. The main advantage of this approach is that it achieves close-to-optimal adaptation to individual receiving participants. The main disadvantages are that it can be very computationally expensive to the RTP mixer, typically degrades media Quality of Experience (QoE) such as creating end-to-end delay for the receiving participants, and requires the RTP mixer to have access to media content. 💀
- Switching a subset of all received RTP streams or substreams to each receiving participant, where the used subset is typically specific to each receiving participant. The main advantages of this approach are that it is computationally cheap to the RTP mixer, has very limited impact on media QoE, and does not require the RTP mixer to have (full) access to media content. The main disadvantage is that it can be difficult to combine a subset of received RTP streams into a perfect fit for the resource situation of a receiving participant. It is also a disadvantage that sending multiple RTP streams consumes more network resources from the sending participant to the RTP mixer. 🍓
The use of simulcast relates to the latter approach, where it is more important to reduce the load on the RTP mixer and/or minimize QoE impact than to achieve an optimal adaptation of resource usage. That's the whole vibe, bestie 💅
3.1. Reaching a Diverse Set of Receivers 💖
Okay bestie, so the media sources from one sending participant might need to reach a bunch of receivers who don't all have the same resources 🦋 The receiver resources that usually differ include, but are not limited to:
- Codec: This includes codec type (such as RTP payload format MIME type) and can include codec configuration. A couple of codec resources that differ only in codec configuration will be "different" if they are somehow not "compatible", such as if they differ in video codec profile or the transport packetization configuration.
- Sampling: This relates to how the media source is sampled, in spatial as well as temporal domain. For video streams, spatial sampling affects image resolution, and temporal sampling affects video frame rate. For audio, spatial sampling relates to the number of audio channels, and temporal sampling affects audio bandwidth. This may be used to suit different rendering capabilities or needs at the receiving endpoints.
- Bitrate: This relates to the number of bits sent per second to transmit the media source as an RTP stream, which typically also affects the QoE for the receiving user.
Letting the sending participant create a simulcast of a few differently configured RTP streams per media source can be a good trade-off when using an RTP switch as middlebox, instead of sending a single RTP stream and using an RTP mixer to create individual transcodings to each receiving participant. ✨
This only works if the receiving participants can be sorted into categories by their available resources, and the sending participant can pick a matching configuration for a single RTP stream per category and media source. 🧚 For example, say a set of receivers only differ in screen resolution: some can display video at most 360p, and some support 720p. A sending participant can then reach all receivers with the best possible resolution by creating a simulcast of RTP streams with 360p and 720p resolution for each sent video media source. It's giving smart, no cap 💅
The maximum number of simulcasted RTP streams that can be sent is mainly limited by the amount of processing and uplink network resources available to the sending participant. 💀
3.2. Application-Specific Media Source Handling 🎀
The application logic that runs the communication session can have special handling for some media sources. Like, it's super common that the media from a sending participant doesn't get sent back to itself 🙈
It's also really common for a currently active speaker to be shown bigger or in higher quality than everyone else (that's the sampling or bitrate stuff from Section 3.1) in a receiving client 👑 Lots of conferencing systems don't send the active speaker's media back to the sender, which means some other participant's media gets forwarded to the active speaker instead, usually the previous active speaker. So the previous active speaker is needed both in a larger size (for the current active speaker) and in a small size (for everyone else). That can be solved with a simulcast from the previously active speaker to the RTP switch. Iconic problem, iconic fix 🦋
3.3. Receiver Media-Source Preferences 🌸
The application logic that runs the communication session may let receiving participants state preferences about the characteristics of the RTP stream they'd like to receive, for example in terms of the aspects listed in Section 3.1. Sending a simulcast of RTP streams is one way to accommodate receivers whose preferences conflict or are otherwise incompatible 🫶
4. Overview ✨
This memo defines SDP [RFC4566] signaling that covers the simulcast use cases and functionalities described above. A number of requirements for such signaling are elaborated in Appendix A. 🌈
The Restriction Identifier (RID) mechanism, as defined in [RFC8851], lets an SDP offerer or answerer specify a number of different RTP stream restrictions for a rid-id by using the a=rid line. Examples of such restrictions are maximum bitrate, maximum spatial video resolution (width and height), maximum video frame rate, and so on. Each rid-id may also be restricted to use only a subset of the RTP payload types in the associated SDP media description. Those RTP payload types can have their own configurations and parameters affecting what can be sent or received, using the a=fmtp line as well as other SDP attributes. 💜
A new SDP media-level attribute, a=simulcast, is defined. The attribute describes, independently for "send" and "receive" directions, the number of simulcast RTP streams as well as potential alternative formats for each simulcast RTP stream. Each simulcast RTP stream, including alternatives, is identified using the RID identifier (rid-id), defined in [RFC8851].
a=simulcast:send 1;2,3 recv 4If this line is included in an SDP offer, the "send" part shows the offerer's capability and proposal to send two simulcast RTP streams. Each simulcast stream is described by one or more RTP stream identifiers (rid-ids), and each group of rid-ids for a simulcast stream is separated by a semicolon (";"). When a simulcast stream has multiple rid-ids separated by a comma (","), they describe alternative representations for that particular simulcast RTP stream. So the "send" part shown above is an intention to send two simulcast RTP streams. The first simulcast RTP stream is identified and restricted according to rid-id 1. The second simulcast RTP stream can be sent as two alternatives, identified and restricted according to rid-ids 2 and 3. The "recv" part of the line shown here says the offerer wants to receive a single RTP stream (no simulcast) according to rid-id 4. 🍓
A more complete example SDP offer media description is in Figure 1. 🎀
m=video 49300 RTP/AVP 97 98 99
a=rtpmap:97 H264/90000
a=rtpmap:98 H264/90000
a=rtpmap:99 VP8/90000
a=fmtp:97 profile-level-id=42c01f;max-fs=3600;max-mbps=108000
a=fmtp:98 profile-level-id=42c00b;max-fs=240;max-mbps=3600
a=fmtp:99 max-fs=240; max-fr=30
a=rid:1 send pt=97;max-width=1280;max-height=720
a=rid:2 send pt=98;max-width=320;max-height=180
a=rid:3 send pt=99;max-width=320;max-height=180
a=rid:4 recv pt=97
a=simulcast:send 1;2,3 recv 4
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-idFigure 1: Example Simulcast Media Description in Offer
Read at a high level, the SDP media description in Figure 1 says the offerer can send two simulcast RTP streams: one H.264 encoded stream in up to 720p resolution, and one additional stream encoded as either H.264 or VP8 with a maximum resolution of 320x180 pixels. The offerer can also receive one H.264 stream with maximum 720p resolution. 😭
The receiver of this SDP offer can generate an SDP answer that says what it accepts. It uses the a=simulcast attribute to indicate simulcast capability and to specify which simulcast RTP streams and alternatives it wants to receive and/or send. An example of such an answering a=simulcast attribute, matching the offer above, is:
a=simulcast:recv 1;2 send 4With this SDP answer, the answerer shows in the "recv" part that it wants to receive the two simulcast RTP streams. It has removed an alternative it doesn't support (rid-id 3). The "send" part confirms to the offerer that it will receive one stream for this media source according to rid-id 4. The corresponding, more complete example SDP answer media description could look like Figure 2. 🧚
m=video 49674 RTP/AVP 97 98
a=rtpmap:97 H264/90000
a=rtpmap:98 H264/90000
a=fmtp:97 profile-level-id=42c01f;max-fs=3600;max-mbps=108000
a=fmtp:98 profile-level-id=42c00b;max-fs=240;max-mbps=3600
a=rid:1 recv pt=97;max-width=1280;max-height=720
a=rid:2 recv pt=98;max-width=320;max-height=180
a=rid:4 send pt=97
a=simulcast:recv 1;2 send 4
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-idFigure 2: Example Simulcast Media Description in Answer
It is assumed that a single SDP media description is used to describe a single media source. This lines up with the concepts defined in [RFC7656] and works in a WebRTC context, both with and without BUNDLE grouping of media descriptions [RFC8843]. 🦋
To sum up: the a=simulcast line describes "send"- and "receive"-direction simulcast streams separately. Each direction can describe one or more simulcast streams, separated by semicolons. The identifiers describing simulcast streams on the a=simulcast line are rid-ids, as defined by a=rid lines in [RFC8851]. Each simulcast stream can be offered as a list of alternative rid-ids, with each alternative separated by a comma, as in the example offer in Figure 1. A detailed specification is in Section 5, and more detailed examples are outlined in Section 5.6. 👑
5. Detailed Description 💅
This section adds detail to the overview in Section 4. First comes the formal syntax (Section 5.1), followed by the rest of the SDP attribute definition in Section 5.2. "Relating Simulcast Streams" (Section 5.5) defines the RTP/RTCP mechanisms used. The section wraps up with a number of examples. ✨
5.1. Simulcast Attribute 🌈
This document defines a new SDP media-level a=simulcast attribute, with a value that follows the syntax in Figure 3. That syntax uses ABNF [RFC5234] and its update, "Case-Sensitive String Support in ABNF" [RFC7405]:
sc-value = ( sc-send [SP sc-recv] ) / ( sc-recv [SP sc-send] )
sc-send = %s"send" SP sc-str-list
sc-recv = %s"recv" SP sc-str-list
sc-str-list = sc-alt-list *( ";" sc-alt-list )
sc-alt-list = sc-id *( "," sc-id )
sc-id-paused = "~"
sc-id = [sc-id-paused] rid-id
; SP defined in [RFC5234]
; rid-id defined in [RFC8851]Figure 3: ABNF for Simulcast Value
The a=simulcast attribute takes a parameter made of one or two simulcast stream descriptions. Each one has a direction ("send" or "recv") followed by a list of one or more simulcast streams. Each simulcast stream consists of one or more alternative simulcast formats, and each simulcast format is identified by a simulcast stream identifier (rid-id). The rid-id MUST have the form of an RTP stream identifier, as described by "RTP Payload Format Restrictions" [RFC8851]. 🎀
In the list of simulcast streams, each simulcast stream is separated by a semicolon (";"). Each simulcast stream can, in turn, be offered in one or more alternative formats, represented by rid-ids separated by commas (","). Each rid-id can also be specified as initially paused [RFC7728], which is shown by putting a "~" in front of the rid-id. The reason separate initial pause states are allowed for each rid-id is that pause capability can be specified individually for each RTP payload type referenced by a rid-id. Pause capability specified via the a=rtcp-fb attribute applies only to the specified payload types, and a rid-id specified by a=rid can refer to multiple different payload types. So it is unfeasible to pause a stream with a rid-id where any of the related RTP payload type(s) do not have pause capability. 🥺
5.2. Simulcast Capability
okay bestie, simulcast capability gets announced through a brand new media-level SDP attribute, a=simulcast (see Section 5.1) 💖. using it at the session level is undefined, so implementations of this specification MUST NOT use it at the session level, and they MUST ignore it if it's received at the session level 🙈. extensions to this spec can define session-level usage if they want. each SDP media description MUST contain at most one a=simulcast line ✨.
there are separate and independent sets of simulcast streams in the "send" and "receive" directions 🦋. when you list multiple directions, each direction MUST NOT occur more than once on the same line, no doubles 💅.
simulcast streams using undefined rid-ids MUST NOT be used as valid simulcast streams by an RTP stream receiver 💀. the direction for a rid-id MUST be aligned with the direction specified for the corresponding RTP stream identifier on the a=rid line, so keep those in sync 🫶.
the listed number of simulcast streams for a direction sets a limit on how many simulcast streams are supported in that direction 🌸. the order of the listed simulcast streams in the "send" direction suggests a proposed order of preference, in decreasing order: the rid-id listed first is the most preferred, and the later ones get progressively lower preference 👑. in the "recv" direction, the order of the listed rid-ids tells you which simulcast streams are preferred, with the leftmost being most preferred. this matters if the number of actually sent simulcast streams has to get reduced for some reason 😭.
rid-ids that have explicit dependencies [RFC5583] [RFC8851] to other rid-ids (even in the same media description) MAY be used 🧚.
use of more than a single, alternative simulcast format for a simulcast stream MAY be specified as part of the attribute parameters, by writing the simulcast stream as a comma-separated list of alternative rid-ids 🎀. the order of the rid-id alternatives within a simulcast stream is significant, and they're listed from (left) most preferred to (right) least preferred. for simulcast, this overrides the normal codec preference that's expressed by format-type ordering on the "m=" line, using regular SDP rules. the whole point is to separate general codec preferences from simulcast-stream configuration preferences 🍓. however, the choice of which alternative to use per simulcast stream is independent, and there is currently no mechanism for the offerer to force the answerer to choose the same alternative for multiple simulcast streams 🤷♀️.
a simulcast stream can use a codec defined so that the same RTP synchronization source (SSRC) changes RTP payload type multiple times during a session, possibly even on a per-packet basis 🌈. a typical example is a speech codec that uses formats for Comfort Noise [RFC3389] and/or dual-tone multifrequency (DTMF) [RFC4733] 🎶.
if RTP stream pause/resume [RFC7728] is supported, any rid-id MAY be prefixed by a "~" character to say that the corresponding simulcast stream is already paused from the start of the RTP session ⏸️. in this case, support for RTP stream pause/resume MUST also be included under the same "m=" line where "a=simulcast" is included. all RTP payload types related to such an initially paused simulcast stream MUST be listed in the SDP as pause/resume capable as specified by [RFC7728] -- e.g., by using the "*" wildcard format for a=rtcp-fb 🦄.
an initially paused simulcast stream in the "send" direction, for the endpoint sending the SDP, MUST be considered equivalent to an unsolicited locally paused stream and handled accordingly 💜. initially paused simulcast streams get resumed as described by the RTP pause/resume specification. an RTP stream receiver that wants to resume an unsolicited locally paused stream needs to know the SSRC of that stream 🔍. the SSRC of an initially paused simulcast stream can be found in an RTP stream sender RTCP Sender Report (SR) or Receiver Report (RR) that includes both the desired SSRC as initial SSRC in the source description (SDES) chunk, optionally a MID SDES item [RFC8843] (if used and if rid-ids are not unique across "m=" lines), and the rid-id value in an RtpStreamId RTCP SDES item [RFC8852] 📋.
if the endpoint sending the SDP includes a "recv"-direction simulcast stream that's initially paused, then the remote RTP sender receiving the SDP SHOULD put its RTP stream in an unsolicited locally paused state 🌙. the simulcast stream sender does not put the stream in the locally paused state if there are other RTP stream receivers in the session that don't mark the simulcast stream as initially paused. however, in centralized conferencing, the RTP sender usually can't see the SDP signaling from RTP receivers, so it can't make this call 📡. the reason an initially paused "recv" stream has to be considered locally paused by the remote RTP sender, instead of being treated as an implicit pause request, is that the pausing RTP sender can't know which receiving SSRC owns the restriction when Temporary Maximum Media Stream Bit Rate Request (TMMBR) and Temporary Maximum Media Stream Bit Rate Notification (TMMBN) are used for pause/resume signaling (Section 5.6 of [RFC7728]). that's because the RTP receiver's SSRC in the "send" direction is sometimes not known yet 🙃.
using the redundant audio data format [RFC2198] could be seen as a kind of simulcast for loss-protection purposes, but it's not considered to conflict with the mechanisms in this memo, so it MAY be used like any other format 🎧. in that case, the "red" format, rather than the carried formats, SHOULD be the one listed as a simulcast stream on the a=simulcast line 💫.
the media formats and the characteristics of simulcast streams SHOULD be chosen so they're different, e.g., as different SDP formats with differing a=rtpmap and/or a=fmtp lines, or as differently defined RTP payload format restrictions 🍒. if that difference isn't required, it is RECOMMENDED to use RTP duplication procedures [RFC7104] instead of simulcast 🌷. to avoid complications in implementations, a single rid-id MUST NOT occur more than once per a=simulcast line 🚫. note that this doesn't rule out using simulcast as an RTP duplication mechanism, since it's possible to define multiple different rid-ids that are effectively equivalent 💖.
5.3. Offer/Answer Use
Note: the inclusion ofa=simulcastor the use of simulcast does not change any of the interpretation or Offer/Answer procedures for other SDP attributes, such asa=fmtpora=rid. 🎀
5.3.1. Generating the Initial SDP Offer
an offerer that wants to use simulcast for a media description SHALL include one a=simulcast attribute in that media description in the offer 🦋. an offerer that lists a set of receive simulcast streams and/or alternative formats as rid-ids in the offer MUST be prepared to receive RTP streams for any of those simulcast streams and/or alternative formats from the answerer 🫶.
5.3.2. Creating the SDP Answer
an answerer that doesn't understand the concept of simulcast won't know the attribute either, so it will remove it in the SDP answer, as defined in the existing SDP offer/answer procedures [RFC3264] 🙈. since SDP session-level simulcast is undefined in this memo, an answerer that receives an offer with the a=simulcast attribute at the SDP session level SHALL remove it in the answer 🧹. an answerer that understands the attribute but receives multiple a=simulcast attributes in the same media description SHALL disable use of simulcast by removing all a=simulcast lines for that media description in the answer 😵.
an answerer that does understand the attribute and wants to support simulcast in an indicated direction SHALL reverse the directionality of the unidirectional direction parameters -- "send" becomes "recv" and vice versa -- and include it in the answer 🔄.
an answerer that receives an offer with simulcast containing an a=simulcast attribute listing alternative rid-ids MAY keep all the alternative rid-ids in the answer, but it MAY also choose to remove any nondesirable alternative rid-ids in the answer ✂️. the answerer MUST NOT add any alternative rid-ids in the "send" direction to the answer that weren't present in the offer receive direction 🚷. the answerer MUST be prepared to receive any of the receive-direction rid-id alternatives, and it MAY send any of the "send"-direction alternatives that are part of the answer 💌.
an answerer that receives an offer with simulcast that lists a number of simulcast streams MAY reduce the number of simulcast streams in the answer, but it MUST NOT add simulcast streams 📉.
an answerer that receives an offer without RTP stream pause/resume capability MUST NOT mark any simulcast streams as initially paused in the answer 🚫⏸️.
an RTP stream answerer capable of pause/resume that receives an offer with RTP stream pause/resume capability MAY mark any rid-ids that refer to pause/resume capable formats as initially paused in the answer 🌸.
an answerer that receives indication in an offer of a rid-id being initially paused SHOULD mark that rid-id as initially paused in the answer too, regardless of direction, unless it has a good reason for that rid-id not being initially paused 💜. one reason to remove an initial pause in the answer compared to the offer could be, for example, that all "receive"-direction simulcast streams for a media source the answerer accepts in the answer would otherwise be paused 🌧️.
5.3.3. Offerer Processing the SDP Answer
an offerer that receives an answer without a=simulcast MUST NOT use simulcast towards the answerer 🙅♀️. an offerer that receives an answer with a=simulcast but without any rid-id in a specified direction MUST NOT use simulcast in that direction 💔.
an offerer that receives an answer where some rid-id alternatives are kept MUST be prepared to receive any of the kept "send"-direction rid-id alternatives, and it MAY send any of the kept "receive"-direction rid-id alternatives 🎁.
an offerer that receives an answer where some of the rid-ids have been removed compared to the offer MAY release the corresponding resources (codec, transport, etc) in its "receive" direction, and it MUST NOT send any RTP packets corresponding to the removed rid-ids 🗑️.
an offerer that offered some of its rid-ids as initially paused and receives an answer that doesn't indicate RTP stream pause/resume capability MUST NOT initially pause any simulcast streams 🙅♀️⏸️.
an offerer with RTP stream pause/resume capability that receives an answer where some rid-ids are marked as initially paused SHOULD initially pause those RTP streams, even if they were also marked as initially paused in the offer, unless it has a good reason for those RTP streams not being initially paused 🌟. one such reason could be, for example, that the answerer would otherwise not receive any media of that type at all 🥺.
5.3.4. Modifying the Session ✨
Okay so offers made inside a session that's already going follow the same vibes as the initial SDP offer, with two extra rules 💅:
- Any rid-id marked as initially paused in the offerer's "send" direction SHALL reflect the offerer's opinion of its current pause state at the moment the offer gets created 🫶. This is purely informational, and RTP stream pause/resume signaling [RFC7728] in the ongoing session SHALL take precedence if anything conflicts or gets fuzzy 🌸.
- Same deal for "receive": rid-ids marked as initially paused there SHALL (just like in an initial offer) reflect the offerer's desired rid-id pause state. The one exception is when the offerer already paused that RTP stream using RTP stream pause/resume [RFC7728] signaling. Otherwise it's identical to an initial offer 🦋.
Creating SDP answers and processing SDP answers inside an existing session follow the same rules we just went over for the initial SDP offer/answer 💖. Also, session modification restrictions in Section 6.5 of "RTP Payload Format Restrictions" [RFC8851] apply too, so don't forget those 🙈.
5.4. Use with Declarative SDP
Not gonna lie, this document doesn't define how to use a=simulcast in declarative SDP 🍓. Part of the reason is that the simulcast format identification [RFC8851] isn't defined for declarative SDP at all. If concrete use cases for simulcast in declarative SDP show up in the future, the authors of this memo expect that additional specifications will address them 🧚.
5.5. Relating Simulcast Streams
Okay, this part is super important 🔥. Simulcast RTP streams MUST be related on the RTP level through RtpStreamId [RFC8852], as specified in the SDP a=simulcast attribute (Section 5.2) parameters. That's enough as long as there's only a single media source per SDP media description 👑.
When you're using BUNDLE [RFC8843], where multiple SDP media descriptions jointly specify a single RTP session, the SDES MID (Media Identification) mechanism in BUNDLE lets you relate RTP streams back to individual media descriptions. After that, the RtpStreamId relations we just talked about can do their thing ✨. Using the RTP header extension for the RTCP source description items [RFC7941] for both MID and RtpStreamId identifications can be important for rapid initial reception, which is required to correctly interpret and process the RTP streams 🎀. Implementers of this specification MUST support the RTCP source description (SDES) item method and SHOULD support the RTP header extension method to signal RtpStreamId on the RTP level 💅.
NOTE: For the case where it's clear from SDP that the RTP PT uniquely maps to a corresponding RtpStreamId, an RTP receiver can use RTP PT to relate simulcast streams. This can sometimes enable decoding even before RtpStreamId information shows up in RTCP SDES and/or RTP header extensions 🙈.
RTP streams MUST only use a single alternative rid-id at a time (based on RTP timestamps), but they MAY change format (and rid-id) on a per-RTP packet basis 🦋. This matches the existing (nonsimulcast) SDP offer/answer case where multiple formats are included on the "m=" line in the SDP answer, which enables per-RTP packet change of RTP payload type. Iconic, honestly 😭.
5.6. Signaling Examples
Okay, these examples describe a client-to-video-conference service that uses a centralized media topology with an RTP mixer 🌈.
+---+ +-----------+ +---+
| A |<---->| |<---->| B |
+---+ | | +---+
| Mixer |
+---+ | | +---+
| F |<---->| |<---->| J |
+---+ +-----------+ +---+
Figure 4: Four-Party Mixer-Based Conference5.6.1. Single-Source Client
Alice is calling in to the mixer with a simulcast-enabled client that's capable of a single media source per media type 💖. The client can send a simulcast of 2 video resolutions and frame rates: HD 1280x720p 30fps and thumbnail 320x180p 15fps. This is defined below using the "imageattr" [RFC6236]. In this example, only the "pt" a=rid parameter is used to describe simulcast stream formats, which effectively achieves a 1:1 mapping between RtpStreamId and media formats (RTP payload types) 🍓. Alice's Offer:
v=0
o=alice 2362969037 2362969040 IN IP4 192.0.2.156
s=Simulcast-Enabled Client
c=IN IP4 192.0.2.156
t=0 0
m=audio 49200 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 49300 RTP/AVP 97 98
a=rtpmap:97 H264/90000
a=rtpmap:98 H264/90000
a=fmtp:97 profile-level-id=42c01f;max-fs=3600;max-mbps=108000
a=fmtp:98 profile-level-id=42c00b;max-fs=240;max-mbps=3600
a=imageattr:97 send [x=1280,y=720] recv [x=1280,y=720]
a=imageattr:98 send [x=320,y=180] recv [x=320,y=180]
a=rid:1 send pt=97
a=rid:2 send pt=98
a=rid:3 recv pt=97
a=simulcast:send 1;2 recv 3
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-idFigure 5: Single-Source Simulcast Offer
The only thing in the SDP that signals simulcast capability is the line in the video media description containing the simulcast attribute 🫶. The included a=fmtp and a=imageattr parameters show that sent simulcast streams can differ in video resolution 🌸. The RTP header extension for RtpStreamId is offered to avoid issues with the initial binding between RTP streams (SSRCs) and the RtpStreamId identifying the simulcast stream and its format 🧚.
The answer from the server says it, too, is simulcast capable 👑. Should it not have been simulcast capable, the a=simulcast line would not have been present, and communication would have started with the media negotiated in the SDP 🦋. Also, the usage of the RtpStreamId RTP header extension is accepted ✨.
v=0
o=server 823479283 1209384938 IN IP4 192.0.2.2
s=Answer to Simulcast-Enabled Client
c=IN IP4 192.0.2.43
t=0 0
m=audio 49672 RTP/AVP 0
a=rtpmap:0 PCMU/8000
m=video 49674 RTP/AVP 97 98
a=rtpmap:97 H264/90000
a=rtpmap:98 H264/90000
a=fmtp:97 profile-level-id=42c01f;max-fs=3600;max-mbps=108000
a=fmtp:98 profile-level-id=42c00b;max-fs=240;max-mbps=3600
a=imageattr:97 send [x=1280,y=720] recv [x=1280,y=720]
a=imageattr:98 send [x=320,y=180] recv [x=320,y=180]
a=rid:1 recv pt=97
a=rid:2 recv pt=98
a=rid:3 send pt=97
a=simulcast:recv 1;2 send 3
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-idFigure 6: Single-Source Simulcast Answer
Since the server is the simulcast media receiver, it reverses the direction of the simulcast and rid attribute parameters 🙈.
5.6.2. Multisource Client
Fred is calling in to the same conference as in the example above, with a two-camera, two-display system. That means he's capable of handling two separate media sources in each direction, and each media source is simulcast enabled in the "send" direction 📸. Fred's client is restricted to a single media source per media description, which is giving strict but cute energy 🎀.
The first two simulcast streams for the first media source use different codecs, H264-SVC [RFC6190] and H264 [RFC6184]. These two simulcast streams also have a temporal dependency 💜. Two different video codecs, VP8 [RFC7741] and H264, are offered as alternatives for the third simulcast stream for the first media source. Only the highest-fidelity simulcast stream is sent from the start, and the lower-fidelity streams are initially paused 💅.
The second media source is offered with three different simulcast streams. All video streams of this second media source are loss protected by RTP retransmission [RFC4588] 🛡️. On top of that, all but the highest-fidelity simulcast stream are initially paused. Note that the lower resolution is more prioritized than the medium-resolution simulcast stream 👑.
Fred's client also uses BUNDLE to send all RTP streams from all media descriptions in the same RTP session over a single media transport 🦋. Even with lots of different simulcast streams, using RtpStreamId as the simulcast stream identification lets the client get away with a low number of RTP payload types ✨. Also, when using both BUNDLE [RFC8843] and "a=rid" [RFC8851], it is recommended to use the RTP header extension for the RTCP source description items [RFC7941] for carrying these RTP stream-identification fields, which is why it's also included in the SDP 🌈. Note also that for "a=rid", the corresponding RtpStreamId SDES attribute RTP header extension is named rtp-stream-id [RFC8852] 🧚.
v=0
o=fred 238947129 823479223 IN IP6 2001:db8::c000:27d
s=Offer from Simulcast-Enabled Multi-Source Client
c=IN IP6 2001:db8::c000:27d
t=0 0
a=group:BUNDLE foo bar zen
m=audio 49200 RTP/AVP 99
a=mid:foo
a=rtpmap:99 G722/8000
m=video 49600 RTP/AVPF 100 101 103
a=mid:bar
a=rtpmap:100 H264-SVC/90000
a=rtpmap:101 H264/90000
a=rtpmap:103 VP8/90000
a=fmtp:100 profile-level-id=42400d;max-fs=3600;max-mbps=216000; \
mst-mode=NI-TC
a=fmtp:101 profile-level-id=42c00d;max-fs=3600;max-mbps=108000
a=fmtp:103 max-fs=900; max-fr=30
a=rid:1 send pt=100;max-width=1280;max-height=720;max-fps=60;depend=2
a=rid:2 send pt=101;max-width=1280;max-height=720;max-fps=30
a=rid:3 send pt=101;max-width=640;max-height=360
a=rid:4 send pt=103;max-width=640;max-height=360
a=depend:100 lay bar:101
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:2 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=rtcp-fb:* ccm pause nowait
a=simulcast:send 1;2;~4,3
m=video 49602 RTP/AVPF 96 104
a=mid:zen
a=rtpmap:96 VP8/90000
a=fmtp:96 max-fs=3600; max-fr=30
a=rtpmap:104 rtx/90000
a=fmtp:104 apt=96;rtx-time=200
a=rid:1 send max-fs=921600;max-fps=30
a=rid:2 send max-fs=614400;max-fps=15
a=rid:3 send max-fs=230400;max-fps=30
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:2 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=rtcp-fb:* ccm pause nowait
a=simulcast:send 1;~3;~2Figure 7: Fred's Multisource Simulcast Offer
5.6.3. Simulcast and Redundancy ✨
okay bestie, this example is where simulcast meets redundancy, and it's giving main character energy 🎀 The audio media description uses codec and bitrate restrictions, plus the RTP payload for redundant audio data [RFC2198], so packet loss doesn't get to ruin the vibe 🛡️ The video media description puts resolution and bitrate restrictions together with Forward Error Correction (FEC) in the form of flexible FEC [RFC8627] and RTP retransmission [RFC4588] 🔥
The audio source is offered as two simulcast streams. The first one is encoded with Opus, restricted to 64 kbps (rid-id=1). The second one (rid-id=2) is encoded with either G.711, or G.711 combined with linear predictive coding (LPC) for redundancy and explicit comfort noise (CN) 🎧 Both simulcast streams include telephone-event capability 📞 Notice that stand-alone LPC is not offered as a possible payload type for the second simulcast stream's RID 🙈 The example doesn't explain why, but it could be motivated by something like not providing sufficient quality, which is a very valid reason to skip it 💅
The video source is offered as two simulcast streams, and both come with two alternative simulcast formats 🎬 Redundancy and repair are offered as both flexible FEC and RTP retransmission. The flexible FEC isn't bound to any particular RTP streams, so it can be used across all the RTP streams sent as part of this media description 🦋
o=fred 238947129 823479223 IN IP6 2001:db8::c000:27d
s=Offer from Simulcast-Enabled Client using Redundancy
c=IN IP6 2001:db8::c000:27d
t=0 0
a=group:BUNDLE foo bar
m=audio 49200 RTP/AVP 97 98 99 100 101 102
a=mid:foo
a=rtpmap:97 G711/8000
a=rtpmap:98 LPC/8000
a=rtpmap:99 OPUS/48000/1
a=rtpmap:100 RED/8000/1
a=rtpmap:101 CN/8000
a=rtpmap:102 telephone-event/8000
a=fmtp:99 useinbandfec=1;usedtx=0
a=fmtp:100 97/98
a=fmtp:102 0-15
a=ptime:20
a=maxptime:40
a=rid:1 send pt=99,102;max-br=64000
a=rid:2 send pt=100,97,101,102
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:2 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=simulcast:send 1;2
m=video 49600 RTP/AVPF 103 104 105 106 107
a=mid:bar
a=rtpmap:103 H264/90000
a=rtpmap:104 VP8/90000
a=rtpmap:105 rtx/90000
a=rtpmap:106 rtx/90000
a=rtpmap:107 flexfec/90000
a=fmtp:103 profile-level-id=42c00d;max-fs=3600;max-mbps=108000
a=fmtp:104 max-fs=3600; max-fr=30
a=fmtp:105 apt=103;rtx-time=200
a=fmtp:106 apt=104;rtx-time=200
a=fmtp:107 repair-window=100000
a=rid:1 send pt=103;max-width=1280;max-height=720;max-fps=30
a=rid:2 send pt=104;max-width=1280;max-height=720;max-fps=30
a=rid:3 send pt=103;max-width=640;max-height=360;max-br=300000
a=rid:4 send pt=104;max-width=640;max-height=360;max-br=300000
a=extmap:1 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:2 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=rtcp-fb:* ccm pause nowait
a=simulcast:send 1,2;3,4Figure 8: Simulcast and Redundancy Example
6. RTP Aspects 🌸
okay so this section is all about what each entity in a simulcast media path can expect to happen at the RTP level 🧚 We follow the media from source to sink, starting at an endpoint with a media source that gets simulcasted to an RTP middlebox. That RTP middlebox can send media sources on to other RTP middleboxes (cascaded middleboxes), and it also selects one simulcast format of the media source and sends it on to the receiving endpoints. Different types of RTP middleboxes use the simulcast formats differently, so you get several different behaviors ✨
6.1. Outgoing from Endpoint with Media Source 🦋
this is the most straightforward simulcast case: the RTP streams being emitted by the endpoint that originates a media source 💖 When simulcast has been negotiated in the sending direction, the endpoint can transmit up to the number of RTP streams needed for the negotiated simulcast streams for that media source. Each RTP stream (SSRC) is identified by associating it (Section 5.5) with an RtpStreamId SDES item. That item is sent in RTCP and possibly also as an RTP header extension 🏷️ When multiple media sources have been negotiated for the same RTP session, and therefore BUNDLE [RFC8843] is used, the MID SDES item gets sent too, just like the RtpStreamId 🎀
Each RTP stream might not be continuously transmitted, and that can happen for any of these reasons: it's temporarily paused using Pause/Resume [RFC7728], the sender's application logic temporarily pauses it, or there isn't enough network capacity to send that simulcast stream. Still, all negotiated simulcast streams keep active, maintained SSRCs (at least in regular RTCP reports), even when no RTP packets are currently being sent 🫶 The relation between an RTP stream (SSRC) and a particular simulcast stream is not expected to change, except in exceptional situations such as SSRC collisions. When SSRCs do change, MID and RtpStreamId should let the receiver keep identifying the RTP streams correctly 🔍
6.2. RTP Middlebox to Receiver 💜
RTP streams in a multiparty RTP session can be used in several different ways when the session uses simulcast, at least on the media-source-to-middlebox legs 🌈 A big part of that comes from the different RTP middlebox behaviors, but the application's needs matter too. This text assumes the RTP middlebox will select a media source and choose which simulcast stream for that media source to deliver to a specific receiver. In many cases, at most one simulcast stream per media source is forwarded to a particular receiver at any instant, even though the selected simulcast stream may vary over time 🍓 If that doesn't hold because of application needs, the RTP stream aspects fall under the middlebox-to-middlebox case (Section 6.3) 💅
Which simulcast streams get forwarded toward the receiver is application specific. In conferencing applications, though, active speaker selection is very common 🎤 If the number of media sources that can be forwarded, N, is less than the total number of media sources in a multimedia session, the current and previous speakers (up to N in total) are often the ones forwarded 👑 To avoid needing media-specific processing to figure out the current speaker(s) in the RTP middlebox, the endpoint providing a media source may include metadata, such as the RTP header extension for client-to-mixer audio level indication [RFC6464] 🔊
How stream switching works depends on the media type, but for media types with significant interframe dependencies, like most video coding, switching needs to happen at suitable switching points that break or otherwise deal with the dependency structure 🎬 Switching points can be included periodically, but it's common to use mechanisms like Full Intra Requests [RFC5104] to request switching points from the endpoint performing the encoding of the media source 🙏
Including the RtpStreamId SDES item for an SSRC in the middlebox-to-receiver direction should only happen when use of RtpStreamId has been negotiated in that direction 🧾 Here's a fun fact: one can signal multiple RtpStreamIds when simulcast signaling indicates only a single simulcast stream, which lets you use all of those RtpStreamIds as alternatives for that one simulcast stream 🦄 One reason to include the RtpStreamId in the middlebox-to-receiver direction is to let the receiver know which restrictions apply to the RTP stream currently being delivered 💡 If RtpStreamId is negotiated for use, remember that the identifiers are specific to each signaling session. Even if the central entity tries to coordinate, the RtpStreamIds will likely need to be translated to leg-specific values 🔁 The cases below assume that RtpStreamId is not used in the mixer-to-receiver direction 🫧
6.2.1. Media-Switching Mixer ✨
okay so this section is about when the RTP middlebox acts like a media-switching mixer in RTP topologies (Section 3.6.2 of [RFC7667]) 🎀 the big idea is that the media sources the middlebox hands out are the mixer's own conceptual or functional ones 🦋 like, one media source could be the main speaker in high-res video, while a bunch of others are thumbnails of each participant 🌸
so the RTP stream the mixer makes ends up switching between a bunch of incoming RTP streams, for different media sources and different simulcast versions 💅 the mixer picks which media source gets sent as one of the RTP streams, and then picks the most appropriate simulcast stream for it 🔥 the picking criteria include the available bandwidth on the mixer-to-receiver path and restrictions based on how the receiver is actually using the RTP stream ✨ for example, it's totally unnecessary to forward full HD video to a receiver whose display area is just a thumbnail 🙈 so restrictions can exist that stop some simulcast streams from being forwarded for some of the mixer's media sources 💖
this means one single RTP stream gets used for each of the RTP mixer's media sources 💜 at any moment, that RTP stream is a selection of one particular RTP stream arriving at the mixer, and the RTP header-field values are rewritten so it comes out as one consistent, single RTP stream 🧚 if the RTP mixer doesn't get any incoming stream matched to this media source, the SSRC won't transmit but gets kept alive using RTCP 🌈 the SSRC, and therefore the RTP stream for the mixer's media source, is expected to be long-term stable 🫶 it only changes because of signaling or other disruptive events ⭐ also, even though we've been talking about a single RTP stream, there can sometimes be multiple RTP streams carrying the selected simulcast stream for the originating media source, including redundancy or other auxiliary RTP streams 🍓
the mixer may tell the receiver who the originating media source is by putting the Contributing Source (CSRC) field in with the originating media source's SSRC value 🦋 heads up though: because the RTP mixer can switch between simulcast versions of the media source, the CSRC value can change even when the media source stays the same 😭
this part is SO important 🚨 any MID SDES item from the originating media source has to be removed and must not be associated with the RTP stream's SSRC 💀 the signaling between the mixer and the receiver is built around the mixer's media sources only, not the originating ones 🎀 if the MID were tied to the SSRC, the receiver would likely think there was an SSRC collision and that the RTP stream is spurious, because it doesn't have the identifiers needed to link it to the right context 🙈 this doesn't apply to CSRC values though, as long as they're never used as an SSRC 💅 in those cases, CNAME and MID can be provided as SDES items, and a receiver can use them to figure out which CSRC values are associated with the same originating media source ✨
if RtpStreamIds are used in the scenario this section describes, the RtpStreamId on a particular SSRC will change based on which actual simulcast stream was selected for switching 🌸 these RtpStreamId identifiers are local to this leg's signaling context 🧚 and the defined RtpStreamIds and their parameters need to cover all the media sources and simulcast streams the RTP mixer received that can be switched into this media source and sent by the RTP mixer 👑
6.2.2. Selective Forwarding Middlebox 🦋
this section covers when the RTP middlebox acts like a Selective Forwarding Middlebox (SFM) in RTP topologies (Section 3.7 of [RFC7667]) 💖 with this kind of middlebox, every originating media source gets a corresponding media source on the leg between the middlebox and the receiver 🌈 an SFM could go all the way and expose every simulcast stream for a media source 🔥 but this section focuses on having a single simulcast stream that can contain any of the simulcast formats 💅 it assumes the SFM projection mechanism works at the media-source level and maps one of the media source's simulcast streams onto one RTP stream from the SFM to the receiver ✨
this usage means the individual RTP stream(s) for one media source can switch between active and paused, depending on which subset of media sources the SFM wants to give the receiver at that moment 🫶 with SFMs there's no reason to use CSRC to show the originating stream, since the mapping is one-to-one 🎀 if the application needs to know which simulcast version was received in order to work well, then RtpStreamId should be negotiated on the SFM-to-receiver leg 🧚 otherwise, which simulcast stream is being forwarded isn't made explicit unless RtpStreamId is used on that leg 🍓
any MID SDES items the SFM sends to the receiver are only the ones agreed between the SFM and the receiver 🙈 no MID values from the originating side of the SFM are to be forwarded 💜
an SFM could expose corresponding RTP streams for all the media sources and their simulcast streams, and then for any media source that's to be provided, forward one selected simulcast stream 🌸 however, this is not recommended 🚫 it would unnecessarily increase the number of RTP streams and make the receiver detect switching between simulcast streams in a timely way 😭 the usage above needs the same SFM switching functionality, but it skips the uncertainty of timely detecting that an RTP stream has ended 🦋 the upside of exposing all the streams is that the received simulcast stream is implicitly provided by whichever RTP stream is active for a media source 🌈 but using RtpStreamId to make that explicit also reveals which alternative format is being used 💅 so the conclusion is that using one RTP stream per simulcast stream is unnecessary 👑
here's the catch: detecting the end of a stream in a timely way is hard, whether the stream is stopped temporarily or long term, because there's no explicit indication that the transmission was intentionally stopped 💀 the RTCP-based pause and resume mechanism [RFC7728] includes a PAUSED indication that gives the last RTP sequence number transmitted before the pause ✨ but the timeliness of this solution depends on when RTCP delivery can happen relative to the transmission of the last RTP packet ⭐ and if no explicit information is provided at all, then detection based on nonincreasing RTCP SR field values and timers needs to be used to decide that RTP packet delivery has paused 🙈 so when the last RTP packet arrives (if it arrives at all), you usually can't tell that it's the last one 😭 you only find out it was the last one later 🫶
6.3. RTP Middlebox to RTP Middlebox 🌸
this is about sending simulcast streams between RTP middleboxes, or in other usages where you want to deliver multiple simultaneous simulcast streams per media source but the transmitting entity isn't the originating endpoint 🧚 for a particular direction between middleboxes A and B, this looks a lot like the originating-to-middlebox case on a media-source basis 💖 the difference is that here there are usually multiple media sources, coming from multiple endpoints 🔥 that can lead to situations where limits on the number of simultaneously received media streams come up, for example because of limited network bandwidth 🌈 in that case, a subset of not only the simulcast streams but also the media sources can be selected 💅 as a result, individual RTP streams can become paused at any point and be resumed later based on various criteria ✨
the MIDs used between A and B are the ones agreed between those two identities in signaling 🎀 the RtpStreamId values are also provided so there's explicit information about which simulcast stream each one is 🦋 the RTP-stream-to-MID and -RtpStreamId associations should be long-term stable here 👑
7. Network Aspects 🌈
simulcast, in this memo, is defined as the act of sending multiple alternative encoded streams of the same underlying media source 💖 transmitting multiple independent streams that come from the same source could be done in several different ways using RTP 🧚 there's a general discussion of considerations for using the different RTP multiplexing alternatives in "Guidelines for Using the Multiplexing Features of RTP to Support Multiple Media Streams" [RFC8872] 🦋 and discussion and clarification on handling multiple streams in an RTP session can be found in [RFC8108] 🫶
the network aspects that matter for simulcast are these 👇
- Quality of Service (QoS): when you're using simulcast, you might want to prioritize a particular simulcast stream instead of treating all streams equally 🔥 for example, lower-bitrate streams may be prioritized over higher-bitrate ones to minimize congestion or packet losses in the low-bitrate streams 💅 so there's a real benefit to using a simulcast solution with good QoS support ✨
- NAT/FW Traversal (Network Address Translator / Firewall Traversal): using multiple RTP sessions costs more for NAT/FW traversal, unless they can reuse the same transport flow 😭 that can be done by multiplexing negotiation using SDP port numbers [RFC8843] 🎀
7.1. Bitrate Adaptation 💜
using multiple simulcast streams can need a lot of network resources 🙈 the aggregate bandwidth for all simulcast streams for a media source (and so for the SDP media description) is bounded by any applicable SDP b= line for that media source 🌸 the application is assumed to use a suitable congestion-control mechanism to make sure it doesn't cause persistent congestion 🧚 if the amount of available network resources changes during an RTP session so it no longer matches what was negotiated in SDP, the bitrate used by the different simulcast streams may have to be reduced dynamically 💀 when a simulcasting media source uses a single media transport for all of its simulcast streams, it's likely that joint congestion control across all simulcast streams is used for that media source 🦋 which simulcast streams to prioritize when allocating available bitrate among them in such adaptation SHOULD be taken from the simulcast stream order on the a=simulcast line and the ordering of alternative simulcast formats (Section 5.2) 👑 simulcast streams that have pause/resume capability, and that the adaptation process would give such low bitrate that they're not really useful, can be temporarily paused until the limiting condition clears 🌈✨
8. Limitation ✨
okay bestie, there's one limitation we have to be upfront about 🥺. This approach uses a single RTP session for all the simulcast formats of a media source, and that happens because every simulcast stream related to that source lives under the same SDP media description 💅.
It's not possible to put different simulcast streams on different media transports 🙅♀️, so you can't give different simulcast streams different QoS. With unicast, QoS based on marking individual packets still works, since you don't have to split the streams into separate RTP sessions to use it ✨.
You also can't split the simulcast streams into different multicast groups so a multicast receiver can grab just the one it wants instead of getting all of them 😭. The only reasonable way to do that is to use a different RTP session for each multicast group, so the reporting and other RTCP functions still work the way they're supposed to 🎀. Simulcast in a multicast context is out of scope for this document and would need additional specification 🫶.
9. IANA Considerations 📋
this doc registers a new media-level SDP attribute, simulcast, in the att-field (media level only) registry inside the Session Description Protocol (SDP) Parameters registry, following the procedures of [RFC4566] and [RFC8859] 🦋.
Here's the registration template, exactly as written:
Contact name, email: The IESG (iesg@ietf.org)
Attribute name: simulcast
Long-form attribute name: Simulcast stream description
Charset dependent: No
Attribute value: sc-value; see Section 5.1 of RFC 8853.
Purpose: Signals simulcast capability for a set of RTP streams
Mux category: NORMAL10. Security Considerations 🚨
okay this section is SERIOUS bestie 😭🔥. The simulcast capability, the configuration attributes, and the parameters are vulnerable to attacks in signaling. Nobody wants that drama 💀.
A false inclusion of the a=simulcast attribute may result in simultaneous transmission of multiple RTP streams that would otherwise not be generated. The impact is limited by the media description joint bandwidth, which is shared by all simulcast streams no matter how many there are 🌸. Still, there may be a large number of unwanted RTP streams, and those can eat into the bandwidth that was meant for the RTP stream you actually wanted 😤.
A hostile removal of the a=simulcast attribute will result in simulcast not being used 🙈.
Integrity protection and source authentication of all SDP signaling, including the simulcast attributes, can mitigate the risks of attacks that try to alter signaling 🛡️✨.
Security considerations related to the use of a=rid and the RtpStreamId SDES item are covered in [RFC8851] and [RFC8852]. There are no additional security concerns related to their use in this specification 💖.
11. References 📚
11.1. Normative References
these are the must-read refs, bestie 🤓:
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model
with Session Description Protocol (SDP)", RFC 3264,
DOI 10.17487/RFC3264, June 2002,
<https://www.rfc-editor.org/info/rfc3264>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, DOI 10.17487/RFC4566,
July 2006, <https://www.rfc-editor.org/info/rfc4566>.
[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234,
DOI 10.17487/RFC5234, January 2008,
<https://www.rfc-editor.org/info/rfc5234>.
[RFC7405] Kyzivat, P., "Case-Sensitive String Support in ABNF",
RFC 7405, DOI 10.17487/RFC7405, December 2014,
<https://www.rfc-editor.org/info/rfc7405>.
[RFC7728] Burman, B., Akram, A., Even, R., and M. Westerlund, "RTP
Stream Pause and Resume", RFC 7728, DOI 10.17487/RFC7728,
February 2016, <https://www.rfc-editor.org/info/rfc7728>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8843] Holmberg, C., Alvestrand, H., and C. Jennings,
"Negotiating Media Multiplexing Using the Session
Description Protocol (SDP)", RFC 8843,
DOI 10.17487/RFC8843, January 2021,
<https://www.rfc-editor.org/info/rfc8843>.
[RFC8851] Roach, A.B., Ed., "RTP Payload Format Restrictions",
RFC 8851, DOI 10.17487/RFC8851, January 2021,
<https://www.rfc-editor.org/info/rfc8851>.
[RFC8852] Roach, A.B., Nandakumar, S., and P. Thatcher, "RTP Stream
Identifier Source Description (SDES)", RFC 8852,
DOI 10.17487/RFC8852, January 2021,
<https://www.rfc-editor.org/info/rfc8852>.
[RFC8859] Nandakumar, S., "A Framework for Session Description
Protocol (SDP) Attributes When Multiplexing", RFC 8859,
DOI 10.17487/RFC8859, January 2021,
<https://www.rfc-editor.org/info/rfc8859>.11.2. Informative References
these ones are good background reading, no pressure 🧚:
[RFC2198] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V.,
Handley, M., Bolot, J.C., Vega-Garcia, A., and S. Fosse-
Parisis, "RTP Payload for Redundant Audio Data", RFC 2198,
DOI 10.17487/RFC2198, September 1997,
<https://www.rfc-editor.org/info/rfc2198>.
[RFC3389] Zopf, R., "Real-time Transport Protocol (RTP) Payload for
Comfort Noise (CN)", RFC 3389, DOI 10.17487/RFC3389,
September 2002, <https://www.rfc-editor.org/info/rfc3389>.
[RFC4588] Rey, J., Leon, D., Miyazaki, A., Varsa, V., and R.
Hakenberg, "RTP Retransmission Payload Format", RFC 4588,
DOI 10.17487/RFC4588, July 2006,
<https://www.rfc-editor.org/info/rfc4588>.
[RFC4733] Schulzrinne, H. and T. Taylor, "RTP Payload for DTMF
Digits, Telephony Tones, and Telephony Signals", RFC 4733,
DOI 10.17487/RFC4733, December 2006,
<https://www.rfc-editor.org/info/rfc4733>.
[RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman,
"Codec Control Messages in the RTP Audio-Visual Profile
with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104,
February 2008, <https://www.rfc-editor.org/info/rfc5104>.
[RFC5109] Li, A., Ed., "RTP Payload Format for Generic Forward Error
Correction", RFC 5109, DOI 10.17487/RFC5109, December
2007, <https://www.rfc-editor.org/info/rfc5109>.
[RFC5583] Schierl, T. and S. Wenger, "Signaling Media Decoding
Dependency in the Session Description Protocol (SDP)",
RFC 5583, DOI 10.17487/RFC5583, July 2009,
<https://www.rfc-editor.org/info/rfc5583>.
[RFC6184] Wang, Y.-K., Even, R., Kristensen, T., and R. Jesup, "RTP
Payload Format for H.264 Video", RFC 6184,
DOI 10.17487/RFC6184, May 2011,
<https://www.rfc-editor.org/info/rfc6184>.
[RFC6190] Wenger, S., Wang, Y.-K., Schierl, T., and A.
Eleftheriadis, "RTP Payload Format for Scalable Video
Coding", RFC 6190, DOI 10.17487/RFC6190, May 2011,
<https://www.rfc-editor.org/info/rfc6190>.
[RFC6236] Johansson, I. and K. Jung, "Negotiation of Generic Image
Attributes in the Session Description Protocol (SDP)",
RFC 6236, DOI 10.17487/RFC6236, May 2011,
<https://www.rfc-editor.org/info/rfc6236>.
[RFC6464] Lennox, J., Ed., Ivov, E., and E. Marocco, "A Real-time
Transport Protocol (RTP) Header Extension for Client-to-
Mixer Audio Level Indication", RFC 6464,
DOI 10.17487/RFC6464, December 2011,
<https://www.rfc-editor.org/info/rfc6464>.
[RFC7104] Begen, A., Cai, Y., and H. Ou, "Duplication Grouping
Semantics in the Session Description Protocol", RFC 7104,
DOI 10.17487/RFC7104, January 2014,
<https://www.rfc-editor.org/info/rfc7104>.
[RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and
B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms
for Real-Time Transport Protocol (RTP) Sources", RFC 7656,
DOI 10.17487/RFC7656, November 2015,
<https://www.rfc-editor.org/info/rfc7656>.
[RFC7667] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667,
DOI 10.17487/RFC7667, November 2015,
<https://www.rfc-editor.org/info/rfc7667>.
[RFC7741] Westin, P., Lundin, H., Glover, M., Uberti, J., and F.
Galligan, "RTP Payload Format for VP8 Video", RFC 7741,
DOI 10.17487/RFC7741, March 2016,
<https://www.rfc-editor.org/info/rfc7741>.
[RFC7941] Westerlund, M., Burman, B., Even, R., and M. Zanaty, "RTP
Header Extension for the RTP Control Protocol (RTCP)
Source Description Items", RFC 7941, DOI 10.17487/RFC7941,
August 2016, <https://www.rfc-editor.org/info/rfc7941>.
[RFC8108] Lennox, J., Westerlund, M., Wu, Q., and C. Perkins,
"Sending Multiple RTP Streams in a Single RTP Session",
RFC 8108, DOI 10.17487/RFC8108, March 2017,
<https://www.rfc-editor.org/info/rfc8108>.
[RFC8627] Zanaty, M., Singh, V., Begen, A., and G. Mandyam, "RTP
Payload Format for Flexible Forward Error Correction
(FEC)", RFC 8627, DOI 10.17487/RFC8627, July 2019,
<https://www.rfc-editor.org/info/rfc8627>.
[RFC8872] Westerlund, M., Burman, B., Perkins, C., Alvestrand, H.,
and R. Even, "Guidelines for Using the Multiplexing
Features of RTP to Support Multiple Media Streams",
RFC 8872, DOI 10.17487/RFC8872, January 2021,
<https://www.rfc-editor.org/info/rfc8872>.Appendix A. Requirements 🎀
okay so this is the checklist the whole solution has to pass 💅 every requirement below exists to support the use cases from Section 3, and yes, they really are all musts 😭
- REQ-1: Identification 🦋
REQ-1.1: It must be possible to identify a set of simulcasted RTP streams as originating from the same media source in SDP signaling. 💖REQ-1.2: An RTP endpoint must be capable of identifying the simulcast stream that a received RTP stream is associated with, knowing the content of the SDP signaling. 🧚
- REQ-2: Transport usage 🚚 The solution must work when using:
REQ-2.1: Legacy SDP with separate media transports per SDP media description. 🌸REQ-2.2: Bundled [RFC8843] SDP media descriptions. 🫶
- REQ-3: Capability negotiation 🤝 The following must be possible:
REQ-3.1: The sender can express capability of sending simulcast. 📣REQ-3.2: The receiver can express capability of receiving simulcast. 👂REQ-3.3: The sender can express the maximum number of simulcast streams that can be provided. 🔢REQ-3.4: The receiver can express the maximum number of simulcast streams that can be received. 🔢REQ-3.5: The sender can detail the characteristics of the simulcast streams that can be provided. ✨REQ-3.6: The receiver can detail the characteristics of the simulcast streams that it prefers to receive. 💜
- REQ-4: Distinguishing features 🌈 It must be possible to have different simulcast streams use different codec parameters, as can be expressed by SDP format values and RTP payload types. Main character energy, literally 👑
- REQ-5: Compatibility 🧩 It must be possible to use simulcast in combination with other RTP mechanisms that generate additional RTP streams:
REQ-5.1: RTP retransmission [RFC4588]. 🔁REQ-5.2: RTP Forward Error Correction [RFC5109]. 🛡️REQ-5.3: Related payload types such as audio Comfort Noise and/or DTMF. 🎶REQ-5.4: A single simulcast stream can consist of multiple RTP streams, to support codecs where a dependent stream is dependent on a set of encoded and dependent streams, each potentially carried in their own RTP stream. This one is a lot, but it's giving layers 🧅✨
- REQ-6: Interoperability 🌍 The solution must be possible to use in:
REQ-6.1: Interworking with nonsimulcast legacy clients using a single media source per media type. 🤖REQ-6.2: WebRTC environment with a single media source per SDP media description. 🕸️
Acknowledgements 🫶
The authors would like to thank Bernard Aboba, Thomas Belling, Roni Even, Adam Roach, Iñaki Baz Castillo, Paul Kyzivat, and Arun Arunachalam for the feedback they provided during the development of this document. Thank you, queens 👑💖
Contributors 🌷
Morgan Lindqvist and Fredrik Jansson, both from Ericsson, have contributed with important material to the first draft versions of this document. Robert Hanton and Cullen Jennings from Cisco, Peter Thatcher from Google, and Adam Roach from Mozilla contributed significantly to subsequent versions. Iconic team behavior 🦋✨
Authors' Addresses 📮
Bo Burman
Ericsson
Gronlandsgatan 31
SE-164 60 Stockholm
Sweden
Email: bo.burman@ericsson.com
Magnus Westerlund
Ericsson
Torshamnsgatan 23
SE-164 83 Stockholm
Sweden
Email: magnus.westerlund@ericsson.com
Suhas Nandakumar
Cisco
170 West Tasman Drive
San Jose, CA 95134
United States of America
Email: snandaku@cisco.com
Mo Zanaty
Cisco
170 West Tasman Drive
San Jose, CA 95134
United States of America
Email: mzanaty@cisco.com📦 where this page comes from
- ietf: rfc8853 2021-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms