RFC 8872
Guidelines for Using the Multiplexing Features of RTP to Support Multiple Media Streams
📮 RFC 8872: Guidelines for Using the Multiplexing Features of RTP to Support Multiple Media Streams 💖
okay bestie, this one is a whole vibe 🦋 it's an Informational doc (so it's NOT a Standards Track spec, just helpful guidance 🎀) published by the IETF in January 2021 🌸. The ISSN is 2070-1721 ✨. It doesn't obsolete or update any other RFCs, so it's a fresh start 💅.
The authors are M. Westerlund, B. Burman (both at Ericsson 🧚), C. Perkins (University of Glasgow 🎓), H. Alvestrand (Google 💜), and R. Even 🫶. Ericsson and Google are giving main character energy here 👑.
This is part 1 of 10, so stay tuned, more is coming 🌈.
Abstract
okay so the Real-time Transport Protocol (RTP) is a super flexible protocol 🦋. It can be used in a ton of different apps, networks, and system setups ✨. That flexibility is what makes it so widely useful, but honestly it can make designing an app kinda confusing 💀. One question that has gotten a LOT of attention is how to support multiple media streams in RTP 🎀. This memo talks through the options, the design trade-offs, and gives guidelines for using RTP's multiplexing features to do it 💅.
Status of This Memo
this doc is not an Internet Standards Track spec, it's just published for info purposes 🌸. It's still an IETF product that represents the consensus of the IETF community 🫶, got public review, and was approved by the Internet Engineering Steering Group (IESG) ✨. Not every IESG-approved doc is a candidate for an Internet Standard, so see Section 2 of RFC 7841 for that 💅.
For the current status, any errata, or how to send feedback, check https://www.rfc-editor.org/info/rfc8872 🔥.
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved 💜.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as of the publication date 📜. Read these carefully, because they spell out your rights and restrictions for this doc 🙈. Code Components pulled out of this doc have to include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License 🍓.
Table of Contents
1. Introduction
2. Definitions
2.1. Terminology
2.2. Focus of This Document
3. RTP Multiplexing Overview
3.1. Reasons for Multiplexing and Grouping RTP Streams
3.2. RTP Multiplexing Points
3.2.1. RTP Session
3.2.2. Synchronization Source (SSRC)
3.2.3. Contributing Source (CSRC)
3.2.4. RTP Payload Type
3.3. Issues Related to RTP Topologies
3.4. Issues Related to RTP and RTCP
3.4.1. The RTP Specification
3.4.2. Multiple SSRCs in a Session
3.4.3. Binding Related Sources
3.4.4. Forward Error Correction
4. Considerations for RTP Multiplexing
4.1. Interworking Considerations
4.1.1. Application Interworking
4.1.2. RTP Translator Interworking
4.1.3. Gateway Interworking
4.1.4. Legacy Considerations for Multiple SSRCs
4.2. Network Considerations
4.2.1. Quality of Service
4.2.2. NAT and Firewall Traversal
4.2.3. Multicast
4.3. Security and Key-Management Considerations
4.3.1. Security Context Scope
4.3.2. Key Management for Multi-party Sessions
4.3.3. Complexity Implications
5. RTP Multiplexing Design Choices
5.1. Multiple Media Types in One Session
5.2. Multiple SSRCs of the Same Media Type
5.3. Multiple Sessions for One Media Type
5.4. Single SSRC per Endpoint
5.5. Summary
6. Guidelines
7. IANA Considerations
8. Security Considerations
9. References
9.1. Normative References
9.2. Informative References
Appendix A. Dismissing Payload Type Multiplexing
Appendix B. Signaling Considerations
B.1. Session-Oriented Properties
B.2. SDP Prevents Multiple Media Types
B.3. Signaling RTP Stream Usage
Acknowledgments
Contributors
Authors' Addresses1. Introduction ✨
okay so RTP [RFC3550] is the go-to protocol for real-time media transport 🎀. It gives you tons of flexibility and can support a big set of different apps 🦋. RTP was built from day one for multiple participants in a communication session 💜. It supports lots of topology paradigms and usages, as defined in [RFC7667] 🌸.
RTP has several multiplexing points, and each one is designed for a different purpose 🧚. These points let you support multiple RTP streams and switch between encoding or packetization techniques for the media 💅. By using multiple RTP sessions, you can structure sets of RTP streams for efficient processing or identification 🫶. So to meet an app's needs, the RTP app designer has to figure out how to best use the RTP session, the RTP stream identifier (synchronization source (SSRC)), and the RTP payload type 🔥.
There's been increased interest in more advanced RTP usage 👑. For example, multiple RTP streams are useful when one endpoint has several media sources (like multiple cameras or microphones) that need to send media at the same time 📸. That raises questions about the most appropriate RTP usage ✨. Limitations in some implementations, RTP/RTCP extensions, and signaling have also come to light 🙈. This document aims to clarify what some RTP functionalities are actually good for, which hopefully leads to future implementations that are more complete 🌈.
The point of this doc is to give clear info about what RTP can do when it comes to multiplexing 💖. The RTP app designer needs to understand the implications of a particular use of the RTP multiplexing points 🦋. This doc gives some guidelines and recommends against some usages as unsuitable, either in general or for particular purposes 💅.
The doc starts with some definitions, then covers existing RTP functionality around multiplexing 🎀. Both the desired behavior and the implications of a particular behavior depend on which topologies are used, so this topic needs some thought 🧠. Then we discuss some choices about multiplexing behavior and their impacts 🌸. Some RTP usage designs are also discussed 🧚. Finally, there are some guidelines and examples 💜.
2. Definitions 📚
2.1. Terminology 💅
The definitions in Section 3 of [RFC3550] are referenced normatively 🫶.
The taxonomy defined in [RFC7656] is referenced normatively ✨.
These terms and abbreviations are used in this doc 🎀:
- Multi-party: communication that includes multiple endpoints. In this doc, "multi-party" refers to scenarios where more than two endpoints communicate 👑.
- Multiplexing: an operation that takes multiple entities as input and aggregates them onto some common resource, while keeping each individual entity addressable so they can later be fully and unambiguously separated (demultiplexed) again 🦋.
- RTP Receiver: an endpoint or middlebox that receives RTP streams and RTCP messages 🌸. It uses at least one SSRC to send RTCP messages. An RTP receiver may also be an RTP sender 💜.
- RTP Sender: an endpoint that sends one or more RTP streams and also sends RTCP messages 🧚.
- RTP Session Group: one or more RTP sessions used together to perform some function 🫶. Examples include multiple RTP sessions used to carry different layers of a layered encoding 🔥. In an RTP Session Group, CNAMEs are assumed to be valid across all RTP sessions and designate synchronization contexts that can cross RTP sessions. That means SSRCs that map to a common CNAME can be assumed to have RTCP Sender Report (SR) timing information derived from a common clock, so they can be synchronized for playout 🌈.
- Signaling: the process of configuring endpoints to participate in one or more RTP sessions 💖.
Note: The above definitions of "RTP receiver" and "RTP sender" are consistent with the usage in [RFC3550] 📝.
2.2. Focus of This Document 🎯
This doc focuses on issues that affect RTP 💅. Issues that involve signaling protocols, like whether SIP [RFC3261], Jingle [JINGLE], or some other protocol is used for session configuration, the particular syntaxes used to define RTP session properties, or the constraints that particular choices in the signaling protocols impose, are only mentioned as examples to describe the RTP issues more precisely 🦋.
This doc assumes that apps will use RTCP 🎀. There are apps that don't send RTCP, but they don't conform to the RTP specification, so they can be regarded as reusing the RTP packet format without implementing RTP 🙈.
3. RTP Multiplexing Overview 🌸
3.1. Reasons for Multiplexing and Grouping RTP Streams 🧚
There are several reasons an endpoint might choose to send multiple media streams 💜. As you read below, keep in mind that the reasons for having multiple RTP streams vary and include, but are not limited to, the following 🫶:
- There might be multiple media sources ✨.
- Multiple RTP streams might be needed to represent one media source, for example:
- To carry different layers of a scalable encoding of a media source 🌈.
- Alternative encodings during simulcast, using different codecs for the same audio stream 🎀.
- Alternative formats during simulcast, multiple resolutions of the same video stream 📸.
- A retransmission stream might repeat some parts of the content of another RTP stream 🔁.
- A Forward Error Correction (FEC) stream might provide material that can be used to repair another RTP stream 🛠️.
For each of these reasons, you have to decide whether each additional RTP stream is sent within the same RTP session as the other RTP streams, or whether you need additional RTP sessions to group them 💅. For a combo of reasons, the right choice for one situation might not be the right choice for another 🙈. Choosing is easiest when multiplexing multiple media sources of the same media type 🦋. However, every reason deserves discussion and clarification about how to handle it 🌸. As the discussion below shows, a single solution does not suit all purposes, so to use RTP well and as efficiently as possible, both are needed 👑. The real issue is knowing when to create multiple RTP sessions versus when to send multiple RTP streams in a single RTP session 💖.
Part 1 of 10 done 🫶. Say the word and I'll send Part 2 with Section 3.2 onwards 🦋.
3.2. RTP Multiplexing Points
Okay bestie, this is where we get into all the spots RTP can use to tell its streams apart 🦋 Figure 1 walks through how an incoming RTP stream gets sorted out, starting from one or more sockets that receive transport flows, like the ones picked by the UDP destination port 🎀 The same demux step also separates RTP/RTCP from other protocols sharing the same transport, like Session Traversal Utilities for NAT (STUN) [RFC5389] and DTLS-SRTP [RFC5764], as described in [RFC7983] 💅 The Processing and Buffering (PB) step in the figure is where RTP/RTCP gets terminated and the RTP payload gets prepped for the decoder ✨
| | |
| | | packets
+-- v v v
| +------------+
| | Socket(s) | Transport Protocol Demultiplexing
| +------------+
| || ||
RTP | RTP/ || |+-----> DTLS (SRTP keying, SCTP, etc.)
Session | RTCP || +------> STUN (multiplexed using same port)
+-- ||
+-- ||
| ++(split by SSRC)-++---> Identify SSRC collision
| || || || ||
| (associate with signaling by MID/RID)
| vv vv vv vv
RTP | +--+ +--+ +--+ +--+ Jitter buffer,
Streams | |PB| |PB| |PB| |PB| process RTCP, etc.
| +--+ +--+ +--+ +--+
+-- | | | |
(select decoder based on payload type (PT))
+-- | / | /
| +-----+ | /
| / | |/
Payload | v v v
Formats | +---+ +---+ +---+
| |Dec| |Dec| |Dec| Decoders
| +---+ +---+ +---+
+--
Figure 1: RTP Demultiplexing Process3.2.1. RTP Session
An RTP session is the highest semantic layer in RTP, and it's basically the association between a group of communicating endpoints 🌸 RTP doesn't have a session identifier of its own, but different RTP sessions still have to be identifiable, both across a set of different endpoints and from the view of one single endpoint 🧚
For RTP session separation across endpoints, the participants that form an RTP session are the ones that share a single SSRC space [RFC3550]. So if a group of participants each know about the SSRC identifiers belonging to the others, those participants are in one single RTP session 👑 A participant can learn about an SSRC identifier by receiving an RTP packet that has it in the SSRC field or contributing source (CSRC) list, by receiving an RTCP packet that lists it in an SSRC field, or through signaling, like the Session Description Protocol (SDP) [RFC4566] "a=ssrc:" attribute [RFC5576] 🫶 So the scope of an RTP session is decided by the participants' network interconnection topology, combined with the RTP and RTCP forwarding strategies the endpoints and any middleboxes use, plus the signaling 💜
For RTP session separation within a single endpoint, RTP leans on the underlying transport layer and the signaling to identify RTP sessions in a way that makes sense to the application 🎀 One endpoint can have one or more transport flows for the same RTP session, and a single RTP session can span multiple transport-layer flows even if every endpoint uses just one transport-layer flow per endpoint for that session 🌈 The signaling layer might hand RTP sessions an explicit identifier, or the identification might be implicit, based on the addresses and ports used 😭 So a single RTP session can have multiple identifiers, explicit and implicit, belonging to different contexts ✨ For example, when running RTP on top of UDP/IP, an endpoint can identify and delimit an RTP session from other RTP sessions by their UDP source and destination IP addresses and their UDP port numbers 🦋 A single RTP session can use multiple IP/UDP flows for receiving and/or sending RTP packets to other endpoints or middleboxes, even if the endpoint only has one IP address. Using multiple IP addresses just makes it more likely that multiple IP/UDP flows will be required 💅 Another example is SDP media descriptions (the "m=" line and the subsequent associated lines), which signal the transport flow and RTP session configuration for the endpoint's part of the RTP session 🍓 The SDP grouping framework [RFC5888] lets you label media descriptions so that RTP Session Groups can be created 🔥 And through "Negotiating Media Multiplexing Using the Session Description Protocol (SDP)" [RFC8843], multiple media descriptions can become part of a common RTP session, where each media description represents the RTP streams sent or received for a media source 🫶
RTP makes no normative statements about the relationship between different RTP sessions 🙈 Still, applications that use more than one RTP session need to understand how the different RTP sessions they create relate to one another 💀
3.2.2. Synchronization Source (SSRC)
An SSRC identifies a source of an RTP stream, or an RTP receiver when sending RTCP 🌸 Every endpoint has at least one SSRC identifier, even if it never sends RTP packets 🧚 RTP endpoints that are only RTP receivers still send RTCP, and they use their SSRC identifiers in the RTCP packets they send 💖 An endpoint can have multiple SSRC identifiers if it sends multiple RTP streams 💜 Endpoints that act as both RTP sender and RTP receiver use the same SSRC(s) in both roles ✨
The SSRC is a 32-bit identifier 🎀 It's present in every RTP and RTCP packet header, and in the payload of some RTCP packet types 🌈 It can also show up in SDP signaling 🦋 Unless it's presignaled, e.g., using the SDP "a=ssrc:" attribute [RFC5576], the SSRC is chosen at random 🫶 It doesn't depend on the endpoint's network address, and it's intended to be unique within an RTP session 💅 SSRC collisions can happen, and they get handled as specified in [RFC3550] and [RFC5576], which means the SSRC of the colliding RTP streams or receivers can change 😭 An endpoint that changes its network transport address during a session has to pick a new SSRC identifier, so it doesn't get interpreted as a looped source 🔥 The exception is when a mechanism providing a virtual transport, like Interactive Connectivity Establishment (ICE) [RFC8445], hides those changes 👑
SSRC identifiers that belong to the same synchronization context (meaning they represent RTP streams that can be synchronized using information in RTCP SR packets) use identical CNAME chunks in their corresponding RTCP source description (SDES) packets 🧚 SDP signaling can also be used to provide explicit SSRC grouping [RFC5576] 🍓
Sometimes the same SSRC identifier value gets used to relate streams in two different RTP sessions, like in RTP retransmission [RFC4588] 🙈 This is to be avoided, since there's no guarantee that SSRC values are unique across RTP sessions 💀 For RTP retransmission [RFC4588], it's recommended to use explicit binding between the source RTP stream and the redundancy stream, e.g., using the RepairedRtpStreamId RTCP SDES item [RFC8852] 🌸 The RepairedRtpStreamId is a rather recent mechanism, so you can't expect older applications to follow this recommendation 🥹
Note that the RTP sequence number and RTP timestamp are scoped by the SSRC, so they're specific to each RTP stream ✨
Different types of entities use an SSRC to identify themselves, like this:
- A real media source uses the SSRC to identify a "physical" media source 🎥
- A conceptual media source uses the SSRC to identify the result of applying some filtering function in a network node. For example, that could be a filtering function in an RTP mixer that provides the most active speaker based on some criteria, or a mix representing a set of other sources 🎶
- An RTP receiver uses the SSRC to identify itself as the source of its RTCP reports 📊
An endpoint that generates more than one media type, like a conference participant sending both audio and video, need not (and, honestly, should not) use the same SSRC value across RTP sessions 🙅♀️ Using RTCP compound packets that contain the CNAME SDES item is the designated method for binding an SSRC to a CNAME, which effectively cross-correlates SSRCs within and between RTP sessions as coming from the same endpoint 🔗 The main property of SSRCs associated with the same CNAME is that they come from a particular synchronization context and can be synchronized at playback ⭐
An RTP receiver that gets a previously unseen SSRC value will interpret it as a new source 🤔 It might actually be a previously existing source that had to change its SSRC number because of an SSRC conflict 🔄 Using the media identification (MID) extension [RFC8843] helps identify which media source the new SSRC represents, and using the restriction identifier (RID) extension [RFC8851] helps identify what encoding or redundancy stream it represents, even though the SSRC changed 🧩 Still, the originator of the previous SSRC ought to have ended the conflicting source by sending an RTCP BYE for it before starting to send with the new SSRC, which makes the new SSRC a new source 👋
3.2.3. Contributing Source (CSRC)
The CSRC isn't a separate identifier 🦋 Instead, an SSRC identifier gets listed as a CSRC in the RTP header of a packet generated by an RTP mixer or video Multipoint Control Unit (MCU) / switch, if that SSRC was in the header of one of the packets that contributed to the output 💖
Generally, you can't extract the media represented by an individual CSRC, since it's typically the result of a media merge (e.g., mix) operation on the individual media streams that correspond to the CSRC identifiers 🥣 The exception is when only a single CSRC is indicated, because that represents the forwarding of an RTP stream that might have been modified 🎀 The RTP header extension ("A Real-time Transport Protocol (RTP) Header Extension for Mixer-to-Client Audio Level Indication" [RFC6465]) adds to the receiver's information about a packet with a CSRC list 🌈 Because of these restrictions, a CSRC won't be considered a fully qualified multiplexing point, so it gets disregarded for the rest of this document 🙃
3.2.4. RTP Payload Type 🎀
Okay so every RTP stream uses one or more RTP payload formats 💖. A payload format describes how the output of a particular media codec gets framed and encoded into RTP packets 🌸. You identify it with the payload type (PT) field in the RTP packet header ✨. That means the combo of SSRC and PT is what pins down one specific RTP stream in one specific encoding format 💅. The format definition can come from [RFC3551] for statically allocated payload types, but it ought to be explicitly defined in signaling, such as SDP, for both static and dynamic payload types 🧚.
The word "format" here covers everything described by out-of-band signaling 🫶. In SDP, "format" includes:
- media type
- RTP timestamp sampling rate
- codec
- codec configuration
- payload format configurations
- various robustness mechanisms, such as redundant encodings [RFC2198]
The RTP payload type is scoped by the sending endpoint within an RTP session 🦋. PT means the same thing across all RTP streams in an RTP session, and all SSRCs sent from a single endpoint share the same payload type definitions 👑. The design is that only one payload type is valid at any instant on the RTP stream's timestamp timeline. So if things change, you're basically time-multiplexing the different payload types 🍓. The payload type can change on a per-packet basis for an SSRC. A speech codec using generic comfort noise [RFC3389] is a good example 🙈. If there's a true need to send multiple payload types for the same SSRC that are valid at the same instant, redundant encodings [RFC2198] can be used 🔥. There are several more constraints on top of the ones above that you need to meet to make this work, and one of them is that the combined payload sizes of the different payload types ought not exceed the transport MTU 💀.
Other aspects of using the RTP payload format are described in "How to Write an RTP Payload Format" [RFC8088] 🌈.
The payload type is not a multiplexing point at the RTP layer 🙅 (see Appendix A for a detailed discussion of why using the payload type as an RTP multiplexing point does not work). It still gets used to figure out how to consume and decode an RTP stream ✨. Sometimes the RTP payload type number is also used to tie an RTP stream to the signaling, and that usually requires unique RTP payload type numbers in each context 😭. Using MID, for example when bundling "m=" sections [RFC8843], can replace the payload type as a signaling association 💜. Once that's in place, unique RTP payload types are no longer required for that purpose 🦋.
3.3. Issues Related to RTP Topologies 🌷
How RTP multiplexing is done will usually vary depending on how the RTP session participants are interconnected, as described in "RTP Topologies" [RFC7667] 🗺️.
Even the most basic use case, "Topo-Point-to-Point" as described in [RFC7667], brings up a bunch of considerations 🤯. We'll go through them in detail in the following sections. They cover aspects like:
- Does my communication peer support RTP as defined with multiple SSRCs per RTP session?
- Do I need network differentiation in the form of QoS (Section 4.2.1)?
- Can the application more easily process and handle the media streams if they are in different RTP sessions?
- Do I need to use additional RTP streams for RTP retransmission or FEC?
For some point-to-multipoint topologies (for example, Topo-ASM and Topo-SSM [RFC7667]), multicast is used to connect the session participants 📡. Special considerations (documented in Section 4.2.3) are then needed, because multicast is a one-to-many distribution system 📣.
Sometimes a communication session ends up with peers that just aren't compatible 😩. There are a few reasons this happens:
- There's no common media codec for a media type, so transcoding is required.
- Support for multiple RTP streams and RTP sessions is different between the peers.
- The media transport protocols differ, so one peer uses RTP but the other uses a different transport protocol.
- The transport protocols differ, for example UDP, the Datagram Congestion Control Protocol (DCCP), or TCP.
- The security solutions differ, such as IPsec, TLS, DTLS, or the Secure Real-time Transport Protocol (SRTP), with different keying mechanisms.
These compatibility issues can often be fixed by putting a translator between the two peers, the Topo-PtP-Translator described in [RFC7667] 🪄. The translator's main job is to make the peers look compatible to each other 🤝. There can also be non-compatibility reasons for adding a translator, in the form of a middlebox or gateway. An example is needing to monitor the RTP streams 👀. Heads up though: changing the stream transport characteristics in the translator can take a deep understanding of everything from congestion control and media-level adaptations to application-layer semantics ⚠️.
Within the uses enabled by the RTP standard, the point-to-point topology can contain one or more RTP sessions. Each session can have one or more media sources, and each media source can have one or more RTP streams 💕.
3.4. Issues Related to RTP and RTCP 💜
Using multiple RTP streams is a well-supported feature of RTP ✨. But for most implementers, or people writing RTP/RTCP applications or extensions trying to apply multiple streams, it can be really unclear when it's best to add another RTP stream to an existing RTP session, and when it's better to use multiple RTP sessions 😵💫. This section goes through the considerations you need to take into account 📝.
3.4.1. The RTP Specification 📜
RFC 3550 has some recommendations and a numbered list (Section 5.2 of [RFC3550]) of five arguments about different aspects of RTP multiplexing 🧐. Go check out Section 5.2 of [RFC3550] yourself. Here are the five important aspects, quoted:
1.
1. | If, say, two audio streams shared the same RTP session and the
| same SSRC value, and one were to change encodings and thus
| acquire a different RTP payload type, there would be no
| general way of identifying which stream had changed encodings.This argument pushes for a different SSRC for each individual RTP stream 🦋. It's fundamental to how RTP works 💯.
2.
2. | An SSRC is defined to identify a single timing and sequence
| number space. Interleaving multiple payload types would
| require different timing spaces if the media clock rates
| differ and would require different sequence number spaces to
| tell which payload type suffered packet loss.This argument is against demultiplexing RTP streams within a session based only on their RTP payload type numbers 🚫. It still stands, as you can see from the big list of issues in Appendix A 📋.
3.
3. | The RTCP sender and receiver reports (see Section 6.4) can
| only describe one timing and sequence number space per SSRC
| and do not carry a payload type field.This is yet another argument against payload type multiplexing 🙅♀️.
4.
4. | An RTP mixer would not be able to combine interleaved streams
| of incompatible media into one stream.This argument is against multiplexing RTP packets that need different handling into the same session 🙃. Most of the time, the RTP mixer has to embed application logic to handle streams. Splitting streams by stream type is just another piece of application logic, and it might or might not fit a particular application 🤷♀️. One type of application that can mix different media sources blindly is the audio-only telephone bridge. That works because of the well-defined scenario that comes with using a single media type, even though the individual streams may use incompatible codec types 📞. Most other types of applications need application-specific logic to mix correctly 🧠.
5.
5. | Carrying multiple media in one RTP session precludes: the use
| of different network paths or network resource allocations if
| appropriate; reception of a subset of the media if desired,
| for example just audio if video would exceed the available
| bandwidth; and receiver implementations that use separate
| processes for the different media, whereas using separate RTP
| sessions permits either single- or multiple-process
| implementations.This argument talks about network aspects covered in Section 4.2 🌐. It also gets into implementation aspects, like split component terminals (see Section 3.10 of [RFC7667]). Those are endpoints where different processes or interconnected devices handle different parts of the whole multimedia session 🧩.
To sum it up, RFC 3550's take on multiplexing is: use unique SSRCs for anything that's its own media/packet stream, and use different RTP sessions for media streams that don't share a media type 🎯. This document backs up the first point, and it's super valid 🫶. The second part needs more discussion, because forcing one solution on every RTP usage would be inappropriate 🙅. "Sending Multiple Types of Media in a Single RTP Session" [RFC8860] updates RFC 3550 to allow multiple media types in an RTP session. It also gives a detailed look at the potential benefits and issues of having multiple media types in the same RTP session 🔍. So [RFC8860] opens up a wider scope for an RTP session, and treats multiple media types in one RTP session as a possible choice for the RTP application designer 🌟.
3.4.2. Multiple SSRCs in a Session 🎀
Using multiple SSRCs at one endpoint in an RTP session means some unclear parts of the RTP specification have to be sorted out first 🧩. These items could lead to interoperability issues, and also to some potentially significant inefficiencies. This is covered further in "Sending Multiple RTP Streams in a Single RTP Session" [RFC8108] 📚. An RTP application designer should think about these issues, and about how the application could be affected if the peer endpoints lack appropriate RTP handling or optimization 🫠.
Using multiple RTP sessions can potentially soften application issues caused by multiple SSRCs in an RTP session 🦋.
3.4.3. Binding Related Sources ✨
okay so bestie, binding related RTP streams together has been a recurring drama across a bunch of RTP extensions 😭 It's the same problem whether you're using extra SSRCs or multiple RTP sessions, so everyone's been trying to solve it 💅
There are a few groups of solutions:
- RTP/RTCP based 💖
- Signaling based, e.g., SDP 🎀
- Grouping related RTP sessions 🌸
- Grouping SSRCs within an RTP session 🦋
Most of them are explicit, but some implicit methods have also been tried. Iconic behavior from the implicit side 🙈
The SDP-based signaling solutions are:
- SDP media description grouping: The SDP grouping framework [RFC5888] uses various semantics to group any number of media descriptions. It's mostly been used to group RTP sessions, but combined with [RFC8843] it can also group multiple media descriptions within a single RTP session. Main character energy 👑
- SDP media multiplexing: "Negotiating Media Multiplexing Using the Session Description Protocol (SDP)" [RFC8843] uses info from both SDP and RTCP to associate RTP streams with SDP media descriptions. So both SDP and RTCP can group the RTP streams belonging to an SDP media description, and group multiple SDP media descriptions into a single RTP session. Literally so efficient 🧚
- SDP SSRC grouping: "Source-Specific Media Attributes in the Session Description Protocol (SDP)" [RFC5576] has a solution for grouping SSRCs the same way the grouping framework groups media descriptions 🍓
All of those grouping constructs cover a lot of use cases, ngl. But they start to struggle when the session is super dynamic and it's hard, or just a huge resource drain, to keep the list of related SSRCs up to date 😩
Now the RTP/RTCP-based option. One way is to use the RTCP SDES CNAME to bind related RTP streams to an endpoint or a synchronization context. If an application has a single RTP stream per type (media, source, or redundancy stream), the CNAME is enough for that, whether you're using one RTP session or several 🫶 Some apps skip the CNAME, though, because it feels complicated or they don't want to implement RTCP. Instead they reuse the same SSRC value to bind related streams across multiple RTP sessions. RTP retransmission [RFC4588], when set up to use multiple RTP sessions, and generic FEC [RFC5109] both use the CNAME method to relate their RTP streams. That can work, but it might have some downsides in RTP sessions with lots of participating SSRCs 🤔
It is not recommended to use identical SSRC values across RTP sessions to relate RTP streams. When an SSRC collision happens, that SSRC has to change in all RTP sessions, which resynchronizes every stream instead of only the one media stream that actually had the collision. Big yikes 💀
Another way to implicitly bind SSRCs is used by RTP retransmission [RFC4588] when the retransmissions go over the same RTP session as the source RTP stream. A receiver that's missing a packet sends an RTP retransmission request, then waits for a new SSRC carrying the RTP retransmission payload, and that SSRC has to come from the same CNAME. This means a requester can only have one outstanding retransmission request on any new SSRCs per endpoint. Very controlled, very intentional 🎀
"RTP Payload Format Restrictions" [RFC8851] gives an RTP/RTCP-based mechanism to clearly identify the RTP streams within an RTP session. It also restricts the streams' payload format parameters in a codec-agnostic way, beyond what regular payload types can do 🌈 The mapping works by putting an a=rid value in the SDP offer/answer signaling, and having the matching RtpStreamId value show up as an SDES item and an RTP header extension [RFC8852]. The RID solution also binds redundancy RTP streams to their original source RTP streams, as long as those streams use RID identifiers. The redundancy stream uses the RepairedRtpStreamId SDES item and RTP header extension to declare the RtpStreamId of the source stream, and that's how the binding gets made ✨
Experience has shown that an explicit binding between RTP streams, one that doesn't depend on SSRC values, behaves really well 🙌 That way, solutions using multiple RTP streams in a single RTP session and solutions using multiple RTP sessions all use the same type of binding. Consistency is everything, periodt 💜
3.4.4. Forward Error Correction 🛡️
There are a bunch of FEC-based schemes designed to reduce packet loss in the original streams 🦋 Most of them protect a single source flow. They do this by sending a certain amount of redundant information, encoded so it can repair one or more instances of packet loss across the set of packets it protects. That redundant information has to be sent as its own media stream or, in some cases, instead of the original media stream. So many of these schemes need a way to bind related flows, like we talked about above 🔗 Historically, some schemes use multiple SSRCs, some use multiple RTP sessions, and some support both modes 🌸
Using multiple RTP sessions is great when some receivers can't use the FEC information. If you put it in a separate RTP session, and separate those sessions at the transport level, the FEC can be ignored at the transport level without looking at any RTP-layer information at all. Clean and simple 💅
Multicast works similarly. Sending FEC information in a separate multicast group gives the same kind of flexibility 🧚 This is especially useful when receivers see different packet loss rates. A receiver can decide, based on its measured packet loss rates, whether to join a multicast group with suitable FEC data repair capabilities. Personalized repair, bestie 💖
4. Considerations for RTP Multiplexing 🎀
4.1. Interworking Considerations 🌈
There are several kinds of interworking, but this section covers two: interworking directly between different applications, and interworking through an RTP translator. The discussion looks at what happens when the RTP multiplexing point choices might differ, plus some limitations you have to keep in mind when working with certain legacy applications 🕰️
4.1.1. Application Interworking 🤝
It's not uncommon for applications or services with similar (but not identical) usage, especially interactive communication ones, to end up wanting to connect two or more of them 💬
When that happens, you often end up needing a gateway to interconnect the applications. The gateway either has to change the multiplexing structure, or follow the limitations of each application. Choose your fighter 😤
There are two fundamental ways to build a gateway. One is RTP translator interworking (RTP bridging), where the gateway acts as an RTP translator and the two interconnected applications are members of the same RTP session. The other is gateway interworking (Section 4.1.3) with RTP termination, where there are independent RTP sessions between each interconnected application and the gateway 🦋
For interworking to be feasible, whatever security solution is in use has to be compatible, and capable of exchanging keys with either the peer or the gateway under the trust model being used 🔐 Also, the applications need to use media streams in a way that makes sense in both applications. Obvious, but people forget 🙈
4.1.2. RTP Translator Interworking 🔁
From an RTP perspective, the RTP translator approach could work if all the applications use the same codecs with the same payload types, have made the same multiplexing choices, and have the same capabilities regarding the number of simultaneous RTP streams, along with the same set of supported RTP/RTCP extensions. Unfortunately, that might not always be true 😭
When a gateway is implemented as an RTP translator, an important question is whether the two interconnected applications need to use the same multiplexing approach. If one side uses RTP session multiplexing and the other uses SSRC multiplexing with BUNDLE [RFC8843], the RTP translator might be able to map the RTP streams between both sides using some method, e.g., based on the number and order of SDP "m=" lines from each side 🧩 There are also challenges with SSRC collision handling. Unless SSRC translation is applied on the RTP translator, there can be a collision on the SSRC multiplexing side that the RTP session multiplexing side never knows about 👀
And here's another wrinkle: if one application can work in several modes (like using additional RTP streams in one RTP session or in multiple RTP sessions, whichever it wants) and the other can't, a successful connection depends on locking the more flexible application into the one mode that works for interconnection. That happens even if none of the participants are using the less flexible application when the RTP sessions are being created 🥺
4.1.3. Gateway Interworking 🌉
When you terminate RTP sessions at the gateway, the gateway has some jobs it has to handle 📋
- Generating appropriate RTCP reports for all RTP streams (possibly based on incoming RTCP reports) that originate from SSRCs controlled by the gateway.
- Handling SSRC collision resolution in each application's RTP sessions.
- Signaling, choosing, and policing appropriate bitrates for each session.
For applications that use any security mechanism, e.g., in the form of SRTP, the gateway needs to be able to decrypt and verify the source integrity of incoming packets, and then re-encrypt, integrity protect, and sign the packets as the peer in the other application's security context. That's needed even if all you're doing is a simple remapping of SSRC numbers. If the gateway does this, it also has to be a member of the security contexts on both sides, which makes it a trusted entity 🫣
The gateway might also need to apply transcoding (for incompatible codec types), media-level adaptations that can't be solved through media negotiation (like rescaling for incompatible video size requirements), suppression of content known not to be handled in the destination application, or adding or removing redundancy coding or scalability layers to fit the needs of the destination domain 🎞️
So the gateway needs an intimate knowledge of the application requirements. A gateway is application specific by nature and is definitely not a commodity product 🧠
These gateways can also block application evolution. If they block RTP and RTCP extensions that the applications have been extended with, but that the gateway doesn't know about, the newer features just can't get through 🚧
If you use a security mechanism like SRTP, the gateway and the trust the peers have to place in it add an extra risk to communication security 🚨 The gateway also adds complexity with the decrypt-encrypt cycles needed for every forwarded packet. And SRTP, because of its keying structure, requires that each RTP session use different master keys. For some ciphers, using the same key in two RTP sessions can cause a one-time pad to be reused, which completely breaks the confidentiality of the packets. Not a vibe at all 💀🔑
4.1.4. Legacy Considerations for Multiple SSRCs 🦋
okay so historically, the most common RTP setups have been point-to-point VoIP or streaming apps 📞, usually with no more than one media source per endpoint and media type (think audio or video). even in conferencing apps, especially voice-only ones, the conference focus or bridge gives each person one single stream that's a mix of everyone else 🎧. it's also super common to have separate RTP sessions between each endpoint and the RTP mixer, which means the mixer is basically acting as an RTP-terminating gateway 🧚
Applications and systems that haven't been updated to handle multiple streams following these recommendations can run into trouble joining RTP sessions with multiple SSRCs in a single session 😭, like:
- having to handle more than one stream at the same time, instead of swapping an existing stream out for a new one 🔄
- being able to decode multiple streams simultaneously 🎶
- being able to render multiple streams simultaneously 🖼️
so gateways trying to connect to this kind of device have to make sure that only one RTP stream of each media type gets delivered to the endpoint if that's all it expects 🙅♀️, and that the multiplexing format is the one the device expects. honestly, it's highly unlikely that RTP translator-based interworking can work in that context 💀
4.2. Network Considerations 🌐
the RTP implementer needs to keep in mind that the RTP multiplexing choice also affects network-level mechanisms 🧠
4.2.1. Quality of Service 💎
QoS mechanisms are either flow based or packet marking based. RSVP [RFC2205] is an example of a flow-based mechanism, and Diffserv [RFC2474] is an example of a packet-marking-based one.
For flow-based schemes, extra SSRCs get the same QoS as every other RTP stream in the same 5-tuple (protocol, source address, destination address, source port, destination port), which is the most common selector for flow-based QoS ✨
For packet-marking-based schemes, the multiplexing method won't affect whether QoS is possible 🙌. Different Differentiated Services Code Points (DSCPs) can be assigned to different packets within a transport flow (5-tuple) and also within an RTP stream, assuming UDP or other transport protocols that don't have issues with packet reordering within the transport flow (5-tuple) are being used. To avoid reordering issues, packets belonging to the same RTP flow should limit their use of DSCPs to packets whose corresponding Per-Hop Behavior (PHB) doesn't enable reordering. If the transport protocol assumes in-order delivery of packets (e.g., TCP and the Stream Control Transmission Protocol (SCTP)), then a single DSCP should be used. For more on this topic, see [RFC7657].
The method for assigning marking to packets can affect how many RTP sessions to choose. If this marking is done by a network ingress function, it can struggle to tell the different RTP streams apart 🔍. The network API on the endpoint also needs to be able to set the marking on a per-packet basis to get full functionality 💅
4.2.2. NAT and Firewall Traversal 🚧
In today's networks, there are tons of middleboxes 📦. The ones that usually hit RTP the hardest are Network Address Translators (NATs) and Firewalls (FWs).
Below, we break down how requiring more underlying transport flows plays out with NATs and FWs 🔎:
Endpoint Port Consumption: A given IP address only has 65536 local ports per transport protocol for all the consumers of ports on the machine. This is normally never an issue for an end-user machine. It can become one for servers handling a huge number of simultaneous streams 🖥️. However, if the application uses ICE to authenticate STUN requests, a server can serve multiple endpoints from the same local port and use the whole 5-tuple (source and destination address, source and destination port, protocol) as the identifier of flows, after securely binding them to the remote endpoint address using the STUN request 🔐. In theory, the minimum number of media server ports needed is the maximum number of simultaneous RTP sessions a single endpoint can use. In practice, implementations will probably benefit from using more server ports to simplify implementation or avoid performance bottlenecks 🚀.
NAT State: If an endpoint sits behind a NAT, each flow it generates to an external address results in state that has to be kept in the NAT. That state is a limited resource 🫠. In home or Small Office/Home Office (SOHO) NATs, the most limited resource is memory or processing. For large-scale NATs serving many internal endpoints, available external ports are likely the scarce resource. Port limitations are mainly a problem for larger centralized NATs where endpoint-independent mapping requires each flow to use one port for the external IP address. This affects the maximum number of internal users per external IP address. But as a comparison, a real-time video conference session with audio and video likely uses less than 10 UDP flows, compared to certain web applications that can use 100+ TCP flows to various servers from a single browser instance 🤯.
Extra Delay Added by NAT Traversal: Performing the NAT/FW traversal takes a certain amount of time for each flow ⏱️. The best-case scenario for additional NAT/FW traversal time after finding the first valid candidate pair following the specified ICE procedures is 1.5*RTT + Ta*(Additional_Flows-1), where Ta is the pacing timer. That assumes a message in one direction, immediately followed by a return message in the opposite direction to confirm reachability. It isn't more, because ICE first finds one candidate pair that works, before attempting to establish multiple flows. So there is no extra time until a working candidate pair has been found. Based on that working pair, the extra time is needed to establish the additional flows (two or three, in most cases) in parallel. However, packet loss causes extra delays of at least 500 ms (the minimal retransmission timer for ICE) 🐢.
NAT Traversal Failure Rate: Because you need to establish more than a single flow through the NAT, there's some risk that establishing the first flow will succeed but one or more of the additional flows will fail. The risk of this happening is hard to quantify, but it should be fairly low, since one flow from the same interfaces has just been successfully established. So only rare events, like NAT resource overload or selecting particular port numbers that are filtered, etc., ought to be reasons for failure 🤞.
Deep Packet Inspection and Multiple Streams: FWs differ in how deeply they inspect packets 🔬. Previous experience using FWs and Session Border Gateways (SBGs) with RTP shows that there is a significant risk that they will reject RTP sessions that use multiple SSRCs 🚫.
Using additional RTP streams in the same RTP session and transport flow doesn't add any extra NAT traversal complexity per RTP stream ✅. This can be compared with (normally) one or two additional transport flows per RTP session when using multiple RTP sessions. Additional lower-layer transport flows will be needed, unless an explicit demultiplexing layer is added between RTP and the transport protocol. At the time of this writing, no such mechanism was defined 💜
4.2.3. Multicast 🦋
Okay bestie, multicast groups are a seriously powerful tool for real-time apps, especially ones that want broadcast-like vibes, where one endpoint sends to a whole crowd of receivers, like in IPTV 📺. There's also an RTP/RTCP extension to better support Source-Specific Multicast (SSM) [RFC5760]. And many-to-many communication, which RTP [RFC3550] was originally built to support, has a few limitations in common with multicast 💅.
One limitation is that, for any group, sender-side adaptations meant to suit everyone would have to adapt to the most limited receiver, the one with the worst conditions in the group, which drags everyone down 😭. For broadcast with a large number of receivers, that is not acceptable 💀. So instead, various receiver-based solutions get used so each receiver can achieve the best performance it can 🌈. By using scalable encoding and putting each scalability layer in its own multicast group, the receiver gets to control how much traffic it takes in 🎀. To get each layer into a different multicast group, you use one RTP session per multicast group 🧚.
The transport flow stuff in multicast is also a bit different from unicast. NATs with port translation aren't useful in the multicast environment, which means the entire port range of each multicast address is available for telling RTP sessions apart 🦋.
So for broadcast apps, it seems easiest and most straightforward to use multiple RTP sessions for sending the different media flows used to adapt to network conditions ✨. It's also common for streams that improve transport robustness to get their own multicast group, so they can interwork with legacy applications or support different levels of protection 🍓.
Many-to-many apps have different needs, and the most appropriate multiplexing choice depends on how the actual app is realized 👑. Multicast apps that can use sender-side congestion control can skip the multiple multicast sessions and RTP sessions that come from receiver-side congestion control 💖.
The properties of a broadcast app using RTP multicast are:
- The app uses a group of RTP sessions, not just one. Each endpoint will need to be a member of a number of RTP sessions in order to perform well.
- Within each RTP session, the number of RTP receivers is likely to be much larger than the number of RTP senders.
- The app needs signaling functions to identify the relationships between RTP sessions.
- The app needs signaling or RTP/RTCP functions to identify the relationships between SSRCs in different RTP sessions when more complex relations than those that can be expressed by the CNAME exist.
Both broadcast and many-to-many multicast apps share a signaling requirement: all of the participants need the same RTP and payload type configuration. Otherwise, A could, for example, be using payload type 97 as the video codec H.264 while B thinks it's MPEG-2 😱. SDP offer/answer [RFC3264] is not appropriate for ensuring this property in a broadcast/multicast context 🙅. The signaling aspects of broadcast/multicast aren't explored further in this memo.
Security solutions for this kind of group communication are also challenging 🔐. First, the key-management mechanism and the security protocol need to support group communication. Second, source authentication requires special solutions. For more discussion on this topic, please review "Options for Securing RTP Sessions" [RFC7201].
4.3. Security and Key-Management Considerations 🔒
When dealing with point-to-point two-member RTP sessions only, there are few security issues relevant to choosing one RTP session or multiple RTP sessions. However, a few aspects of multi-party sessions might warrant consideration 🫶. For general information about possible methods of securing RTP, please review [RFC7201].
4.3.1. Security Context Scope 🗝️
When using SRTP [RFC3711], the security context scope is important and can be a necessary differentiation in some applications. SRTP's crypto suites are (so far) built around symmetric keys, so the receiver will need to have the same key as the sender. As a result, no one in a multi-party session can be certain that a received packet was really sent by the claimed sender and not by another party with access to the key 😬. The single SRTP algorithm without this property is Timed Efficient Stream Loss-Tolerant Authentication (TESLA) source authentication [RFC4383]. However, TESLA adds delay to achieve source authentication. In most cases, symmetric ciphers provide sufficient security properties, but in a few cases they can create issues.
The first case is when someone leaves a multi-party session and you want to make sure that party can no longer access the RTP streams. This requires that everyone rekey without disclosing the new keys to the excluded party 🚪.
A second case is when security is used as an enforcing mechanism for stream access differentiation between different receivers. Take, for example, a scalable layer or a high-quality simulcast version that only users paying a premium are allowed to access. The mechanism preventing a receiver from getting the high-quality stream can be based on the stream being encrypted with a key that users can't access without paying a premium, using the key-management mechanism to limit access to the key 💎.
As specified in [RFC3711], SRTP uses unique keys per SSRC; however, the original assumption was a single-session master key from which SSRC-specific RTP and RTCP keys were derived. That assumption was proven incorrect, as application usage and the developed key-management mechanisms have chosen many different methods for ensuring unique keys per SSRC 🤯. The key-management functions have different abilities to establish different sets of keys, normally on a per-endpoint basis. For example, DTLS-SRTP [RFC5764] and Security Descriptions [RFC4568] establish different keys for outgoing and incoming traffic from an endpoint. This key usage has to be written into the cryptographic context, possibly associated with different SSRCs. So limitations do exist, depending on the chosen key-management method and on the integration of particular implementations of the key-management method and SRTP 🧩.
4.3.2. Key Management for Multi-party Sessions 🔑
The capabilities of the key-management method, combined with the RTP multiplexing choices, affect the resulting security properties, control over the secured media, and who has access to it.
Multi-party sessions contain at least one RTP stream from each active participant. Depending on the multi-party topology [RFC7667], each participant can both send and receive multiple RTP streams. Transport translator-based sessions (Topo-Trn-Translator) and multicast sessions (Topo-ASM) can use neither Security Descriptions [RFC4568] nor DTLS-SRTP [RFC5764] without an extension, because each endpoint provides its own set of keys. In centralized conferences, the signaling counterpart is a conference server, and the transport translator is the media-plane unicast counterpart (to which DTLS messages would be sent). Thus, an extension like Encrypted Key Transport [RFC8870], or a solution based on Multimedia Internet KEYing (MIKEY) [RFC3830] that allows for keying all session participants with the same master key, is needed.
Privacy-Enhanced RTP Conferencing (PERC) also enables a different trust model with semi-trusted media-switching RTP middleboxes [RFC8871] 🛡️.
4.3.3. Complexity Implications 🧠
There can be complex interactions between the choice of multiplexing and topology and the security functions. This becomes especially evident in RTP topologies with any type of middlebox that processes or modifies RTP/RTCP packets. While the overhead of an RTP translator or mixer rewriting an SSRC value in the RTP packet of an unencrypted session is low, the cost is higher when using cryptographic security functions. For example, if using SRTP [RFC3711], the actual security context and exact crypto key are determined by the SSRC field value. If you change the SSRC value, the encryption and authentication must use another key. So changing the SSRC value implies a decryption using the old SSRC and its security context, followed by an encryption using the new one 🧁.
5. RTP Multiplexing Design Choices ✨
This section discusses how some RTP multiplexing design choices can be used in applications to achieve certain goals, and it summarizes the implications of those choices. The benefits and downsides of each design are also discussed 💜.
5.1. Multiple Media Types in One Session 🎬
This design uses a single RTP session for multiple different media types, like audio and video, and possibly also transport robustness mechanisms like FEC or retransmission. An endpoint can send zero, one, or multiple media sources per media type, resulting in a number of RTP streams of various media types for both source and redundancy streams.
Advantages:
- Only a single RTP session is used, which implies:
- Minimal need to keep NAT/FW state.
- Minimal NAT/FW traversal cost.
- Fate-sharing for all media flows.
- Minimal overhead for security association establishment.
- Dynamic allocation of RTP streams can be handled almost entirely at the RTP level. The extent to which this allocation can be kept at the RTP level depends on the application's needs for an explicit indication of stream usage and on how timely a fashion that information can be signaled ⏱️.
Disadvantages:
- It is less suitable for interworking with other applications that use individual RTP sessions per media type or multiple sessions for a single media type, due to the risk of SSRC collisions and thus a potential need for SSRC translation 😵.
- Negotiation of individual bandwidths for the different media types is currently only possible in SDP when using RID [RFC8851].
- It is not suitable for split component terminals (see Section 3.10 of [RFC7667]).
- Flow-based QoS cannot be used to provide separate treatment of RTP streams compared to others in the single RTP session.
- If there is significant asymmetry between the RTP streams' RTCP reporting needs, there are some challenges related to configuration and usage to avoid wasting RTCP reporting on the RTP stream that does not need such frequent reporting.
- It is not suitable for applications where some receivers like to receive only a subset of the RTP streams, especially if multicast or a transport translator is being used.
- There are some additional concerns regarding legacy implementations that do not support the RTP specification fully when it comes to handling multiple SSRCs per endpoint, as multiple simultaneous media types are sent as separate SSRCs in the same RTP session 🕰️.
- If the applications need finer control over which session participants are included in different sets of security associations, most key-management mechanisms will have difficulties establishing such a session 🙈.
Okay bestie, picking up right where we left off 🫶
5.2. Multiple SSRCs of the Same Media Type ✨
So in this design, each RTP session only serves ONE media type 🎀 The session can hold a bunch of RTP streams, from one endpoint or from several endpoints. Usually that means a low number of sessions, basically one for audio and one for video, which means you need two listening ports when you're doing RTP/RTCP multiplexing [RFC5761]. Lowkey so clean 💅
Advantages:
- It vibes perfectly with split component terminals (see Section 3.10 of [RFC7667]) where the split is per media type. 🦋
- It lets you do flow-based QoS with different priority levels between media types. 🔥
- For apps where streams are constantly getting added and removed (like, constantly), most of the state lives at the RTP session level instead of per individual SSRC. So you don't need in-session signaling about every single SSRC. In simple cases you can have unsignaled RTP streams, where session-level info plus an RTCP SDES item (like CNAME) is enough. 🧚 For the more complex stuff where you need more source-specific metadata, the SSRC can be tied to an intermediate identifier, like the MID sent as an SDES item, defined in Section 15 of [RFC8843]. ✨
- The overhead of setting up security associations is low. 🔐💖
Disadvantages:
- You end up needing a slightly higher number of RTP sessions compared to having multiple media types in one session (Section 5.1). Which means:
- More NAT/FW state is needed. 😭
- NAT/FW traversal costs more, in both processing and delay. 💀
- There's some potential for concern with legacy implementations that don't fully support the RTP specification when it comes to handling multiple SSRCs per endpoint. 🙈
- You can't control security associations for sets of RTP streams within the same media type with today's key-management mechanisms, unless you split them into different RTP sessions (Section 5.3). 🥺
Not gonna lie, this is the trade-off that bites people. Keep it in mind 👀
For RTP apps where every stream of the same media type has the same usage, this structure gives you efficiency gains in network state and more fate-sharing with other flows of the same type. 🍓 It also keeps almost all the functionality for negotiating per-media-type properties in signaling, and still lets you do flow-based QoS prioritization between media types. It handles multi-party sessions really well, whether or not you're using multicast or centralized transport distribution, since extra sources can join and leave the session whenever they want. Iykyk 👑
5.3. Multiple Sessions for One Media Type 🌈
This design goes one step further than Section 5.2 by ALSO using multiple RTP sessions for a single media type. The main reason you'd do this is when the app needs to separate RTP streams by their usage, like scalability over multicast, simulcast, extended QoS prioritization, or fine-grained signaling using RTP session-focused signaling tools. Main character energy 👸
Advantages:
- This is way better for multicast, since receivers can pick which RTP sessions they want to join, as long as each RTP session has its own multicast group. 🫶
- When there are multiple different usages, the app can indicate how it uses each RTP stream at the RTP session level. 🎀
- There's less need for SSRC-specific explicit signaling for each media stream, so you don't need as much explicit, timely signaling when streams get added or removed. 💅
- It enables detailed QoS prioritization for flow-based mechanisms. 🔥
- It works well with split component terminals (see Section 3.10 of [RFC7667]). 🦋
- The scope of who's included in a security association can be structured around the different RTP sessions, so you can do that with existing key-management mechanisms. 🔐✨
Disadvantages:
- There's more session configuration state compared to multiple SSRCs of the same media type (Section 5.2), because there are more RTP sessions. 😭
- For RTP streams that are part of scalability, simulcast, or transport robustness, you need a method for binding sources across multiple RTP sessions. 🧩
- Same legacy concern as before: there's some potential for worry with legacy implementations that don't fully support the RTP specification when handling multiple SSRCs per endpoint. 🙈
- The overhead of security association establishment is higher, because there are more RTP sessions. 💀
- If the apps need finer control over which participants in a given RTP session are included in different sets of security associations, most of today's key-management mechanisms will struggle to set that up. 🥲
For more complex RTP apps that have several different usages for streams of the same media type, or that use scalability or simulcast, this solution makes those features possible, at the cost of extra overhead from the additional sessions. 🌸 This structure is great for more advanced apps, and also for multicast-based apps that need to tell different participants apart. 🧚
5.4. Single SSRC per Endpoint 🦋
In this design, each endpoint in a point-to-point session has only ONE SSRC, so the RTP session only has two SSRCs: one local and one remote. 💖 It can be used unidirectionally (one SSRC sends an RTP stream that the other SSRC receives) or bidirectionally (both SSRCs send an RTP stream and receive the one the other endpoint sends). If the app needs more media flows between the endpoints, it has to set up additional RTP sessions. 🙃
Advantages:
- This design has great interoperability potential with legacy apps, since it won't tax any RTP stack implementations. 🫶
- The signaling system lets you negotiate and describe the exact formats and bitrates for each RTP stream, especially with today's SDP tools. 📝
- You can control security associations per RTP stream with current key-management functions, because each RTP stream maps directly to an RTP session, and the most commonly used keying mechanisms work per session. 🔐
Disadvantages:
- The amount of NAT/FW state grows linearly with the number of RTP streams. 📈😭
- NAT/FW traversal increases delay and resource consumption. 💀
- There are likely more signaling messages and more signaling processing needed, because there's more session-related info. 🥱
- There's a higher chance that a single RTP stream fails in transport between endpoints, since every stream needs its own NAT/FW traversal and there's only one stream per session. 😬
- The amount of explicit state for relating RTP streams grows, depending on how the app relates them. 🧠
- Port consumption might become a problem for centralized services, where the central node's port or 5-tuple filter consumption grows fast with the number of sessions. 🚨
- For apps where usage is super dynamic (entities constantly entering and leaving sessions), the signaling can get intense. Timely establishment of extra RTP sessions can also cause issues. 🌀
- If, against the recommendation in [RFC3550], the same SSRC value gets reused across multiple RTP sessions instead of being randomly chosen, interworking with apps that use a different multiplexing structure will require SSRC translation. 🙅♀️
RTP apps that really need to interwork with legacy RTP apps can benefit from this structure. 🎀 Still, a large number of media descriptions in SDP can run into issues with existing implementations. For any app needing lots of media flows, the overhead can get very significant. 😵 This structure also isn't suitable for non-mixed multi-party sessions, because every RTP stream from each participant, even with the same usage in the app, needs its own RTP session. 🥴 On top of that, the dynamic behavior that can happen in multi-party apps can strain the signaling system and make timely media establishment harder. 💅
5.5. Summary 💜
Both the "single SSRC per endpoint" (Section 5.4) and "multiple media types in one session" (Section 5.1) cases need full explicit signaling of media stream relationships. 🧩 They work on two different levels, though: the first mainly enables session-level binding, and the second needs SSRC-level binding. Basically, these two are the two extremes when it comes to how many RTP sessions you need. 🎀
The other two designs, multiple SSRCs of the same media type (Section 5.2) and multiple sessions for one media type (Section 5.3), are examples that mainly allow some implicit mapping of the role or usage of RTP streams, based on which RTP session they appear in. ✨ So they can potentially need less signaling and reduce the need for real-time signaling in sessions where the number of RTP streams keeps changing. 🌈 They also sit between the first two designs in how many RTP sessions get set up, which means they try to balance the number of sessions against the functionality the communication session gives you, at both the network and signaling levels. 🫶💖
More parts to come, so stay tuned bestie 💅🎀
6. Guidelines 📝✨
Okay bestie, this section is basically the multi-stream cheat sheet 💅 It's a bunch of guidelines for implementers, system designers, and spec writers, so let's go 🧚
Do not require the use of the same SSRC value across RTP sessions: as we talked about in Section 3.4.3, reusing the same SSRC across RTP sessions to bind related RTP streams together has downsides 💀 So instead, it is recommended to use a mechanism that explicitly signals the relationship, either in RTP/RTCP or in the signaling mechanism used to set up the RTP session(s). 🫶
Use additional RTP streams for additional media sources: if an RTP endpoint needs to send more RTP streams of the same media type in the application, and they have the same processing requirements at the network and RTP layers, it is suggested to send them in the same RTP session. 🎀 Picture a telepresence room with three cameras where each camera catches two people sitting at the table. We suggest that each camera sends its own RTP stream within a single RTP session 📸 so cute, right?
Use additional RTP sessions for streams with different requirements: when RTP streams have different processing requirements from the network or the RTP layer at the endpoints, it is suggested that the different types of streams go in different RTP sessions 🦋 This also covers the case where different participants want different subsets of the set of RTP streams. 👑
Use grouping when using multiple RTP sessions: when you're using multiple RTP session solutions, it is suggested to explicitly group the involved RTP sessions when needed, using a signaling mechanism. For example, see "The Session Description Protocol (SDP) Grouping Framework" [RFC5888], and pick some appropriate grouping semantics 🧠
Ensure that RTP/RTCP extensions support multiple RTP streams as well as multiple RTP sessions: when defining an RTP or RTCP extension, the creator needs to think about whether it applies to additional SSRCs and multiple RTP sessions. Any extension meant to be generic must support both 🔥 Extensions that aren't as generally applicable will have to weigh whether interoperability is better served by defining a single solution or by providing both options. 🤔
Provide adequate extensions for transport support: when defining new RTP/RTCP extensions for transport support, like the retransmission or FEC mechanisms, they must include support for both multiple RTP streams in the same RTP session and multiple RTP sessions. That way, application developers can pick freely from the set of mechanisms without stressing about which multiplexing choice a particular solution supports 💖
7. IANA Considerations 📋
This document has no IANA actions. 🌸
8. Security Considerations 🔐
Okay, security is serious business, so listen up 😭 The security stuff from the RTP specification [RFC3550] applies, and so does any RTP profile you pick [RFC3551] [RFC4585] [RFC3711]. The same goes for the extensions for sending multiple media types in a single RTP session [RFC8860], RID [RFC8851], BUNDLE [RFC8843], [RFC5760], and [RFC5761]. These only apply if you select them, but if you do, they need to be considered in your evaluation 🛡️
Section 4.3 spills the tea on the security implications of choosing multiple SSRCs vs. multiple RTP sessions 💅
9. References 📚
9.1. Normative References
Here are the receipts 🧾
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/info/rfc3551>.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, DOI 10.17487/RFC3711, March 2004,
<https://www.rfc-editor.org/info/rfc3711>.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585,
DOI 10.17487/RFC4585, July 2006,
<https://www.rfc-editor.org/info/rfc4585>.
[RFC5576] Lennox, J., Ott, J., and T. Schierl, "Source-Specific
Media Attributes in the Session Description Protocol
(SDP)", RFC 5576, DOI 10.17487/RFC5576, June 2009,
<https://www.rfc-editor.org/info/rfc5576>.
[RFC5760] Ott, J., Chesterfield, J., and E. Schooler, "RTP Control
Protocol (RTCP) Extensions for Single-Source Multicast
Sessions with Unicast Feedback", RFC 5760,
DOI 10.17487/RFC5760, February 2010,
<https://www.rfc-editor.org/info/rfc5760>.
[RFC5761] Perkins, C. and M. Westerlund, "Multiplexing RTP Data and
Control Packets on a Single Port", RFC 5761,
DOI 10.17487/RFC5761, April 2010,
<https://www.rfc-editor.org/info/rfc5761>.
[RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and
B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms
for Real-Time Transport Protocol (RTP) Sources", RFC 7656,
DOI 10.17487/RFC7656, November 2015,
<https://www.rfc-editor.org/info/rfc7656>.
[RFC7667] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667,
DOI 10.17487/RFC7667, November 2015,
<https://www.rfc-editor.org/info/rfc7667>.
[RFC8843] Holmberg, C., Alvestrand, H., and C. Jennings,
"Negotiating Media Multiplexing Using the Session
Description Protocol (SDP)", RFC 8843,
DOI 10.17487/RFC8843, January 2021,
<https://www.rfc-editor.org/info/rfc8843>.
[RFC8851] Roach, A.B., Ed., "RTP Payload Format Restrictions",
RFC 8851, DOI 10.17487/RFC8851, January 2021,
<https://www.rfc-editor.org/info/rfc8851>.
[RFC8852] Roach, A.B., Nandakumar, S., and P. Thatcher, "RTP Stream
Identifier Source Description (SDES)", RFC 8852,
DOI 10.17487/RFC8852, January 2021,
<https://www.rfc-editor.org/info/rfc8852>.
[RFC8860] Westerlund, M., Perkins, C., and J. Lennox, "Sending
Multiple Types of Media in a Single RTP Session",
RFC 8860, DOI 10.17487/RFC8860, January 2021,
<https://www.rfc-editor.org/info/rfc8860>.
[RFC8870] Jennings, C., Mattsson, J., McGrew, D., Wing, D., and F.
Andreasen, "Encrypted Key Transport for DTLS and Secure
RTP", RFC 8870, DOI 10.17487/RFC8870, January 2021,
<https://www.rfc-editor.org/info/rfc8870>.9.2. Informative References
Okay so bestie, this is the informative references list, aka the big ol' reading list 📚💖 Nothing in here is required to implement the spec, but it's where all the cool friends it talks to live. Here we go ✨
[JINGLE] Ludwig, S., Beda, J., Saint-Andre, P., McQueen, R., Egan,
S., and J. Hildebrand, "XEP-0166: Jingle", September 2018,
<https://xmpp.org/extensions/xep-0166.html>.
[RFC2198] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V.,
Handley, M., Bolot, J.C., Vega-Garcia, A., and S. Fosse-
Parisis, "RTP Payload for Redundant Audio Data", RFC 2198,
DOI 10.17487/RFC2198, September 1997,
<https://www.rfc-editor.org/info/rfc2198>.
[RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S.
Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1
Functional Specification", RFC 2205, DOI 10.17487/RFC2205,
September 1997, <https://www.rfc-editor.org/info/rfc2205>.
[RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black,
"Definition of the Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers", RFC 2474,
DOI 10.17487/RFC2474, December 1998,
<https://www.rfc-editor.org/info/rfc2474>.
[RFC2974] Handley, M., Perkins, C., and E. Whelan, "Session
Announcement Protocol", RFC 2974, DOI 10.17487/RFC2974,
October 2000, <https://www.rfc-editor.org/info/rfc2974>.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261,
DOI 10.17487/RFC3261, June 2002,
<https://www.rfc-editor.org/info/rfc3261>.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model
with Session Description Protocol (SDP)", RFC 3264,
DOI 10.17487/RFC3264, June 2002,
<https://www.rfc-editor.org/info/rfc3264>.
[RFC3389] Zopf, R., "Real-time Transport Protocol (RTP) Payload for
Comfort Noise (CN)", RFC 3389, DOI 10.17487/RFC3389,
September 2002, <https://www.rfc-editor.org/info/rfc3389>.
[RFC3830] Arkko, J., Carrara, E., Lindholm, F., Naslund, M., and K.
Norrman, "MIKEY: Multimedia Internet KEYing", RFC 3830,
DOI 10.17487/RFC3830, August 2004,
<https://www.rfc-editor.org/info/rfc3830>.
[RFC4103] Hellstrom, G. and P. Jones, "RTP Payload for Text
Conversation", RFC 4103, DOI 10.17487/RFC4103, June 2005,
<https://www.rfc-editor.org/info/rfc4103>.
[RFC4383] Baugher, M. and E. Carrara, "The Use of Timed Efficient
Stream Loss-Tolerant Authentication (TESLA) in the Secure
Real-time Transport Protocol (SRTP)", RFC 4383,
DOI 10.17487/RFC4383, February 2006,
<https://www.rfc-editor.org/info/rfc4383>.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, DOI 10.17487/RFC4566,
July 2006, <https://www.rfc-editor.org/info/rfc4566>.
[RFC4568] Andreasen, F., Baugher, M., and D. Wing, "Session
Description Protocol (SDP) Security Descriptions for Media
Streams", RFC 4568, DOI 10.17487/RFC4568, July 2006,
<https://www.rfc-editor.org/info/rfc4568>.
[RFC4588] Rey, J., Leon, D., Miyazaki, A., Varsa, V., and R.
Hakenberg, "RTP Retransmission Payload Format", RFC 4588,
DOI 10.17487/RFC4588, July 2006,
<https://www.rfc-editor.org/info/rfc4588>.
[RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman,
"Codec Control Messages in the RTP Audio-Visual Profile
with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104,
February 2008, <https://www.rfc-editor.org/info/rfc5104>.
[RFC5109] Li, A., Ed., "RTP Payload Format for Generic Forward Error
Correction", RFC 5109, DOI 10.17487/RFC5109, December
2007, <https://www.rfc-editor.org/info/rfc5109>.
[RFC5389] Rosenberg, J., Mahy, R., Matthews, P., and D. Wing,
"Session Traversal Utilities for NAT (STUN)", RFC 5389,
DOI 10.17487/RFC5389, October 2008,
<https://www.rfc-editor.org/info/rfc5389>.
[RFC5764] McGrew, D. and E. Rescorla, "Datagram Transport Layer
Security (DTLS) Extension to Establish Keys for the Secure
Real-time Transport Protocol (SRTP)", RFC 5764,
DOI 10.17487/RFC5764, May 2010,
<https://www.rfc-editor.org/info/rfc5764>.
[RFC5888] Camarillo, G. and H. Schulzrinne, "The Session Description
Protocol (SDP) Grouping Framework", RFC 5888,
DOI 10.17487/RFC5888, June 2010,
<https://www.rfc-editor.org/info/rfc5888>.
[RFC6465] Ivov, E., Ed., Marocco, E., Ed., and J. Lennox, "A Real-
time Transport Protocol (RTP) Header Extension for Mixer-
to-Client Audio Level Indication", RFC 6465,
DOI 10.17487/RFC6465, December 2011,
<https://www.rfc-editor.org/info/rfc6465>.
[RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP
Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014,
<https://www.rfc-editor.org/info/rfc7201>.
[RFC7657] Black, D., Ed. and P. Jones, "Differentiated Services
(Diffserv) and Real-Time Communication", RFC 7657,
DOI 10.17487/RFC7657, November 2015,
<https://www.rfc-editor.org/info/rfc7657>.
[RFC7826] Schulzrinne, H., Rao, A., Lanphier, R., Westerlund, M.,
and M. Stiemerling, Ed., "Real-Time Streaming Protocol
Version 2.0", RFC 7826, DOI 10.17487/RFC7826, December
2016, <https://www.rfc-editor.org/info/rfc7826>.
[RFC7983] Petit-Huguenin, M. and G. Salgueiro, "Multiplexing Scheme
Updates for Secure Real-time Transport Protocol (SRTP)
Extension for Datagram Transport Layer Security (DTLS)",
RFC 7983, DOI 10.17487/RFC7983, September 2016,
<https://www.rfc-editor.org/info/rfc7983>.
[RFC8088] Westerlund, M., "How to Write an RTP Payload Format",
RFC 8088, DOI 10.17487/RFC8088, May 2017,
<https://www.rfc-editor.org/info/rfc8088>.
[RFC8108] Lennox, J., Westerlund, M., Wu, Q., and C. Perkins,
"Sending Multiple RTP Streams in a Single RTP Session",
RFC 8108, DOI 10.17487/RFC8108, March 2017,
<https://www.rfc-editor.org/info/rfc8108>.
[RFC8445] Keranen, A., Holmberg, C., and J. Rosenberg, "Interactive
Connectivity Establishment (ICE): A Protocol for Network
Address Translator (NAT) Traversal", RFC 8445,
DOI 10.17487/RFC8445, July 2018,
<https://www.rfc-editor.org/info/rfc8445>.
[RFC8871] Jones, P., Benham, D., and C. Groves, "A Solution
Framework for Private Media in Privacy-Enhanced RTP
Conferencing (PERC)", RFC 8871, DOI 10.17487/RFC8871,
January 2021, <https://www.rfc-editor.org/info/rfc8871>.That's the whole reading list, bestie 🎀 Ready for part 10 whenever you are 💜
Appendix A. Dismissing Payload Type Multiplexing 🎀
okay bestie, this appendix is basically a big ol' "why we don't do this" list 💅. It explains why using the payload type as a multiplexing point is unsuitable for most issues with multiple RTP streams. Doing it anyway has well-known negative effects on RTP 💀. If the payload type is the only thing telling streams apart, then every stream is sending with the same SSRC, which means they all share the same timestamp and sequence number space. Here are the many effects, served in full 🔥:
- Timestamp rate gets boxed in 📏. The RTP timestamp rate for the multiplexed media is constrained. RTP streams with different timestamp rates can't be combined, because the timestamp values have to stay consistent across all multiplexed media frames. So streams are forced to use the same RTP timestamp rate. If that's not possible, payload type multiplexing is off the table 🙅♀️.
- Fragments need to stay in line 🧩. Lots of RTP payload formats can split a media object across multiple RTP packets, like parts of a video frame. These formats need to figure out the order of the fragments to decode correctly. So all fragments related to a frame or similar media object need to be sent in sequence and without interruptions within the object. The sender can handle this pretty easily by making sure the fragments of each RTP stream go out in sequence ✨.
- Some formats need unbroken sequence numbers 🔢. Certain media formats need an uninterrupted sequence number space between media parts. For these, any missing RTP sequence number leads to a decoding failure or triggers a repair mechanism within a single media context. The text/t140 payload format [RFC4103] is one example. These formats would need a sequence numbering abstraction function between RTP and the individual RTP stream before they could be used with payload type multiplexing 🛠️.
- Who lost the packet?? 😭 Sending multiple media streams in the same sequence number space makes it impossible to tell which media stream lost a packet. That's a real problem, because the receiver can't work out which stream should get packet-loss concealment or other stream-specific loss-mitigation mechanisms.
- Retransmission gets messy 🙈. If RTP retransmission [RFC4588] is used and packet loss happens, you can ask for missing packets by SSRC and sequence number, not by payload type. If you only care about some of the payload type multiplexed streams, there's no way to tell which missing packets belong to the streams you want. So all lost packets have to be requested, which wastes bandwidth 💸.
- Feedback has no payload type field 📭. The current RTCP feedback mechanisms are built around feedback on RTP streams based on stream ID (SSRC), packet (sequence numbers), and time interval (RTP timestamps). There's almost never a field to say which payload type is being reported. So sending feedback for a specific RTP payload type is hard unless existing RTCP reporting gets extended 🥲.
- Media control is SSRC-only 🎛️. The current RTCP media control messages specification [RFC5104] is oriented around controlling particular media flows. In other words, requests are made by addressing a particular SSRC. Those mechanisms would need to be redefined to support payload type multiplexing.
- Payload types are limited, bestie 🚫. The number of payload types is inherently limited. So payload type multiplexing caps how many streams can be multiplexed and it just doesn't scale. It gets worse with solutions like RTP and RTCP multiplexing [RFC5761], where a number of payload types are blocked because RTP and RTCP overlap 😩.
- Grouping is missing 🗂️. Sometimes you need to group multiplexed streams. That's currently possible for RTP sessions and SSRCs, but there's no defined way to group payload types.
- No per-stream bandwidth 📉. It's currently not possible to signal bandwidth requirements per RTP stream when using payload type multiplexing.
- SDP attributes don't stretch that far 📄. Most existing SDP media-level attributes can't be applied on a per-payload-type basis. They would need to be redefined for that context.
- Legacy endpoints get confused 🤯. A legacy endpoint that doesn't understand that different RTP payload types are different RTP streams might be a little confused by the big pile of possibly overlapping or identically defined RTP payload types. Honestly, relatable 🙃.
Appendix B. Signaling Considerations 📞
Signaling isn't an architectural consideration for RTP itself, so this discussion got moved to an appendix. BUT it's extremely important for anyone building complete applications, so it still deserves a proper look ✨.
We document some issues here that need addressing when you use some form of signaling to set up RTP sessions. These issues can't be fixed by just tweaking, extending, or profiling RTP. They need a dedicated, in-depth look at the signaling primitives that set up the RTP sessions 🧐.
There are various signaling solutions for establishing RTP sessions. Many are based on SDP [RFC4566], but SDP's functionality also depends on the signaling protocols carrying the SDP. The Real-Time Streaming Protocol (RTSP) [RFC7826] and the Session Announcement Protocol (SAP) [RFC2974] both use SDP in a declarative fashion. SIP [RFC3261] uses SDP with the added definition of offer/answer [RFC3264]. The impact on signaling, especially on SDP, needs to be considered, because it can really affect how you deploy a certain multiplexing point choice 💅.
B.1. Session-Oriented Properties
One thing about existing signaling protocols is that they focus on RTP sessions or, in SDP's case, the concept of media descriptions. A bunch of things get signaled at the media description level. Those aren't strictly bound to an RTP session, though, and could be really useful for signaling, especially for a particular RTP stream (SSRC) within the session. These properties have been identified as potentially useful for signaling, and not only at the RTP session level:
- Bitrate/bandwidth is basically an aggregate vibe 📊. Bitrate and/or bandwidth can only be specified today as an aggregate limit, or as a common "any RTP stream" limit. The exception is codec-specific bandwidth limiting, or RTCP signaling using Temporary Maximum Media Stream Bit Rate Request (TMMBR) messages [RFC5104].
- Which SSRC uses which payload type 🎯. This info shows up in the first media packet, but it's sometimes super useful to have before the packet arrives.
Some of these issues are clearly SDP's problem rather than RTP limitations 🙄. But if the goal is a solution that uses several SSRCs and has several sets of RTP streams with different properties (encoding/packetization parameters, bitrate, etc.), putting each set in a different RTP session would directly enable negotiating the parameters for each set 🙌. If you insist on additional SSRCs only, you'd need a number of signaling extensions to make clear that there are multiple sets of RTP streams with different properties, and that they really need to be kept different. A single set just won't satisfy the application's requirements.
For some parameters, like RTP payload type, resolution, and frame rate, an SSRC-linked mechanism has been proposed in [RFC8851] ✨.
B.2. SDP Prevents Multiple Media Types
SDP uses the "m=" line to both delineate an RTP session and specify the top-level media type: audio, video, text, image, application 🎬. That media type is used as the top-level media type for identifying the actual payload format, and it's bound to a particular payload type using the "a=rtpmap:" attribute. That binding has to be loosened to use SDP for describing RTP sessions with multiple top-level media types. Not me being dramatic, it's literally what has to happen 😭.
[RFC8843] describes how to let multiple SDP media descriptions use a single underlying transport in SDP. That allows defining one RTP session with different top-level media types 🌈.
B.3. Signaling RTP Stream Usage
RTP streams carried in RTP have a particular usage in an RTP application. In lots of applications so far, this usage is signaled implicitly. For example, an app might take all incoming audio RTP streams, mix them, and play them out 🎧. But in more advanced applications that use multiple RTP streams, there's more than one usage or purpose among the streams being sent or received. RTP applications will need to somehow signal this usage. The signaling has to identify the RTP streams affected by their RTP-level identifiers, which means identifying them by either their session or their SSRC + session 🔑.
In some applications, the receiver can't use the RTP stream at all until it has received the signaling message describing the stream and its usage. In other applications, there's a default handling method that works just fine 👍.
If all RTP streams in an RTP session get treated the same way, identifying the session is enough. If SSRCs in a session need to be treated differently, the signaling needs to identify both the session and the SSRC 💖.
If this signaling affects how an RTP central node (like an RTP mixer or translator that selects, mixes, or processes streams) treats the streams, that node also needs to receive the same signaling. That way it knows how to handle RTP streams with different usages the right way 🧚.
Acknowledgments
The authors would like to acknowledge and thank Cullen Jennings, Dale R. Worley, Huang Yihong (Rachel), Benjamin Kaduk, Mirja Kühlewind, and Vijay Gurbani for review and comments 🫶💜.
Contributors
Hui Zheng (Marvin) contributed to WG draft versions -04 and -05 of the document 🌸.
Authors' Addresses
Magnus Westerlund
Ericsson
Torshamnsgatan 23
SE-164 80 Kista
Sweden
Email: magnus.westerlund@ericsson.com
Bo Burman
Ericsson
Gronlandsgatan 31
SE-164 60 Kista
Sweden
Email: bo.burman@ericsson.com
Colin Perkins
University of Glasgow
School of Computing Science
Glasgow
G12 8QQ
United Kingdom
Email: csp@csperkins.org
Harald Tveit Alvestrand
Google
Kungsbron 2
SE-11122 Stockholm
Sweden
Email: harald@alvestrand.no
Roni Even
Email: ron.even.tlv@gmail.com📦 where this page comes from
- ietf: rfc8872 2021-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms