RFC 8860

Sending Multiple Types of Media in a Single RTP Session

📅 published January 2021, from ietf

Sending Multiple Types of Media in a Single RTP Session 🎀

Okay bestie, let's start with the paperwork 📋💅 This is RFC 8860 from the IETF, dated January 2021. It's a Standards Track doc, ISSN 2070-1721. The authors are M. Westerlund (Ericsson), C. Perkins (University of Glasgow), and J. Lennox (8x8 / Jitsi). It updates 3550 and 3551, which are [RFC3550] and [RFC3551] 🦋

Abstract

okay so this doc is all about letting one RTP session carry RTP streams with different media types, like audio, video, and text, all together 🎶📹💬 The RTP specs (RFCs 3550 and 3551) used to say no to this, so this doc updates RFCs 3550 and 3551 to allow it. But it's only for apps that meet the applicability rules for mixing media types in one RTP session ✨

Status of This Memo

This is an Internet Standards Track document 🌸

This doc is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check out Section 2 of RFC 7841 🧚

Info about the current status of this document, any errata, and how to give feedback on it may be found at https://www.rfc-editor.org/info/rfc8860 💜

Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please read these carefully, because they spell out your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License. 🫶

Table of Contents

   1.  Introduction
   2.  Terminology
   3.  Background and Motivation
   4.  Applicability
   5.  Using Multiple Media Types in a Single RTP Session
     5.1.  Allowing Multiple Media Types in an RTP Session
     5.2.  Demultiplexing Media Types within an RTP Session
     5.3.  Per-SSRC Media Type Restrictions
     5.4.  RTCP Considerations
   6.  Extension Considerations
     6.1.  RTP Retransmission Payload Format
     6.2.  RTP Payload Format for Generic FEC
     6.3.  RTP Payload Format for Redundant Audio
   7.  Signalling
   8.  Security Considerations
   9.  IANA Considerations
   10. References
     10.1.  Normative References
     10.2.  Informative References
   Acknowledgements
   Authors' Addresses

1. Introduction ✨

So RTP [RFC3550] was built to use separate RTP sessions for different types of media. That means different transport-layer flows carry different RTP streams. Picture a video call app sending its audio and video RTP flows on separate UDP ports 📞 Now, network address/port translation, firewalls, and other middleboxes are everywhere, and it's getting really hard to set up a bunch of transport-layer flows between endpoints 😭 So there's real pressure to cut down how many concurrent transport flows RTP apps use.

This memo updates [RFC3550] and [RFC3551] so that multiple media types can be sent in a single RTP session in certain cases, which means fewer transport-layer flows are needed 🦋 It changes nothing about RTP behaviour when you're using multiple RTP streams with the same media type (like multiple audio streams or multiple video streams) in one RTP session. [RFC8108] does give some important clarifications to RTP behaviour in that case though, so go read it 💅

Here's how this memo is laid out. Section 2 defines terminology. Section 3 gives the background and motivation, and Section 4 describes where this memo applies. Section 5 covers the issues that come up with the base RTP and RTP Control Protocol (RTCP) specifications [RFC3550] [RFC3551] when you use multiple types of media in a single RTP session, and Section 6 looks at the impact on RTP extensions. Signalling is in Section 7, and security considerations are in Section 8 🌈

2. Terminology 📖

The terms "encoded stream", "endpoint", "media source", "RTP session", and "RTP stream" are used as defined in [RFC7656]. We also define these terms:

  • Media Type: The general type of media data used by a real-time application. The media type matches the value used in the <media> field of a Session Description Protocol (SDP) m= line. The media types defined at the time of writing are "audio", "video", "text", "image", "application", and "message" [RFC4566] [RFC6466].
  • Quality of Service (QoS): Network mechanisms that are meant to make sure the packets within a flow, or with a specific marking, get transported with certain properties 🔥

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. So if you see them in caps, they're doing the official job, and if you don't, they're just regular words 🙈

3. Background and Motivation 🧚

RTP was designed for multimedia sessions that carry multiple types of media sent at the same time, using multiple transport-layer flows. The catch is that network address translators, firewalls, and other middleboxes make this complicated, because a mechanism is needed to make sure all the transport-layer flows an app needs can actually get established 😬 That causes three problems:

  1. increased delay to establish a complete session, since each of the transport-layer flows needs to be negotiated and established;
  2. increased state and resource consumption in the middleboxes, which can lead to unexpected behaviour when middlebox resource limits are reached; and
  3. increased risk that a subset of the transport-layer flows will fail to be established, thus preventing the application from communicating.

So fewer transport-layer flows means less risk of communication failure, and that can lead to better reliability and performance ✨

Using multiple transport-layer flows has one big perk: it makes it easy to use network-layer QoS mechanisms to give different flows different performance treatment 💖 But lots of RTP apps don't use network QoS at all, and they don't expect or want their media packets treated differently on the network, whether it's audio, video, or text. If an app doesn't want that, it doesn't need a transport flow structure that makes flow-based QoS easy 💅

Given all that, you might think RTP apps should just bundle all their RTP streams into one RTP session running over one transport-layer flow. But nope, the RTP specs [RFC3550] [RFC3551] prohibit it! 🙅‍♀️ The reason is that RTP's design makes certain assumptions that can clash with sending multiple media types in one RTP session. The RTCP timing rules, for example, assume that all RTP media flows in a single RTP session have broadly similar RTCP reporting and feedback requirements, which can be a problem when different types of media get multiplexed together. Various RTP extensions also assume things about Synchronisation Source (SSRC) use and RTCP reporting that don't work with different media types in one RTP session 😭

This memo updates [RFC3550] and [RFC3551] to let RTP sessions contain more than one media type in certain circumstances, and it gives guidance on when it's safe to send multiple media types in a single RTP session. Next up is Section 4, where we figure out exactly where this applies 👀🦋

4. Applicability 🎀

okay so this spec has limited applicability, bestie 💅 anyone who wants to use it has to check that their app and use case meet all of these criteria:

  • Equal treatment of media: a single RTP session usually gives every kind of media in it similar network treatment 🌸 apps that need really different network QoS or RTCP configuration for different RTP streams are better off sending those streams in separate RTP sessions, each with its own transport-layer flow, because that gives way more flexibility ✨ more guidance on giving some media streams differential treatment is in [RFC8872] and [RFC7657] 🫶
  • Compatible RTCP behaviour: the RTCP timing rules enforce one RTCP reporting interval for every participant in an RTP session 😭 flows with super different media sending rates or RTCP feedback needs can't be multiplexed together, because you'd end up with either too much or too little RTCP for some of them, depending on how the RTCP session bandwidth (and so the reporting interval) is configured 💀 like, it's likely impossible to find one RTCP config that suits both a low-rate audio flow with no feedback and a high-quality video flow with fancy RTCP-based feedback. so combining them into one RTP session is difficult and/or inadvisable 🙈
  • Signalled support: this memo's extensions aren't compatible with unmodified endpoints that follow [RFC3550] 🚫 using them needs signalling and mutual agreement from every participant in the RTP session. this can be a problem for signalling solutions that can't negotiate with all the participants 🥺 for declarative signalling solutions, requiring the session to use multiple media types in one RTP session can be a way to make sure everyone follows the rule. but signalling solutions that have no way to enforce a receiver supporting a specific feature can still run into issues 💔
  • Consistent support for multiparty RTP sessions: if you want to send multiple types of media in a multiparty RTP session, then every participant needs to support sending multiple types of media in a single RTP session 👑 in the general case it's not possible to build a gateway that connects an endpoint sending multiple media types over separate RTP sessions with one or more endpoints sending multiple media types in a single RTP session 🙅‍♀️

The reason is that the same SSRC value can safely be used for different streams in multiple RTP sessions, but once everything is squished into a single RTP session you get an SSRC collision 💥 that wouldn't be a big deal, since SSRC collision detection would sort out the conflict, except that some RTP payload formats and extensions use matching SSRCs to identify related flows and will break in a single RTP session 😭

A middlebox that remaps SSRC values when it combines multiple RTP sessions into one also needs to know about every RTCP packet type that might be used, so it can remap the SSRC values in those packets 🧚 this is impossible to do without restricting the RTCP packet types that can be used to only the ones the middlebox knows about. such a middlebox might also struggle with differences in configured RTCP bandwidth and other parameters between the RTP sessions 🍓

Finally, a middlebox that translates SSRC values can seriously hurt the chances of loop detection, since SSRC/CSRC (Contributing Source) can't be used to detect loops 🌀 instead, you need some other RTP stream or media source identity namespace that's common across all the interconnected parts 💜

  • Ability to operate with limited payload type space: an RTP session has only a single 7-bit payload type space for all of its payload type numbers 🎟️ some apps might find that space limiting (aka overly restrictive) when they use different media types and RTP payload formats within a single RTP session 😩
  • Avoidance of incompatible extensions: some RTP and RTCP extensions rely on there being multiple RTP sessions and relate RTP streams between those sessions 🧩 others report on particular media types and can't be used with other media types. apps that send multiple types of media into a single RTP session need to avoid these extensions 🚫

5. Using Multiple Media Types in a Single RTP Session 🦋

this section spells out what needs to be done, or avoided, to make an RTP session with multiple media types work without issues ✨

5.1. Allowing Multiple Media Types in an RTP Session 💖

Section 5.2 of "RTP: A Transport Protocol for Real-Time Applications" [RFC3550] says:

For example, in a teleconference composed of audio and video media encoded separately, each medium SHOULD be carried in a separate RTP session with its own destination transport address. Separate audio and video streams SHOULD NOT be carried in a single RTP session and demultiplexed based on the payload type or SSRC fields.

this spec changes both of these sentences 🔄 the first one becomes:

For example, in a teleconference composed of audio and video media encoded separately, each medium SHOULD be carried in a separate RTP session with its own destination transport address, unless the guidelines specified in [RFC8860] are followed and the application meets the applicability constraints.

and the second one becomes:

Separate audio and video media sources SHOULD NOT be carried in a single RTP session, unless the guidelines specified in [RFC8860] are followed.

the second paragraph of Section 6 of "RTP Profile for Audio and Video Conferences with Minimal Control" [RFC3551] says:

The payload types currently defined in this profile are assigned to exactly one of three categories or media types: audio only, video only and those combining audio and video. The media types are marked in Tables 4 and 5 as "A", "V" and "AV", respectively. Payload types of different media types SHALL NOT be interleaved or multiplexed within a single RTP session, but multiple RTP sessions MAY be used in parallel to send multiple media types. An RTP source MAY change payload types within the same media type during a session. See the section "Multiplexing RTP Sessions" of RFC 3550 for additional explanation.

this spec's whole purpose is to override that "SHALL NOT" under certain conditions 🙌 so this sentence has to change too, to allow payload types from multiple media types in the same session. the sentence with "SHALL NOT" in that paragraph becomes:

Payload types of different media types SHALL NOT be interleaved or multiplexed within a single RTP session unless [RFC8860] is used and the application conforms to the applicability constraints. Multiple RTP sessions MAY be used in parallel to send multiple media types.

5.2. Demultiplexing Media Types within an RTP Session 🔍

when packets come in from a transport-layer flow, an endpoint first separates the RTP and RTCP packets from the non-RTP packets and passes them to the RTP/RTCP protocol handler 📦 the RTP and RTCP packets then get demultiplexed into the different RTP streams based on their SSRC. for each RTP stream, incoming RTCP packets are processed, and the RTP payload type is used to pick the right media decoder 🎧 this whole process stays the same whether or not multiple media types are sent in a single RTP session 🫶

it's important to note, as explained below, that the RTP payload type is never used to tell RTP streams apart ‼️ RTP packets are demultiplexed into RTP streams by SSRC, and then the RTP payload type is used to select the correct media-decoding pathway for each RTP stream 💅

5.3. Per-SSRC Media Type Restrictions 🚦

an SSRC in an RTP session can switch between media formats of the same type, subject to certain restrictions [RFC7160], but it MUST NOT change its media type during its lifetime 🙅‍♀️ for example, an SSRC can switch between different audio formats, but it can't start sending audio and then switch to sending video. the lifetime of an SSRC ends when an RTCP BYE packet for that SSRC is sent, or when it stops transmitting for long enough that the other participants in the session time it out ⏳

the main motivation is that a given SSRC has its own RTP timestamp and sequence number spaces 🧠 just like you can't send two encoded audio streams with the same SSRC, you can't send one encoded audio stream and one encoded video stream with the same SSRC either. each encoded stream, once it's made into an RTP stream, needs sole control over its sequence number and timestamp space 🎀 if it didn't have that, you couldn't detect packet loss for that particular encoded stream, and you couldn't easily figure out which clock rate a particular SSRC's timestamp increases with ⏰ for more arguments on why multiplexing multiple media sources based on RTP payload type doesn't work, see [RFC8872] 📚

within an RTP session where multiple media types have been configured for use, an SSRC can only send one type of media during its lifetime. it can switch between different audio codecs, since those are both the same type of media, but it can't switch between audio and video 🔀 different SSRCs MUST be used for the different media sources, the same way multiple media sources of the same media type already have to do 🫧 the payload type tells a receiver which media type the SSRC is being used for. so the payload type MUST be unique across all of the payload configurations, no matter which media type is used in the RTP session 🌈

5.4. RTCP Considerations 📡

when sending multiple types of media that have different rates in a single RTP session, endpoints MUST follow the guidelines for handling RTCP provided in Section 7 of [RFC8108] 💜

6. Extension Considerations 🧩

this section lays out known issues and incompatibilities with RTP and RTCP extensions when multiple media types are used in a single RTP session 👀 future extensions to RTP and RTCP need to think about, and document, any potential incompatibilities 📝

6.1. RTP Retransmission Payload Format ✨

okay so the RTP retransmission payload format [RFC4588] has two modes it can run in: SSRC-multiplexed and session-multiplexed 💅

In SSRC-multiplexed mode, the retransmitted RTP packets live in the same RTP session as the originals, but they get a different SSRC. Both share the same RTCP Source Description (SDES) CNAME 🫶 If each endpoint only sends one original RTP stream and one retransmission stream in the session, this is enough. But if an endpoint sends multiple original and retransmission streams (which happens when you send multiple media types in one RTP session), then each original stream and its retransmission stream have to be matched up using heuristics 🧠 One trick: if a receiver only has retransmission requests outstanding for one SSRC that hasn't been mapped yet, it can figure out which original and retransmission streams belong together 🔍 Another option is using different RTP payload types. The signalled "apt" (associated payload type) parameter [RFC4588] of the retransmission payload format then links the retransmitted packets to the originals 🎀

Session-multiplexed mode sends the retransmission stream in a separate RTP session from the original, but uses the same SSRC for each one. The streams get associated by matching SSRCs across the two sessions 🦋 Multiple media types in a single RTP session don't mess this up at all, because each media type is sent with its own SSRC in the original session, and those same SSRCs get reused in the retransmission session, so everything still lines up 😭 You can signal this in SDP using the BUNDLE grouping extension [RFC8843] and the Flow Identification (FID) grouping extension [RFC5888]. The catch is that these SDP extensions say each "m=" line can only be in a single FID group, but the retransmission format uses FID groups to show which "m=" lines are an original and retransmission pair. So when you use BUNDLE to send multiple media types in one RTP session, every original media source ("m=" line) that gets retransmitted needs a matching "m=" line in the retransmission RTP session 🌸 And if there are multiple media lines for retransmission, they form their own independent BUNDLE group, separate from the source streams' group.

Here's an SDP fragment showing the grouping structure. It is not legal SDP, and only the most important attributes are left in, so don't copy-paste it and expect it to run 🙈 Also, this isn't an initial BUNDLE offer. You can see two bundle groups here: one for the source RTP session and one for the retransmissions. Then each media source gets grouped with its retransmission flow using FID, which adds three more groupings ✨

a=group:BUNDLE foo bar fiz
a=group:BUNDLE zoo kelp glo
a=group:FID foo zoo
a=group:FID bar kelp
a=group:FID fiz glo
m=audio 10000 RTP/AVP 0
a=mid:foo
a=rtpmap:0 PCMU/8000
m=video 10000 RTP/AVP 31
a=mid:bar
a=rtpmap:31 H261/90000
m=video 10000 RTP/AVP 31
a=mid:fiz
a=rtpmap:31 H261/90000
m=audio 40000 RTP/AVPF 99
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=0;rtx-time=3000
a=mid:zoo
m=video 40000 RTP/AVPF 100
a=rtpmap:100 rtx/90000
a=fmtp:199 apt=31;rtx-time=3000
a=mid:kelp
m=video 40000 RTP/AVPF 100
a=rtpmap:100 rtx/90000
a=fmtp:199 apt=31;rtx-time=3000
a=mid:glo

Figure 1: SDP Example of Session-Multiplexed RTP Retransmission

6.2. RTP Payload Format for Generic FEC 🔥

The RTP payload format for generic Forward Error Correction (FEC) is defined in [RFC5109] (and its predecessor, [RFC2733]). It can either send the FEC stream as its own separate RTP stream, or send the FEC combined with the original RTP stream as a redundant encoding [RFC2198] 💖

When you send FEC as a separate stream, the generic FEC payload format requires that stream to go in a separate RTP session from the original, using the same SSRC. The FEC stream gets associated by matching SSRCs between the sessions 🧚 The RTP session holding the originals can include multiple RTP streams, and those streams can use multiple media types. The repair session only needs one RTP payload type to mark FEC data, no matter how many FEC streams there are, because the SSRC is what ties each FEC stream to its original 👑 So it is RECOMMENDED that the FEC stream use the "application/ulpfec" media type when supporting [RFC5109], and the "application/parityfec" media type when supporting [RFC2733] 🎀 It is legal, but NOT RECOMMENDED, to send FEC streams using media-specific payload format names (like using both "audio/ulpfec" and "video/ulpfec" for one RTP session with both audio and video flows). Why? Because it (1) unnecessarily uses up RTP payload type values and (2) adds no value for demultiplexing, since there might be multiple streams of the same media type anyway 🙈

You can signal an original RTP session with multiple media types plus its generic FEC session using SDP with the BUNDLE extension [RFC8843]. In that case, the RTP session carrying the FEC streams is its own BUNDLE group. The "m=" line for each original stream and the "m=" line for its FEC stream get grouped with the SDP Grouping Framework, using either the FEC-FR grouping [RFC5956] or, for backwards compatibility, the FEC grouping [RFC4756]. It's pretty similar to what happens with RTP retransmission in session-based multiplexing, which we covered in Section 6.1 🦋

The source-specific media attributes specification [RFC5576] defines an SDP extension (the "FEC" semantic of the "ssrc-group" attribute) for signalling FEC relationships between multiple RTP streams inside a single RTP session. Generic FEC can't use this, though, because FEC repair packets need the same SSRC value as the source packets they protect 😭 There was a proposal (now abandoned) for an Uneven Level Protection (ULP) extension that would let FEC RTP streams live in the same RTP session as the source stream [FEC-Src-Multiplexing]. Sadly, it's dead 💀

When FEC is sent as a redundant encoding, the considerations in Section 6.3 apply ✨

6.3. RTP Payload Format for Redundant Audio 🎧

The RTP payload format for redundant audio [RFC2198] can be used to protect audio streams. It can also be used alongside the generic FEC payload format to send original and repair data in the same RTP packets. Both of these are compatible with RTP sessions that contain multiple media types 💜

This payload format requires each different redundant encoding to use a different RTP payload type number. When you use it with generic FEC in sessions with multiple media types, each media type needs its own payload type for the FEC stream. For example, if audio and text are sent in a single RTP session with generic ULP FEC sent as a redundant encoding for each, then payload types need to be assigned for FEC using the audio/ulpfec and text/ulpfec payload formats. If multiple original payload types are used in the session, a different redundant payload type needs to be allocated for each one. This has the potential to rapidly exhaust the available RTP payload type numbers 😱

7. Signalling 📣

Establishing a single RTP session that uses multiple media types requires signalling. This signalling has to:

  1. ensure that any participant in the RTP session is aware that this is an RTP session with multiple media types;
  2. ensure that the payload types in use in the RTP session are using unique values, with no overlap between the media types;
  3. ensure that RTP session-level parameters (for example, the RTCP RR and RS bandwidth modifiers [RFC3556], the RTP/AVPF trr-int parameter [RFC4585], transport protocol, RTCP extensions in use, and any security parameters) are consistent across the session; and
  4. ensure that RTP and RTCP functions that can be bound to a particular media type are reused where possible, rather than configuring multiple code points for the same thing 🧩

When using SDP signalling, the BUNDLE extension [RFC8843] is used to signal RTP sessions containing multiple media types 🌈

8. Security Considerations 🔐

RTP provides a range of strong security mechanisms that can be used to secure sessions [RFC7201] [RFC7202]. Most of these are independent of the type of media sent in the RTP session. However, it is important to check that the security mechanism you choose is compatible with all types of media sent within the session 🚨

Sending multiple media types in a single RTP session will generally require that all of them use the same security mechanism, whereas media sent using different RTP sessions can be secured in different ways. When different media types have different security requirements, it might be necessary to send them using separate RTP sessions to meet those different requirements. Be warned: this can have significant costs in terms of resource usage, session setup time, etc. 😭🔥

9. IANA Considerations 🌸

This document has no IANA actions. ✨

10. References 📚

10.1. Normative References

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC3550]  Schulzrinne, H., Casner, S., Frederick, R., and V.
           Jacobson, "RTP: A Transport Protocol for Real-Time
           Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
           July 2003, <https://www.rfc-editor.org/info/rfc3550>.

[RFC3551]  Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
           Video Conferences with Minimal Control", STD 65, RFC 3551,
           DOI 10.17487/RFC3551, July 2003,
           <https://www.rfc-editor.org/info/rfc3551>.

[RFC8108]  Lennox, J., Westerlund, M., Wu, Q., and C. Perkins,
           "Sending Multiple RTP Streams in a Single RTP Session",
           RFC 8108, DOI 10.17487/RFC8108, March 2017,
           <https://www.rfc-editor.org/info/rfc8108>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8843]  Holmberg, C., Alvestrand, H., and C. Jennings,
           "Negotiating Media Multiplexing Using the Session
           Description Protocol (SDP)", RFC 8843,
           DOI 10.17487/RFC8843, January 2021,
           <https://www.rfc-editor.org/info/rfc8843>.

10.2. Informative References 📚

Okay, these are the background reads 🤓💖 Here they are, all of them:

Acknowledgements 💜

The authors wanna give a huge shoutout to Christer Holmberg, Gunnar Hellström, Charles Eckel, Tolga Asveren, Warren Kumari, and Meral Shirazipour 🫶✨ Thank you for the feedback on this doc, you're literally the reason it slaps 💅

Authors' Addresses 🎀

Magnus Westerlund 🌸

Magnus Westerlund
Ericsson
Torshamnsgatan 23
SE-164 80 Stockholm
Sweden

Email: magnus.westerlund@ericsson.com

Colin Perkins 🦋

Colin Perkins
University of Glasgow
School of Computing Science
Glasgow
G12 8QQ
United Kingdom

Email: csp@csperkins.org

Jonathan Lennox 🧚

Jonathan Lennox
8x8, Inc. / Jitsi
Jersey City, NJ 07302
United States of America

Email: jonathan.lennox@8x8.com
📦 where this page comes from