RFC 7667
RTP Topologies
📼 RFC 7667: RTP Topologies 🎀
Okay bestie, this is RFC 7667, all about RTP Topologies 🦋. It was written by M. Westerlund (Ericsson) and S. Wenger (Vidyo) 💜, and it's an Informational doc, so it's sharing knowledge, not setting a Standards Track rule 🌸. It was published in November 2015, and its ISSN is 2070-1721 ✨. The IETF (Internet Engineering Task Force) put it out, and it obsoletes RFC 5117 👑. Ngl, that's the most important thing to know up front.
Abstract ✨
Okay so this doc talks about point-to-point calls and multi-endpoint setups in the world of the Real-time Transport Protocol (RTP) 💖. The big focus is on the centralized layouts that video conferencing companies use all the time, and it maps them onto RTP's terminology 🧚. It also says this doc gets updated with additional topologies, and that it replaces RFC 5117 🫶.
Status of This Memo 🌈
Lowkey the most boring part, but it matters. This document is not an Internet Standards Track spec. It's just here to share info 💅. It's a product of the IETF, it reflects what the IETF community agrees on, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🔥. Not everything the IESG approves can become an Internet Standard, though. Check Section 2 of RFC 5741 if you want the deets 🙈.
If you want the current status, any errata, or a way to send feedback, head to http://www.rfc-editor.org/info/rfc7667 🍓.
Copyright Notice 💜
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved ✨.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date 📜. Please actually read them, because they explain your rights and restrictions for this doc 🙈. Any Code Components pulled out of this doc have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come with no warranty, same as described in the Simplified BSD License 🎀.
Table of Contents 🦋
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1. Glossary . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2. Definitions Related to RTP Grouping Taxonomy . . . . . . 5
3. Topologies . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.1. Point to Point . . . . . . . . . . . . . . . . . . . . . 6
3.2. Point to Point via Middlebox . . . . . . . . . . . . . . 7
3.2.1. Translators . . . . . . . . . . . . . . . . . . . . . 7
3.2.2. Back-to-Back RTP sessions . . . . . . . . . . . . . . 11
3.3. Point to Multipoint Using Multicast . . . . . . . . . . . 12
3.3.1. Any-Source Multicast (ASM) . . . . . . . . . . . . . 12
3.3.2. Source-Specific Multicast (SSM) . . . . . . . . . . . 14
3.3.3. SSM with Local Unicast Resources . . . . . . . . . . 15
3.4. Point to Multipoint Using Mesh . . . . . . . . . . . . . 17
3.5. Point to Multipoint Using the RFC 3550 Translator . . . . 20
3.5.1. Relay - Transport Translator . . . . . . . . . . . . 20
3.5.2. Media Translator . . . . . . . . . . . . . . . . . . 21
3.6. Point to Multipoint Using the RFC 3550 Mixer Model . . . 22
3.6.1. Media-Mixing Mixer . . . . . . . . . . . . . . . . . 24
3.6.2. Media-Switching Mixer . . . . . . . . . . . . . . . . 27
3.7. Selective Forwarding Middlebox . . . . . . . . . . . . . 29
3.8. Point to Multipoint Using Video-Switching MCUs . . . . . 33
3.9. Point to Multipoint Using RTCP-Terminating MCU . . . . . 34
3.10. Split Component Terminal . . . . . . . . . . . . . . . . 35
3.11. Non-symmetric Mixer/Translators . . . . . . . . . . . . . 38
3.12. Combining Topologies . . . . . . . . . . . . . . . . . . 38
4. Topology Properties . . . . . . . . . . . . . . . . . . . . . 39
4.1. All-to-All Media Transmission . . . . . . . . . . . . . . 39
4.2. Transport or Media Interoperability . . . . . . . . . . . 40
4.3. Per-Domain Bitrate Adaptation . . . . . . . . . . . . . . 40
4.4. Aggregation of Media . . . . . . . . . . . . . . . . . . 41
4.5. View of All Session Participants . . . . . . . . . . . . 41
4.6. Loop Detection . . . . . . . . . . . . . . . . . . . . . 42
4.7. Consistency between Header Extensions and RTCP . . . . . 42
5. Comparison of Topologies . . . . . . . . . . . . . . . . . . 42
6. Security Considerations . . . . . . . . . . . . . . . . . . . 43
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 45
7.1. Normative References . . . . . . . . . . . . . . . . . . 45
7.2. Informative References . . . . . . . . . . . . . . . . . 45
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 48
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 481. Introduction 🧚
Okay so the Real-time Transport Protocol (RTP) [RFC3550] has topologies, which are basically ways of hooking up RTP entities to each other, plus how they process stuff for RTP and the RTP Control Protocol (RTCP) 💅. This doc is here to clear up the confusion people have had for ages, especially around terms that RTP doesn't define but that the communication industry uses constantly, like the Multipoint Control Unit (MCU) 🫶.
When the Audio-Visual Profile with Feedback (AVPF) [RFC4585] was being built, the main goal was efficient support for point-to-point calls and small multipoint calls that don't use centralized control 💖. In reality though, most multipoint conferences run on those centralized units called MCUs 🌸. MCUs can do mixer or translator stuff (in [RFC3550] terms) and signaling support, and they might also have extra application-layer features 🎀. This doc only focuses on the media transport side of an MCU that can be done with RTP, as covered below. It also looks at the properties of mixers and translators, and how some deployed MCUs don't quite match them 🔥.
This doc also writes down new multipoint designs that showed up recently and weren't planned for in RFC 5117, so this doc replaces [RFC5117] 💜. These designs use scalable video coding and simulcasting, and their centralized units are called Selective Forwarding Middleboxes (SFMs) ✨. Writing all this down gives everyone a shared starting point for future discussion and spec work 🦋.
The new topologies are: Point to Point via Middlebox (Section 3.2), Source-Specific Multicast (Section 3.3.2), SSM with Local Unicast Resources (Section 3.3.3), Point to Multipoint Using Mesh (Section 3.4), Selective Forwarding Middlebox (Section 3.7), and Split Component Terminal (Section 3.10) 🍓. The Point to Multipoint Using the RFC 3550 Mixer Model (Section 3.6) got a big glow-up and now covers two versions: Media-Mixing Mixer (Section 3.6.1) and Media-Switching Mixer (Section 3.6.2) 👑.
This doc tries to clarify and explain parts of the RTP spec [RFC3550], but it's informal. It's not meant to update or change what RFC 3550 says normatively 🙈.
2. Definitions 💅
2.1. Glossary 🌸
- ASM: Any-Source Multicast
- AVPF: The extended RTP profile for RTCP-based feedback
- CSRC: Contributing Source
- Link: The data transport to the next IP hop
- Middlebox: A device that is on the Path that media travel between two endpoints
- MCU: Multipoint Control Unit
- Path: The concatenation of multiple links, resulting in an end-to-end data transfer.
- PtM: Point to Multipoint
- PtP: Point to Point
- SFM: Selective Forwarding Middlebox
- SSM: Source-Specific Multicast
- SSRC: Synchronization Source
2.2. Definitions Related to RTP Grouping Taxonomy 🦋
These definitions were pulled from [RFC7656] ✨.
- Communication Session: A Communication Session is an association among two or more Participants communicating with each other via one or more Multimedia Sessions.
- Endpoint: A single addressable entity sending or receiving RTP packets. It may be decomposed into several functional blocks, but as long as it behaves as a single RTP stack mentity, it is classified as a single "endpoint". 🎀
- Media Source: A Media Source is the logical source of a time progressing digital media stream synchronized to a reference clock. This stream is called a Source Stream.
- Multimedia Session: A Multimedia Session is an association among a group of participants engaged in communication via one or more RTP sessions.
3. Topologies 🌈
This section defines a bunch of topologies that matter for codec control, but also for RTP usage in other contexts 💖. It starts with point-to-point cases, with or without middleboxes. Then it goes through different ways to set up point-to-multipoint communication. Those are organized around the most basic enabler, which is multicast, then a mesh of connections, then translators, mixers, and finally MCUs and SFMs 🫶. It wraps up by covering decomposited terminals, asymmetric middlebox behaviors, and combining topologies ✨.
The topologies can be referenced by other docs using a shortcut name with the prefix "Topo-" 🧚.
For each RTP-defined topology, we talk about how RTP, RTCP, and the carried media get handled. For RTCP, we also cover how RTCP feedback messages are handled, as defined in [RFC4585] and [RFC5104] 🔥.
3.1. Point to Point 🎀
Shortcut name: Topo-Point-to-Point
The Point-to-Point (PtP) topology (Figure 1) is two endpoints talking via unicast 💅. Both RTP and RTCP traffic go endpoint to endpoint using only unicast traffic. That's true even in exotic cases where this unicast traffic happens to ride over an IP multicast address 🙈.
+---+ +---+
| A |<------->| B |
+---+ +---+
Figure 1: Point to PointThe main vibe of this topology is that A sends only to B, and B sends only to A 💖. That skips all the drama of handling multiple endpoints and mixing together their requirements 👑. An endpoint can still use multiple RTP Synchronization Sources (SSRCs) in one RTP session, though 🌸. The number of RTP sessions between A and B can also be whatever, limited only by system stuff like the range of port numbers 🦋.
RTCP feedback messages for the indicated SSRCs go directly between the endpoints. So this topology has minimal (if any) issues with feedback messages 💜. For RTP sessions that use multiple SSRCs per endpoint, it can be relevant to implement cross-reporting suppression, as defined in "Sending Multiple Media Streams in a Single RTP Session" [MULTI-STREAM-OPT] 🍓.
3.2. Point to Point via Middlebox ✨
Here we look at cases where two endpoints communicate but one or more middleboxes are involved in the RTP session 🌷.
3.2.1. Translators 💖
Shortcut name: Topo-PtP-Translator
okay so there are two main kinds of translators, Transport Translators and Media Translators 🎀 and they share a vibe that sets them apart from mixers. For every RTP stream a translator receives, it makes a brand-new RTP stream on the other side ✨ A translator keeps the SSRC of that stream the same across the translation, and that's the big difference from a mixer. A mixer can pick one stream out of many (like switching between audio and video) or blend several together (like audio mixing or video tiling). It always sends under its own SSRC, and it might use the CSRC field to say where the content came from 💅 Mixers are more of a point-to-multipoint thing than a PtP thing 🌸 In PtP, the middlebox is mostly there to make two endpoints that otherwise can't talk to each other actually interoperate, like transcoding into a codec the receiver supports. That's a job for a Media Translator 🧚
As Section 7.1 of [RFC3550] explains, the SSRC space is shared by all participants in the RTP session, no matter which side of the translator they're on 🫶 So the endpoints (the RTP session participants) are the ones in charge of running SSRC collision detection, and the translator can't change the SSRC, full stop 🙅♀️ Any SDES information tied to an SSRC or CSRC also has to be forwarded between the domains for every SSRC/CSRC used in those domains. Don't leave the info behind, bestie 💜
A translator usually doesn't have an SSRC of its own, so it isn't visible as an active participant in the session 🙈 There are two good reasons it might get one. The first is when the translator acts as a quality monitor that sends RTCP reports, which means it's required to have an SSRC. The second is when it wants to use RTCP feedback messages. Picture a translator watching for packet loss on important video packets, and it wants the media sender to fix things by sending feedback 😭 It could reuse the SSRC of the receiving endpoint it's standing in for, but then it would have to rewrite that endpoint's RTCP reports to keep everything consistent, which is a whole mess. It's usually simpler to just expose an extra SSRC in the session ✨ The one catch is that endpoints that don't fully support the RTP specification might struggle with multiple SSRCs reporting on the streams that endpoint sends. Implementers might see this as a pretty exotic use case, so heads up 👀
In general, a translator implementation should think about which RTCP feedback messages or codec-control messages it needs to understand, based on what the translator itself does 🦋 This fits perfectly with the requirement to also translate RTCP messages between the domains, so it's all one big happy family 🌈
3.2.1.1. Transport Relay/Anchoring
Shortcut name: Topo-PtP-Relay
There are lots of middleboxes that can sit between two endpoints at the transport level, for example to change the IP/UDP headers, which makes them basic Transport Translators 🍓 These come in tons of flavours. Some do NAT [RFC3022] traversal by pinning the media path to a public-address relay. Some are in networks where the RTP stream has to pass a specific checkpoint for auditing, which is done by relaying. Some are there for privacy, hiding each peer's transport addresses from the other party 🤫 Other protocols and functions that do this too include Traversal Using Relays around NAT (TURN) [RFC5766] servers, Session Border Gateways, and Media Processing Nodes with media anchoring abilities 💖
+---+ +---+ +---+
| A |<------>| T |<------->| B |
+---+ +---+ +---+
Figure 2: Point to Point with TranslatorHere's the thing: these functions are normally transparent at the RTP level. They don't change any RTP or RTCP packet fields, and they only affect the lower layers ✨ They can still affect the path, though, since RTP and RTCP packets get routed between the endpoints in the RTP session, so they indirectly affect the session too 🦋 You might think Transport Translator-type middleboxes don't need to be in this document, but this topology can actually raise extra requirements for the RTP implementation and how it works with the signaling solution 👀 Both in signaling and in certain RTCP fields, network addresses other than the relay's can show up, because B has a different network address than the relay (T). Implementations that can't handle that won't work properly when endpoints are behind NAT 😭
Transport Relay implementations also have to think about security considerations, and this is serious, bestie 🚨 Source address filtering on incoming packets is usually a big deal for relays. Without it, attackers could inject traffic into a session, and one peer might think that traffic came from the other peer 💀
3.2.1.2. Transport Translator
Shortcut name: Topo-Trn-Translator
Transport Translators (Topo-Trn-Translator) don't modify the RTP stream itself. They care about transport parameters 🎀 For this section, transport parameters are the transport addresses (which can bridge different domains, like unicast to multicast) and the media packetization (which lets other transport protocols connect to a session, as in gateways) 🧚
Translators that bridge different protocol worlds have to think about how the SSRC/CSRC (Contributing Source) concept maps onto the non-RTP protocol 🫶 When you design a translator for a non-RTP-based media transport, a big thing to figure out is how to handle different sources and their identities. This memo won't go into that problem space any further 🙈
Of the Transport Translators, this memo is mainly interested in the ones that use RTP on both sides, and that's the assumption from here on out 💅
The most basic Transport Translators, the ones that work below the RTP level, were already covered in Section 3.2.1.1 ✨
3.2.1.3. Media Translator
Shortcut name: Topo-Media-Translator
Media Translators (Topo-Media-Translator) modify the media inside the RTP stream 🍓 This is commonly called transcoding. The change can be as small as removing parts of the stream, or it can go all the way to a full decode and re-encode (down to the sample level or equivalent) using a different media codec 💖 Media Translators are commonly used to connect endpoints that don't share an interoperability point in media encoding 🌸
Stand-alone Media Translators are rare 😭 Most of the time you'll see a combo of a Transport and a Media Translator, which translates both the media and the transport aspects of the RTP stream carrying that media between two transport domains 🦋
When media translation happens, the translator's job for RTCP traffic gets a lot more complicated 💀 In this case, the translator has to rewrite endpoint B's RTCP receiver reports before forwarding them to endpoint A. That rewriting is needed because the RTP stream B receives isn't the same as the one the other participants get 🙃 For example, B might get fewer packets than A sends, because of the different media format and data rate. So if the receiver reports were forwarded without changes, the extended highest sequence number would make it look like B was way behind in reception, when it most likely wasn't 😬 That's why the translator must translate that number into the matching sequence number for the stream the translator received. Most of the other fields in the RTCP receiver reports need similar treatment 🧚
A Media Translator may sometimes act on behalf of the "real" source (the endpoint that originally sent the media to the translator) and respond to RTCP feedback messages ✨ This can happen, for example, when a receiving endpoint asks for a bandwidth reduction, and the Media Translator hasn't detected any congestion or other reason for a bandwidth reduction between the sending endpoint and itself. In that case, it makes sense for the Media Translator to react to codec control messages itself, for example by transcoding to a lower media rate 💜
Now for a variant worth pointing out, the one shown in Figure 3, where endpoint A sends an RTP stream containing only media to B 🎀 There's a device T on the path that manipulates the RTP streams on A's behalf. One common example is that T adds a second RTP stream carrying Forward Error Correction (FEC) information to protect A's RTP stream, which doesn't have FEC of its own 🛡️ In that case, T needs to semantically bind the new FEC RTP stream to A's media-carrying RTP stream, for example by using the same CNAME as A 🔗
+------+ +------+ +------+
| | | | | |
| A |------->| T |-------->| B |
| | | |---FEC-->| |
+------+ +------+ +------+
Figure 3: Media Translator Adding FECSometimes information gets added into the original RTP stream while most or all of the original RTP packets stay intact (apart from certain RTP header fields, like the sequence number) ✨ One example is injecting metadata into the RTP stream, carried in its own RTP packets 🧚
Similarly, a Media Translator can sometimes remove information from the RTP stream and leave the remaining RTP packets unchanged, again apart from certain RTP header fields 🌈
Either kind of thing, where T manipulates the RTP stream or adds an accompanying RTP stream on behalf of A, is also covered by the Media Translator definition 👑
3.2.2. Back-to-Back RTP sessions
Shortcut name: Topo-Back-To-Back
Okay so this one is giving "middlebox in the middle, but make it two separate sessions" 🦋 There are middleboxes that connect two endpoints (A and B) through themselves (MB), but they don't do it by being part of one common RTP session. Instead they set up two different RTP sessions: one between A and the middlebox, and another between the middlebox and B. This topology is called Topo-Back-To-Back. 💅
|<--Session A-->| |<--Session B-->|
+------+ +------+ +------+
| A |------->| MB |-------->| B |
+------+ +------+ +------+
Figure 4: Back-to-Back RTP Sessions through MiddleboxThe middlebox acts as an application-level gateway and bridges the two RTP sessions. ✨ That bridging can be super basic, like just forwarding the RTP payloads between the sessions, or it can get more complex and include media transcoding. The big difference from the single RTP session setup is how SSRCs and the other session-related identifiers (like CNAMEs) get handled. With two separate sessions these can be freely changed, and it becomes the middlebox's responsibility to keep the correct relations. 🧚
The signaling, or other functionality above the RTP level that references RTP streams, may be what's most impacted by using two RTP sessions and changing identifiers. 🙈 Also, the two-session structure puts a congestion control requirement on the middlebox, because it becomes fully responsible for the media stream it sources into each of the sessions. 👑
Congestion control can be handled locally on each of the two segments, or by bridging statistics from the receiving endpoint through the middlebox back to the sending endpoint. 🌸 From an implementation point of view, though, the latter means dealing with a bunch of inconsistencies. First, packet loss must be detected for an RTP stream sent from A to the middlebox, and that loss must be reported through a skipped sequence number in the RTP stream from the middlebox to B. This coupling and the resulting inconsistencies are conceptually easier to handle when you treat the two RTP streams as belonging to a single RTP session. 🫶
3.3. Point to Multipoint Using Multicast
Multicast is an IP-layer functionality that's available in some networks. There are two main flavors: Any-Source Multicast (ASM) [RFC1112], where any multicast group participant can send to the group address and expect the packet to reach all group participants, and Source-Specific Multicast (SSM) [RFC3569], where only a particular IP host sends to the multicast group. 🎀 Each of these models is discussed below in its own section.
3.3.1. Any-Source Multicast (ASM)
Shortcut name: Topo-ASM (was Topo-Multicast)
+-----+
+---+ / \ +---+
| A |----/ \---| B |
+---+ / Multi- \ +---+
+ cast +
+---+ \ Network / +---+
| C |----\ /---| D |
+---+ \ / +---+
+-----+
Figure 5: Point to Multipoint Using MulticastPoint to Multipoint (PtM) is defined here as using a multicast topology as a transmission model, where traffic from any multicast group participant reaches all the other multicast group participants, except in cases such as: 🦋
- packet loss, or
- when a multicast group participant does not want to receive the traffic for a specific multicast group and therefore has not subscribed to that IP multicast group. This can happen, for example, when a Multimedia Session is distributed using two or more multicast groups and a participant is only subscribed to a subset of those sessions.
In this context, "traffic" covers both RTP and RTCP traffic. The number of multicast group participants can range from one to many, since RTP and RTCP scale to very large multicast groups (the theoretical limit on the number of participants in a single RTP session is in the range of billions). 🌈 The above can be realized using ASM.
For feedback usage, it's useful to define a "small multicast group" as a group where the number of participants is so low (and other factors, such as connectivity, are so good) that it allows participants to use early or immediate feedback, as defined in AVPF [RFC4585]. 💖 Even when the environment would allow a small multicast group, some applications may still want to use the more limited RTCP feedback options available to large multicast groups. For example, that's the case when the threshold of the small multicast group (in terms of participants) may be exceeded during the lifetime of a session. 🙈
RTCP feedback messages in multicast reach every subscriber, just like media data (subject to packet losses and multicast group subscription). Therefore, the feedback suppression mechanism discussed in [RFC4585] is typically required. Each individual endpoint that is a multicast group participant needs to process every feedback message it receives, not only to figure out whether it's affected or whether the message applies only to some other endpoint, but also to derive the timing restrictions for sending its own feedback messages, if any. 💅
3.3.2. Source-Specific Multicast (SSM)
Shortcut name: Topo-SSM
In Any-Source Multicast, any of the multicast group participants can send to all the other participants by sending a packet to the group. 🫶 In contrast, Source-Specific Multicast [RFC3569][RFC4607] covers scenarios where only a single source (Distribution Source) can send to the multicast group, creating a topology that looks like this:
+--------+ +-----+
|Media | | | Source-Specific
|Sender 1|<----->| D S | Multicast
+--------+ | I O | +--+----------------> R(1)
| S U | | | |
+--------+ | T R | | +-----------> R(2) |
|Media |<----->| R C |->+ | : | |
|Sender 2| | I E | | +------> R(n-1) | |
+--------+ | B | | | | | |
: | U | +--+--> R(n) | | |
: | T +-| | | | |
: | I | |<---------+ | | |
+--------+ | O |F|<---------------+ | |
|Media | | N |T|<--------------------+ |
|Sender M|<----->| | |<-------------------------+
+--------+ +-----+ RTCP Unicast
FT = Feedback Target
Transport from the Feedback Target to the Distribution
Source is via unicast or multicast RTCP if they are not
co-located.
Figure 6: Point to Multipoint Using Source-Specific MulticastIn the SSM topology (Figure 6), a number of RTP sending endpoints (RTP sources from here on) (1 to M) are allowed to send media to the SSM group. ✨ These sources send media to a dedicated Distribution Source, which forwards the RTP streams to the multicast group on behalf of the original RTP sources. The RTP streams reach the receiving endpoints (receivers from here on) (R(1) to R(n)). The receivers' RTCP messages can't be sent to the multicast group, because the SSM multicast group by definition has only a single IP sender. 😭 To support RTCP, an RTP extension for SSM [RFC5760] was defined. It uses unicast transmission to send RTCP from each receiver to one or more Feedback Targets (FT). The Feedback Targets either relay the RTCP unmodified, or provide a summary of the participants' RTCP reports to the whole group by forwarding the RTCP traffic to the Distribution Source. 🧚 Figure 6 only shows a single Feedback Target integrated in the Distribution Source, but for scalability the FT can be distributed, with each instance responsible for a subgroup of the receivers. For summary reports, however, there typically must be a single Feedback Target aggregating all the summaries into a common message to the whole receiver group. 🌸
The RTP extension for SSM specifies how feedback (both reception information and specific feedback events) is handled. The more general problems associated with multicast, where everyone receives what the Distribution Source sends, still need to be accounted for. 👑
The situation above results in common behavior for RTP multicast:
- Multicast applications often use a group of RTP sessions, not just one. Each endpoint needs to be a member of most or all of these RTP sessions in order to perform well.
- Within each RTP session, the number of media sinks is likely to be much larger than the number of RTP sources.
- Multicast applications need signaling functions to identify the relationships between RTP sessions.
- Multicast applications need signaling functions to identify the relationships between SSRCs in different RTP sessions.
All multicast configurations share a signaling requirement: all of the endpoints need the same RTP and payload type configuration. Otherwise, endpoint A could, for example, be using payload type 97 to identify the video codec H.264, while endpoint B identifies it as MPEG-2, with unpredictable but almost certainly not visually pleasing results. 😭🙈
Security solutions for this type of group communication are also challenging. First, the key management and the security protocol must support group communication. Source authentication becomes more difficult and requires specialized solutions. For more discussion on this, please review "Options for Securing RTP Sessions" [RFC7201]. 🔥
3.3.3. SSM with Local Unicast Resources ✨
Shortcut name: Topo-SSM-RAMS
Okay so this is where SSM gets iconic 💅 "Unicast-Based Rapid Acquisition of Multicast RTP Sessions" [RFC6285] adds extra pieces to the SSM topology, and it's giving main character energy 🌟
----------- --------------
| |------------------------------------>| |
| |.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.->| |
| | | |
| Multicast | ---------------- | |
| Source | | Retransmission | | |
| |-------->| Server (RS) | | |
| |.-.-.-.->| | | |
| | | ------------ | | |
----------- | | Feedback | |<.=.=.=.=.| |
| | Target (FT)| |<~~~~~~~~~| RTP Receiver |
PRIMARY MULTICAST | ------------ | | (RTP_Rx) |
RTP SESSION with | | | |
UNICAST FEEDBACK | | | |
| | | |
- - - - - - - - - - - |- - - - - - - - |- - - - - |- - - - - - - |- -
| | | |
UNICAST BURST | ------------ | | |
(or RETRANSMISSION) | | Burst/ | |<~~~~~~~~>| |
RTP SESSION | | Retrans. | |.........>| |
| |Source (BRS)| |<.=.=.=.=>| |
| ------------ | | |
| | | |
---------------- --------------
-------> Multicast RTP Stream
.-.-.-.> Multicast RTCP Stream
.=.=.=.> Unicast RTCP Reports
~~~~~~~> Unicast RTCP Feedback Messages
.......> Unicast RTP Stream
Figure 7: SSM with Local Unicast Resources (RAMS)The rapid acquisition extension lets an endpoint that joins an SSM multicast session ask for media to start at the last sync point (the spot where media can be decoded without needing context from earlier packets) and be sent at high speed. This keeps going until, after the decoder has handled those burst-delivered packets, the correct media timing is established. In other words, media packets arrive within adequate buffer intervals for the application 🎀
Here's the flow. First, a unicast PtP RTP session is set up between the Burst/Retransmission Source (BRS) (Figure 7) and the RTP Receiver. That unicast session sends cached packets from the multicast group at higher than normal speed, so the receiver can catch up to the ongoing multicast RTP stream. Once the RTP receiver and its decoder have caught up with the multicast session's current delivery, the receiver switches over to receiving directly from the multicast group 🦋
In lots of deployed applications, the (still existing) PtP RTP session sticks around as a repair channel, i.e., for RTP Retransmission traffic of those packets that were not received from the multicast group 🩷
3.4. Point to Multipoint Using Mesh 🕸
Shortcut name: Topo-Mesh
+---+ +---+
| A |<---->| B |
+---+ +---+
^ ^
\ /
\ /
v v
+---+
| C |
+---+
Figure 8: Point to Multipoint Using MeshBased on the RTP session definition, it's clearly possible to have a joint RTP session with three or more endpoints over multiple unicast transport flows, like the joint three-endpoint session pictured above 🫶 In this case, A needs to send its RTP streams and RTCP packets to both B and C over their respective transport flows. As long as all endpoints do the same, everyone gets a joint view of the RTP session ✨
This topology doesn't create any extra requirements beyond needing multiple transport flows associated with a single RTP session. An endpoint can use a single local port to receive all these transport flows (in which case the sending port, IP address, or SSRC can be used to demultiplex), or it might have separate local reception ports for each of the endpoints 💜
+-A--------------------+
|+---+ |
||CAM| | +-B-----------+
|+---+ +-UDP1------| |-UDP1------+ |
| | | +-RTP1----| |-RTP1----+ | |
| V | | +-Video-| |-Video-+ | | |
|+----+ | | | |<----------------|BV1 | | | |
||ENC |----+-+-+--->AV1|---------------->| | | | |
|+----+ | | +-------| |-------+ | | |
| | | +---------| |---------+ | |
| | +-----------| |-----------+ |
| | | +-------------+
| | |
| | | +-C-----------+
| | +-UDP2------| |-UDP2------+ |
| | | +-RTP1----| |-RTP1----+ | |
| | | | +-Video-| |-Video-+ | | |
| +-------+-+-+--->AV1|---------------->| | | | |
| | | | |<----------------|CV1 | | | |
| | | +-------| |-------+ | | |
| | +---------| |---------+ | |
| +-----------| |-----------+ |
+----------------------+ +-------------+
Figure 9: A Multi-Unicast Mesh with a Joint RTP SessionFigure 9 shows endpoint A's view of using a common RTP session when setting up the mesh from Figure 8. There's only one RTP session (RTP1) but two transport flows (UDP1 and UDP2). The Media Source (CAM) is encoded and transmitted over the SSRC (AV1) across both transport layers. However, since this is a joint RTP session, the two streams must be the same. So a congestion control adaptation needed for the paths A to B and A to C needs to use the most restricting path's properties 🔥
An alternative setup is to use independent RTP sessions between each pair of peers, i.e., three different RTP sessions. In some scenarios the same RTP stream may be sent from the transmitting endpoint; however, this also supports local adaptation taking place in one or more of the RTP streams, making them non-identical 🌈
+-A----------------------+ +-B-----------+
|+---+ | | |
||MIC| +-UDP1------| |-UDP1------+ |
|+---+ | +-RTP1----| |-RTP1----+ | |
| | +----+ | | +-Audio-| |-Audio-+ | | |
| +->|ENC1|--+-+-+--->AA1|------------->| | | | |
| | +----+ | | | |<-------------|BA1 | | | |
| | | | +-------| |-------+ | | |
| | | +---------| |---------+ | |
| | +-----------| |-----------+ |
| | ------------| |-------------|
| | | |-------------+
| | |
| | | +-C-----------+
| | | | |
| | +-UDP2------| |-UDP2------+ |
| | | +-RTP2----| |-RTP2----+ | |
| | +----+ | | +-Audio-| |-Audio-+ | | |
| +->|ENC2|--+-+-+--->AA2|------------->| | | | |
| +----+ | | | |<-------------|CA1 | | | |
| | | +-------| |-------+ | | |
| | +---------| |---------+ | |
| +-----------| |-----------+ |
+------------------------+ +-------------+
Figure 10: A Multi-Unicast Mesh with an Independent RTP SessionLet's look at the independent RTP session setup from A's point of view in Figure 10, covering both how the media is handled and how the RTP sessions are set up. A's microphone gets captured and the audio is fed into two different encoder instances, each with its own independent RTP session, i.e., RTP1 and RTP2 respectively. The SSRCs (AA1 and AA2) in each RTP session are completely independent, and the media bitrate from each encoder can be tuned differently to handle any congestion control needs that differ between the paths A to B and A to C 🎧
From a topology standpoint, there's an important difference in how RTCP behaves. When a single RTP session spans all three endpoints A, B, and C, and their connecting RTP streams, one common RTCP bandwidth is calculated and used for that single joint session. With multiple independent RTP sessions, though, each RTP session gets its own local RTCP bandwidth allocation 💅
Also, when multiple sessions are used, endpoints not directly involved in a session have no awareness of the conditions in that session. For example, in the three-endpoint configuration in Figure 8, endpoint A has no awareness of the conditions in the session between endpoints B and C (whereas with a single RTP session, it would have that awareness) 👀
Loop detection gets affected too 🙈 With independent RTP sessions, the SSRC/CSRC can't be used to determine when an endpoint receives its own media stream, or a mixed media stream that includes its own media stream (a condition known as a loop). Identifying loops and, in most cases, avoiding them has to be done by other means, for example through signaling or by using an RTP external namespace binding SSRC/CSRC among any communicating RTP sessions in the mesh.
3.5. Point to Multipoint Using the RFC 3550 Translator 🧚
This section covers some extra usages related to point to multipoint with translators, compared to the point-to-point cases in Section 3.2.1 🦋
3.5.1. Relay - Transport Translator ✨
Shortcut name: Topo-PtM-Trn-Translator
okay bestie, this part is all about Transport Translator-only setups that make multipoint sessions happen 🦋 no extra boxes needed, just the translator doing its thing 💅
This section covers those Transport Translator-only usages that enable multipoint sessions 🌸
+-----+
+---+ / \ +------------+ +---+
| A |<---/ \ | |<---->| B |
+---+ / \ | | +---+
+ Multicast +->| Translator |
+---+ \ Network / | | +---+
| C |<---\ / | |<---->| D |
+---+ \ / +------------+ +---+
+-----+
Figure 11: Point to Multipoint Using MulticastFigure 11 shows a Transport Translator doing at least IP address translation 🔁 It lets endpoints B and D, which can't do multicast 😭, join an Any-Source Multicast session with A and C. The translator forwards their unicast traffic to the multicast addresses in use, and the other way around too. It must also forward B's traffic to D, and vice versa, so that both B and D get a complete view of the session 👑
+---+ +------------+ +---+
| A |<---->| |<---->| B |
+---+ | | +---+
| Translator |
+---+ | | +---+
| C |<---->| |<---->| D |
+---+ +------------+ +---+
Figure 12: RTP Translator (Relay) with Only Unicast PathsAnother translator scenario is shown in Figure 12 🦋 Here the translator connects multiple endpoints through unicast. This can be done with a very simple Transport Translator, which this document calls a relay. The relay forwards all traffic it receives, both RTP and RTCP, to all other endpoints. So a multicast network gets emulated without relying on multicast-capable network infrastructure, which is honestly iconic 🧚
For RTCP feedback, the considerations are similar to those in the ASM RTP topology ✨ It also adds some extra signaling requirements to session establishment. For example, a common configuration of RTP payload types is required 📝
Transport Translators and relays should always consider implementing source address filtering. This keeps attackers from using the listening ports on the translator to inject traffic 🚨 The translator can also go one step further, especially if explicit SSRC signaling is used. It can stop endpoints from sending SSRCs other than their own, like the ones other participants in the session are using. This can improve the security properties of the session, even though group keys let anyone impersonate another member of the same RTP session on a cryptographic level. Be careful with this one, bestie 💀
A translator that doesn't change the RTP/RTCP packet content can be operated without needing access to the security contexts used to protect the RTP/RTCP traffic between the participants 🔐
3.5.2. Media Translator 🎀
In the context of multipoint communications, a Media Translator isn't providing new mechanisms to set up a multipoint session 🙈 It's more of an enabler, or facilitator, making sure a given endpoint, or a defined subset of endpoints, can take part in the session 🫶
If endpoint B in Figure 11 were behind a limited network path, the translator may perform media transcoding. That lets the traffic received from the other endpoints reach B without overloading the path 🌈 This transcoding can help the other endpoints in the multicast part of the session too, since A's quality doesn't have to be lowered to the bitrates B can actually receive (and vice versa) ✨
3.6. Point to Multipoint Using the RFC 3550 Mixer Model 💜
Shortcut name: Topo-Mixer
okay so a mixer is a middlebox that aggregates multiple RTP streams that are part of a session. It does this by generating one or more new RTP streams and, in most cases, by manipulating the media data 🎛️ A really common use for a mixer is letting a participant receive a session while using fewer resources 🍓
+-----+
+---+ / \ +-----------+ +---+
| A |<---/ \ | |<---->| B |
+---+ / Multi- \ | | +---+
+ cast +->| Mixer |
+---+ \ Network / | | +---+
| C |<---\ / | |<---->| D |
+---+ \ / +-----------+ +---+
+-----+
Figure 13: Point to Multipoint Using the RFC 3550 Mixer ModelThink of a mixer as a device that terminates the RTP streams received from other endpoints in the same RTP session 💖 It uses the media data carried in those received streams to generate derived RTP streams, which get sent to the receiving endpoints 🦋
The content the mixer provides is the mixed aggregate of what it receives over the PtP or PtM paths, which are part of the same Communication Session ✨
The mixer creates the Media Source and the source RTP stream just like an endpoint does. It mixes the content (often in the uncompressed domain), then encodes and packetizes it for transmission to a receiving endpoint 🎶 The CSRC Count (CC) and CSRC fields in the RTP header can be used to indicate the contributors to the newly generated RTP stream. The SSRCs of the to-be-mixed streams on the mixer input appear as the CSRCs at the mixer output. That output stream uses a unique SSRC that identifies the mixer's stream. The CSRC should be forwarded between the different endpoints so loop detection and identification of sources that are part of the Communication Session can work 🔍 Note that Section 7.1 of RFC 3550 requires the SSRC space to be shared between domains for these reasons. This also means any SDES information normally needs to be forwarded across the mixer 📇
The mixer is responsible for generating RTCP packets in line with its role 📊 It's an RTP receiver, so it should send RTCP receiver reports for the RTP streams it receives and terminates. In its role as an RTP sender, it should also generate RTCP sender reports for the RTP streams it sends. As Section 7.3 of RFC 3550 specifies, a mixer must not forward RTCP unaltered between the two domains 🙅♀️
The mixer in Figure 13 is involved in three domains that need to be separated: the Any-Source Multicast network (including endpoints A and C), endpoint B, and endpoint D 🧩 Assuming all four endpoints want content from all the others, the mixer makes different mixed RTP streams for B and D, since B's might contain content received from D, and vice versa. However, the mixer may only need one SSRC per media type in each domain where it is the receiving entity and transmitter of mixed content 🌸
In the multicast domain, a mixer still needs to provide a mixed view of the other domains. This makes the mixer simpler to implement and avoids issues with advanced RTCP handling or loop detection, which would be a problem if the mixer were providing non-symmetric behavior. Please see Section 3.11 for more discussion on this topic 📚 The mixing operation, however, in each domain could potentially be different 🌈
A mixer is responsible for receiving RTCP feedback messages and handling them appropriately 🎀 What counts as "appropriate" depends on the message itself and the context. In some cases, a mixer receiving a codec-control message may result in it generating and transmitting RTCP feedback messages to the endpoints in the other domain(s). In other cases, a message is handled by the mixer locally and not forwarded to any other domain 💅
When the multicast network in Figure 13 (to the left of the mixer) is replaced with individual unicast paths, as shown in Figure 14, the mixer model is very similar to the one discussed in Section 3.9 below. Please see the discussion in Section 3.9 about the differences between these two models 🫶
+---+ +------------+ +---+
| A |<---->| |<---->| B |
+---+ | | +---+
| Mixer |
+---+ | | +---+
| C |<---->| |<---->| D |
+---+ +------------+ +---+
Figure 14: RTP Mixer with Only Unicast PathsWe're about to get into the different mixing operations a mixer can perform and how they affect RTP and RTCP behavior 🧚✨
3.6.1. Media-Mixing Mixer 🎀
okay so the Media-Mixing Mixer is probably the one everyone pictures when they hear the word "mixer" 🎧✨ Its basic move is that it receives RTP streams from several endpoints and picks which stream(s) go into a media-domain mix 💖 That pick can be static (set up in config) or dynamic and content-dependent, like voice activation 🗣️ Then the mixer makes one outgoing RTP stream out of that mix. iconic, honestly 👑
The most commonly deployed version is probably the audio mixer, the one used in voice conferencing, where the output is a mixture of all the input audio signals 🎙️ It needs minimal signaling to get set up, which is so chill 😌 From a signal processing point of view, audio mixing is relatively straightforward and commonly doable for a reasonable number of endpoints 🌸 Let's say you want to mix N streams from N different endpoints. The mixer has to decode those N streams, usually into the sample domain, and then produce N or N+1 mixes 🧠 Why multiple mixes? Because each endpoint should get a mix of every other source except its own, otherwise they'd hear themselves as an echo and that's a whole mood 🙈 If N is lower than the total number of endpoints, the mixer can make one mix of all N streams for the group that isn't currently in the mix, which is how you get N+1 mixes 🦋 After that, the audio gets encoded again, RTP packetized, and sent out 🚀 And in lots of cases, audio level normalization, noise suppression, and similar signal processing steps are also required or desirable before the actual mixing starts 🧼💅
Now video. In video, "mixing" means something different 🎬 It usually refers to spatially combining the contributed video streams, which is also called "tiling" 🧩 The reconstructed, appropriately scaled-down videos get arranged in a set of tiles, and each tile holds the video from one endpoint (usually a human participant, bestie) 🥰 Tiles can be different sizes, so a really important participant or the loudest speaker can get a bigger tile than everyone else 👀 A self-view can be included too, either produced locally or fed back as the mixer-received, reconstructed video image. That remote loopback is giving confidence monitoring, i.e., you get to see yourself at the same quality the others see you 🪞✨ Tiling normally works on reconstructed video in the sample domain. The tiled image then gets encoded, packetized, and sent by the mixer to the receiving endpoints 📦 It's possible for a middlebox with media mixing duties to have just one mixer of this type, in which case everyone sees the exact same tiled video, even if it's sent over different RTP streams 🤷♀️ More common, though, are setups where there's an individual mixer for each outgoing port of the middlebox, so each receiving endpoint can get its own composition. That's the feature people call personalized layout 💜🌈
Okay but the problems 😭 Media mixing eats up a lot of media processing resources (for the decoding and mixing in the uncompressed domain) and encoding resources (for encoding the mixed signal) 💻 Another issue is quality degradation from decoding and re-encoding, which happens because the most commonly used media codecs are lossy 📉 A third problem is the latency media mixing adds. It can be substantial, and it's annoyingly noticeable with video, or with audio if that mixed audio is lip-synchronized with high-latency video 🙃 Now the upside: it's really easy for the endpoints to handle a single media stream (the mixed aggregate of many sources), because they don't have to deal with multiple decodings, local mixing, and composition 🫶 Honestly, mixers were introduced in pre-RTP times so that legacy, single-stream receiving endpoints (which, in some protocol environments, didn't even need to know about the multipoint nature of the conference) could still take part in what a user would recognize as a multiparty video conference 📺💖 Main character energy for legacy devices 👑
+-A---------+ +-MIXER----------------------+
| +-RTP1----| |-RTP1------+ +-----+ |
| | +-Audio-| |-Audio---+ | +---+ | | |
| | | AA1|--------->|---------+-+-|DEC|->| | |
| | | |<---------|MA1 <----+ | +---+ | | |
| | | | |(BA1+CA1)|\| +---+ | | |
| | +-------| |---------+ +-|ENC|<-| B+C | |
| +---------| |-----------+ +---+ | | |
+-----------+ | | | |
| | M | |
+-B---------+ | | E | |
| +-RTP2----| |-RTP2------+ | D | |
| | +-Audio-| |-Audio---+ | +---+ | I | |
| | | BA1|--------->|---------+-+-|DEC|->| A | |
| | | |<---------|MA2 <----+ | +---+ | | |
| | +-------| |(AA1+CA1)|\| +---+ | | |
| +---------| |---------+ +-|ENC|<-| A+C | |
+-----------+ |-----------+ +---+ | | |
| | M | |
+-C---------+ | | I | |
| +-RTP3----| |-RTP3------+ | X | |
| | +-Audio-| |-Audio---+ | +---+ | E | |
| | | CA1|--------->|---------+-+-|DEC|->| R | |
| | | |<---------|MA3 <----+ | +---+ | | |
| | +-------| |(AA1+BA1)|\| +---+ | | |
| +---------| |---------+ +-|ENC|<-| A+B | |
+-----------+ |-----------+ +---+ +-----+ |
+----------------------------+Figure 15: Session and SSRC Details for Media Mixer
From an RTP point of view, media mixing can be a very simple process, as you can see in Figure 15 🦋 The mixer presents one SSRC towards the receiving endpoint, e.g., MA1 to Peer A, and that stream is the media mix of the other endpoints 💫 Each peer, in this example, gets a different version of the mix from the mixer, so there's no actual relation between the different RTP sessions in terms of actual media or transport-level information 🙅♀️ There are, though, common relationships between RTP1-RTP3, namely SSRC space and identity information 🤝 When A receives the MA1 stream (which is a combination of BA1 and CA1), the mixer may include CSRC information in the MA1 stream to identify the Contributing Sources BA1 and CA1. That lets the receiver identify the Contributing Sources even if it couldn't do that through the media itself or through other signaling means 🔍🌟
The CSRC is, in turn, useful in RTP extensions, like the RTP header extension for Mixer-to-Client Audio Level Indication [RFC6465] 🎚️ If the SSRCs from the endpoint-to-mixer paths are used as CSRCs in another RTP session, then RTP1, RTP2, and RTP3 become one joint session, because they share a common SSRC space 🧚 At this stage, the mixer also has to think about which RTCP information it needs to expose on the different paths. In the scenario above, a mixer would normally expose nothing more than the SDES information and an RTCP BYE for a CSRC leaving the session 👋 The main goal is to enable correct binding against the application logic and other information sources 🎯 And this also enables loop detection in the RTP session 🔁✨
3.6.2. Media-Switching Mixer ✨
okay bestie, so Media-Switching Mixers 💅 are for limited-functionality vibes, where the app doesn't need to show lots of sources at once (or only very few). They also show up in the more complex multi-stream setups with receiver mixing or tiling, even when they're combined with simulcast and/or scalability between the source and the mixer 🦋.
Here's the tea: a media-switching mixer skips the decoding and encoding steps. It basically forwards the encoded media the way it was sent to the mixer ✨. It does still do the decryption and re-encryption cycle though, because it rewrites RTP headers 🙈. Forwarding instead of reconstructing, mixing, and re-encoding saves compute resources and makes the media better (fidelity and reduced latency), so it's giving efficiency 👑.
A Media-Switching Mixer keeps a pool of SSRCs 🌸 that stand for conceptual or functional RTP streams the mixer can produce. These streams are made by picking media from one of the RTP streams the mixer receives, then sending it to the peer using the mixer's own SSRCs. The mixer can switch between available sources when the concept of the source calls for it, like the currently active speaker 🎀. Lowkey, though, the mixer usually still has to do some media processing, because many media formats don't let you "tune into" the stream at any random point in the bitstream 💀.
To get a coherent RTP stream from the mixer's SSRC, the mixer has to rewrite the header of each incoming RTP packet. The steps are:
- The SSRC field gets set to the mixer's SSRC.
- The sequence number has to be the next one in the sequence of outgoing packets the mixer sent.
- The RTP timestamp gets adjusted with an offset that changes every time the mixer switches the Media Source.
- Depending on how the RTP payload type was negotiated, the value for this particular payload configuration might need to change, if the different endpoint-to-mixer paths didn't end up using the same numbering for a given configuration.
This also means the endpoints need to support a common set of codecs. Otherwise, media transcoding for codec compatibility would still be needed 😭.
We're now looking at a Media-Switching Mixer running a video conference with six participating endpoints (A-F), where the two most recent speakers are shown to each receiving endpoint 🧚. So the mixer has two SSRCs sending video to each peer, and each peer can handle two video streams at the same time 💜.
+-A---------+ +-MIXER----------------------+
| +-RTP1----| |-RTP1------+ +-----+ |
| | +-Video-| |-Video---+ | | | |
| | | AV1|------------>|---------+-+------->| S | |
| | | |<------------|MV1 <----+-+-BV1----| W | |
| | | |<------------|MV2 <----+-+-EV1----| I | |
| | +-------| |---------+ | | T | |
| +---------| |-----------+ | C | |
+-----------+ | | H | |
| | | |
+-B---------+ | | M | |
| +-RTP2----| |-RTP2------+ | A | |
| | +-Video-| |-Video---+ | | T | |
| | | BV1|------------>|---------+-+------->| R | |
| | | |<------------|MV3 <----+-+-AV1----| I | |
| | | |<------------|MV4 <----+-+-EV1----| X | |
| | +-------| |---------+ | | | |
| +---------| |-----------+ | | |
+-----------+ | | | |
: : : :
: : : :
+-F---------+ | | | |
| +-RTP6----| |-RTP6------+ | | |
| | +-Video-| |-Video---+ | | | |
| | | FV1|------------>|---------+-+------->| | |
| | | |<------------|MV11 <---+-+-AV1----| | |
| | | |<------------|MV12 <---+-+-EV1----| | |
| | +-------| |---------+ | | | |
| +---------| |-----------+ +-----+ |
+-----------+ +----------------------------+Figure 16: Media-Switching RTP Mixer
Just like the Media-Mixing Mixer, the Media-Switching Mixer can cut the bitrate needed to send media to the different peers. It does this by selecting and forwarding only some of the RTP streams it gets from the sending endpoints 🦋. If the mixer receives simulcast transmissions or a scalable encoding of the Media Source, it has more freedom to pick streams, or subsets of streams, to forward to a receiving endpoint. That choice can be based on transport or endpoint restrictions, as well as application logic ✨.
To make sure a media receiver in an endpoint can decode the RTP stream correctly after a switch, a codec that uses temporal prediction has to start decoding from independent refresh points, or from points in the bitstream that offer similar functionality (like "dirty refresh points") 🫶. For some codecs, like frame-based speech and audio codecs, this is easy: decoding can start at RTP packet boundaries, since each packet boundary is a refresh point (assuming the encoder packetizes properly) 🌈. Other codecs, especially video, have refresh points that are less common in the bitstream, or may not be there at all unless the encoder gets an explicit request. The Full Intra Request [RFC5104] RTCP codec control message was defined for exactly this purpose 💖.
With this type of mixer, you could also think about fully terminating the RTP sessions between the different endpoint and mixer paths. The same arguments and considerations discussed in Section 3.9 need to be taken into account here too, and they apply in this case 🍓.
3.7. Selective Forwarding Middlebox 🦋
okay bestie, this one is a whole vibe 💅 The big idea is that the middlebox "projects" (aka puts out on display) every potential RTP source (SSRC) into a separate RTP session for each endpoint. Then it picks which of the sources that are actually sending media right now go to each endpoint. It's giving Media-Switching Mixer, but the RTP details are different in some important ways 👀
+-A---------+ +-Middlebox-----------------+
| +-RTP1----| |-RTP1------+ +-----+ |
| | +-Video-| |-Video---+ | | | |
| | | AV1|------------>|---------+-+------>| | |
| | | |<------------|BV1 <----+-+-------| S | |
| | | |<------------|CV1 <----+-+-------| W | |
| | | |<------------|DV1 <----+-+-------| I | |
| | | |<------------|EV1 <----+-+-------| T | |
| | | |<------------|FV1 <----+-+-------| C | |
| | +-------| |---------+ | | H | |
| +---------| |-----------+ | | |
+-----------+ | | M | |
| | A | |
+-B---------+ | | T | |
| +-RTP2----| |-RTP2------+ | R | |
| | +-Video-| |-Video---+ | | I | |
| | | BV1|------------>|---------+-+------>| X | |
| | | |<------------|AV1 <----+-+-------| | |
| | | |<------------|CV1 <----+-+-------| | |
| | | | : : : |: : : : : : : : :| | |
| | | |<------------|FV1 <----+-+-------| | |
| | +-------| |---------+ | | | |
| +---------| |-----------+ | | |
+-----------+ | | | |
: : : :
: : : :
+-F---------+ | | | |
| +-RTP6----| |-RTP6------+ | | |
| | +-Video-| |-Video---+ | | | |
| | | FV1|------------>|---------+-+------>| | |
| | | |<------------|AV1 <----+-+-------| | |
| | | | : : : |: : : : : : : : :| | |
| | | |<------------|EV1 <----+-+-------| | |
| | +-------| |---------+ | | | |
| +---------| |-----------+ +-----+ |
+-----------+ +---------------------------+
Figure 17: Selective Forwarding MiddleboxCheck out Figure 17, a six endpoint conference ✨ Endpoint A can see five incoming SSRCs, BV1 through FV1. Now, if the middlebox wants to act like Section 3.6.2, where the mixer gives endpoints the two latest speakers, then only two of those five SSRCs need to be sending media to A at the same time. Lowkey so efficient 🧚
Here's the catch: every projected RTP stream has to get some RTP header fields rewritten as it moves from one session into another 🙈 The sequence number has to keep counting up consecutively, based on the packets actually sent in that particular RTP session. So the sequence number offset changes every time a source gets switched on in a session. The timestamp (possibly offset) stays the same. Periodt.
The RTP sessions are independent, which means their SSRC numbers can be handled independently too. That makes SSRC collision detection and avoidance way simpler, but you now need stuff like remapping tables between the sessions 🎀 Using independent RTP sessions isn't required, though. The switching can also work with one common SSRC space. In that case, though, collision detection turns into a different problem entirely. It's up to the implementation to pick a single common SSRC space or separate ones.
Separate SSRC spaces come with some real consequences, so pay attention bestie 👑 For example, the RTP stream endpoint B sends to the middlebox (BV1) might use an SSRC of 12345678. When the middlebox sends that same stream to endpoint F, it can use any SSRC value, like 87654321. So each endpoint can end up with a different view of how a particular SSRC is being used by the application. Any RTP-level identity info, like SDES items, also has to update the SSRC it references if those SDES items are meant to be global. This is why the application must not use SSRC as a reference to RTP streams when it's talking directly to other peers. It also breaks loop detection, since there's no common namespace or identities across the different legs of the Communication Session at the RTP level. That job falls to the higher layers instead 😭
The application must not use SSRC as references to RTP streams when communicating with other peers directly. Loop detection will fail without a common namespace, so higher layers have to handle it.
The middlebox also handles any RTCP codec control requests that come from an endpoint. It decides whether it can act on the request locally, or whether it needs to pass the request along to the RTP session/transport leg that carries the Media Source 💅 Both the endpoints and the middlebox need to implement conference-related codec control features to give a good experience. Two commonly used ones are the Full Intra Request, which asks the Media Source to provide switching points between the sources, and the Temporary Maximum Media Bitrate Request (TMMBR), which lets the middlebox combine congestion control responses sent toward the Media Source so it can adjust its bitrate. (That only makes sense when the limit isn't on the source-to-middlebox link, obviously.)
The Selective Forwarding Middlebox showed up in recent videoconferencing systems alongside scalable video coding and simulcasting, and it's built to make the most of both 🌈 Annex G of H.264 is an example of scalable video coding. Other codecs, like H.264 AVC and VP8, are scalable too, just only in the temporal dimension. With both scalable coding and simulcast, the video signal is represented by two or more bitstreams, each giving a different fidelity point. The middlebox picks which parts of a scalable bitstream (or which bitstream, for simulcast) go to each receiving endpoint. That choice can depend on things like available bitrate, the desired layout, and so on.
Compared to transcoding MCUs, SFMs are extremely low delay, and they still offer high-end features like personalized layouts and error localization, all without doing any signal processing at the middlebox ✨ They can also scale to a huge number of concurrent users, and because their delay is so low, they can be cascaded too. Iconic behavior 🫶
This version of the middlebox also asks more of the endpoint when it comes to decoder instances and handling the RTP streams carrying media 🍓 Since any projected SSRC can start providing media at any time, the endpoint either has to handle as many decoder instances as the middlebox has received, or do efficient switching of decoder contexts across a smaller set of actual decoder instances. The application also takes on more responsibility for updating how the media is presented to the user.
Note that this topology could be seen as a Media Translator that also has on/off logic built into its media translation. It has the property that every SSRC in the session is visible to an endpoint. It also has mixer aspects, since the streams it provides aren't basically translated versions. Instead, they have a conceptual property assigned to them, they can be turned on and off, and they can be delivered fully or partially. So this topology is kind of a hybrid between the translator and mixer models 🌸
The differences between a Selective Forwarding Middlebox and a Switching-Media Mixer (Section 3.6.2) are pretty minor, and they share most properties. One difference is the decoding instance requirement, or needing efficient switching of decoder contexts. The other is how identification works. The mixer uses CSRC to show what's included in a particular RTP stream representing a particular concept. Selective forwarding gets source info from the SSRC instead, and uses other mechanisms to indicate the stream's intended usage, if needed 💜
3.8. Point to Multipoint Using Video-Switching MCUs ✨
Shortcut name: Topo-Video-switch-MCU
+---+ +------------+ +---+
| A |------| Multipoint |------| B |
+---+ | Control | +---+
| Unit |
+---+ | (MCU) | +---+
| C |------| |------| D |
+---+ +------------+ +---+Figure 18: Point to Multipoint Using a Video-Switching MCU
Okay bestie, this PtM topology was super popular in the early days of multipoint videoconferencing 🎀 because it was so simple! The middlebox design behind it was known as a "video-switching MCU". The more complex RTCP-terminating MCUs (we'll get to those in the next section 💅) became the norm once the tech was cheap enough to be worth it 🫶.
A video-switching MCU forwards a single media stream to a participant, picked from the available streams. The selection is often based on voice activity in the audio-visual conference, but other conference management mechanisms, like presentation mode or explicit floor control, are known to exist too 🦋.
The video-switching MCU may also perform media translation to change the content's bitrate, encoding, or resolution 🌈. Even so, it may still indicate the original sender of the content through the SSRC. In that case, the values of the CC and CSRC fields are retained.
If the MCU is not terminating RTP, the RTCP sender reports are forwarded for the currently selected sender. All RTCP receiver reports are freely forwarded between the endpoints 💜. The MCU may also originate RTCP control traffic to control the session and/or report on status from its own viewpoint.
The video-switching MCU has most of the attributes of a translator, but its stream selection is a mixing behavior, and that comes with some RTP and RTCP issues 😭. Suppressing all but one RTP stream means most participants see only a subset of the sent RTP streams at any given time, often a single RTP stream per conference. So RTCP receiver reports only report on those RTP streams. The result is that endpoints emitting RTP streams that aren't currently forwarded get a view of the session where their RTP streams disappear somewhere en route. That makes using RTCP for congestion control, or any type of quality reporting, very problematic 💀.
To avoid these issues, the MCU needs two features 🔥. First, it needs to act as a mixer (see Section 3.6) and forward the selected RTP stream under its own SSRC and with the appropriate CSRC values. Second, the MCU needs to modify the RTCP RRs it forwards between the domains. Because of this, it is recommended that one implement a centralized video-switching conference using a mixer according to RFC 3550, instead of the shortcut implementation described here 🧚.
3.9. Point to Multipoint Using RTCP-Terminating MCU ✨
Shortcut name: Topo-RTCP-terminating-MCU
+---+ +------------+ +---+
| A |<---->| Multipoint |<---->| B |
+---+ | Control | +---+
| Unit |
+---+ | (MCU) | +---+
| C |<---->| |<---->| D |
+---+ +------------+ +---+Figure 19: Point to Multipoint Using Content Modifying MCUs
In this PtM scenario, each endpoint runs an RTP point-to-point session between itself and the MCU 🌸. This is a very commonly deployed topology in multipoint video conferencing 👑. The content the MCU gives each participant is either:
a. a selection of the content received from the other endpoints, or
b. the mixed aggregate of what the MCU receives from the other PtP paths, which are part of the same Communication Session.
In case (a), the MCU may modify the content in terms of bitrate, encoding format, or resolution. No explicit RTP mechanism establishes the relationship between the original RTP stream of the media being sent and the RTP stream the MCU sends. In other words, the outgoing RTP streams typically use a different SSRC, and may well use a different payload type (PT), even if that different PT happens to be mapped to the same media type. This is because each endpoint gets its own individually negotiated RTP session 🍓.
In case (b), the MCU is the Media Source and generates the Source RTP Stream as it mixes the received content, then encodes and packetizes it for transmission to an endpoint. According to RTP [RFC3550], the SSRC of the contributors is to be signaled using the CSRC/CC mechanism. In practice, though, most deployed MCUs don't implement this feature 🙈. Instead, the identification of the endpoints whose content is included in the mixer's output isn't indicated through any explicit RTP mechanism. Most deployed MCUs set the CC field in the RTP header to zero, which indicates no available CSRC information, even if they could identify the original sending endpoints as RTP suggests.
The main thing that sets this topology apart from what RFC 3550 describes is that the common RTP session gets broken across the centralized device, such as the MCU 🦋. This means the loss of explicit RTP-level indication of all participants. If someone used the mechanisms RTP and RTCP offer to signal this explicitly, the topology would follow the approach of an RTP mixer. Without that explicit indication, there are at least the following potential problems 😬:
- Loop detection cannot be performed on the RTP level. When two misconfigured MCUs are carelessly connected, a loop could be generated.
- There is no information about active media senders available in the RTP packet. Since this information is missing, receivers cannot use it. It also deprives the client of information about currently active senders in a machine-usable way, which prevents clients from indicating currently active speakers in user interfaces, etc.
Note that many or most deployed MCUs (and video conferencing endpoints) rely on signaling-layer mechanisms to identify the Contributing Sources, for example, a SIP conferencing package [RFC4575]. This alleviates, to some extent, the aforementioned issues resulting from ignoring RTP's CSRC mechanism 🫶💖.
3.10. Split Component Terminal 🎀
Shortcut name: Topo-Split-Terminal
Okay so, in some applications (like some telepresence systems) a terminal isn't one neat little box. It's more like a squad of subunits working together 🫶. Picture a telepresence room with multiple cameras and monitors, where each camera-and-monitor pair is its own video conferencing subunit. Or a video conferencing terminal where audio is handled by one subunit and video by another. Each of these subunits has its own physical network interface (like an Ethernet jack) and its own network address. Main character energy, but make it distributed 💅.
The subunits need to be logically and physically interconnected by control functionality, but their media plane functionality may be split. These are called split component terminals. Historically, the earliest ones were probably the independent audio and video conference software tools used over the MBONE in the late 1990s. Iconic origins, honestly 🦋.
Here's an example of a split component terminal in Figure 20. Inside split component terminal A, at least the audio and video subunits are addressed by their own network addresses. In some of these systems, the control stack subunit may also have its own network address.
+---------------------+
| Endpoint A |
| Local Area Network |
| +------------+ |
| +->| Audio |<+-RTP---\
| | +------------+ | \ +------+
| | +------------+ | +-->| |
| +->| Video |<+-RTP-------->| B |
| | +------------+ | +-->| |
| | +------------+ | / +------+
| +->| Control |<+-SIP---/
| +------------+ |
+---------------------+
Figure 20: Split Component TerminalFrom an RTP point of view, each subunit terminates RTP and acts as an endpoint, since each one includes its own independent RTP stack. 🌸 But here's the twist: because the subunits are semantically part of the same terminal, it makes sense to express that relationship in RTCP protocol elements, specifically in the CNAME.
It's also sensible for the subunits to share a common clock, which the RTP and RTCP clocks are derived from. This helps with synchronization and avoids clock drift. Teamwork makes the dream work ⏰✨.
So, to show that audio and video Source Streams generated by different subunits share a common clock and can be synchronized, the RTP streams generated from those Source Streams need to include the same CNAME in their RTCP SDES packets. Using a common CNAME across RTP flows carried in different transport-layer flows is totally normal for RTP and RTCP senders. Fully compliant RTP endpoints, middleboxes, and other tools should have no problem with it 💖.
Now, outside of the split component terminal scenario (and maybe a multihomed endpoint scenario, which isn't discussed further here), using a common CNAME in RTP streams sent from separate endpoints is rare. It's different from using a common CNAME for RTP streams sent on different transport-layer flows between two endpoints. It has been reported that at least some third-party tools, like some network monitors, don't handle endpoints using a common CNAME across multiple transport-layer flows gracefully. They report an error condition saying two separate endpoints are using the same CNAME 😭. Depending on how sophisticated the support staff is, those erroneous reports can lead to support issues.
⚠️ Heads up, bestie: that false "two endpoints, same CNAME" alarm is the exact support headache this section is about.
The support issue can sometimes be avoided if each subunit of a split component terminal uses a different CNAME. The synchronization between the RTP streams is then indicated by some non-RTP signaling channel instead of a common CNAME sent in RTCP. The catch is that this complicates the signaling, especially when there are multiple SSRCs in use with complex synchronization requirements, which is the case in many current telepresence systems. Unless you use RTCP terminating topologies such as Topo-RTCP-terminating-MCU, sessions involving more than one video subunit with a common CNAME are close to unavoidable 🙈.
The different RTP streams in a split terminal system can form a single RTP session or multiple RTP sessions. It all depends on how visible their SSRC values are in RTCP reports. If the receiver of the RTP streams sent by the split terminal reports on all of the RTP flows (that is, on each SSRC) in each RTCP report, then a single RTP session is formed. If, instead, the receiver doesn't send cross-reports in RTCP, then the audio and video form separate RTP sessions. Two paths, one choice 🧚.
Let's look at Figure 20 as an example. B sends RTCP reports to each of A's subunits. If the RTCP packets B sends to the audio subunit of A include reports on the reception quality of the video as well as the audio, and the RTCP packets B sends to the video subunit of A include reports on the reception quality of the audio as well as the video, then a single RTP session is formed. However, if the RTCP packets B sends to the audio subunit of A only report on the received audio, and the RTCP packets B sends to the video subunit of A only report on the received video, then there are two separate RTP sessions. ✨
Forming a single RTP session across the RTP streams sent by the different subunits of a split terminal gives each subunit visibility into the reception quality of the RTP streams sent by the other subunits. Lowkey useful! This information can help diagnose reception quality problems, but it comes at the cost of increased RTCP bandwidth use. Everything is a trade-off, babe 💀.
Last but not least: RTP streams sent by the subunits of a split terminal need to use the same CNAME in their RTCP packets if they are to be synchronized, whether or not a single RTP session is formed. No exceptions, periodt 👑.
3.11. Non-symmetric Mixer/Translators 🦋
Shortcut name: Topo-Asymmetric
It's theoretically possible to build an MCU that acts as a mixer in one direction and a translator in the other. 🌈 The main reason you'd want this is to allow topologies similar to Figure 13, where the mixer doesn't need to mix in the direction from B or D toward the multicast domains with A and C. Instead, the RTP streams from B and D get forwarded without changes. Skipping that mixing saves media processing resources in cases where it isn't needed. Still, there would be a need to mix B's media toward D. The mixer/translator could only work as a translator in the direction B -> multicast domain or D -> multicast domain. In all other directions, it would function as a mixer.
The mixer/translator would still need to process and change the RTCP before forwarding it in the directions of B or D to the multicast domain. Here's the tricky part 😵💫: A and C don't know about the mixed-media stream the mixer sends to either B or D. So any reports related to these streams must be removed. Also, receiver reports related to A's and C's RTP streams would be missing. To stop A and C from thinking that B and D aren't receiving A and C at all, the mixer needs to insert locally generated reports reflecting the situation for the streams from A and C into the sender reports for B and D. In the opposite direction, the receiver reports from A and C about B's and D's streams also need to be aggregated into the mixer's receiver reports sent to B and D. Since B and D only have the mixer as a source for the stream, all RTCP from A and C must be suppressed by the mixer. Whew, that's a lot of bookkeeping 🙈.
This topology is so problematic, and it's so easy to get the RTCP processing wrong, that it is not recommended for implementation. Bestie, maybe just don't 💅.
3.12. Combining Topologies 🧩
Topologies can be combined and linked to each other using mixers or translators. 💜 However, you have to take care with the SSRC/CSRC space. A mixer does not forward RTCP from sources in other domains. Instead, it generates its own RTCP packets for each domain it mixes into, including the necessary SDES information for both the CSRCs and the SSRCs. So in a mixed domain, the only SSRCs you'll see are the ones present in that domain, while there can be CSRCs from all the domains connected together with a combination of mixers and translators. The combined SSRC and CSRC space is common over any translator or mixer.
This matters for loop detection, which is likely to be even more important in combined topologies because of the mixed behavior between the domains. Any hybrid, like the Topo-Video-switch-MCU or Topo-Asymmetric, requires considerable thought on how RTCP is dealt with. Hybrids are a whole vibe, and not the chill kind 🌸.
4. Topology Properties ✨
The topologies discussed in Section 3 have different properties, and this section describes them. Note that even if a certain property is supported within a particular topology concept, the necessary functionality may be optional to implement. So "supported" doesn't automatically mean "you have to build it" 🫶🧚.
4.1. All-to-All Media Transmission 💖
Okay so, quick recap bestie 🫶: multicast, especially ASM, gives you the everyone-sends-to-everyone, everyone-hears-everyone vibe within the session. SSM can kinda do the same thing by having anyone who wants to participate as a sender send their media to the SSM Distribution Source. The SSM Distribution Source then forwards the media to all receivers subscribed to the multicast group. Mesh, MCUs, mixers, Selective Forwarding Middleboxes (SFMs), and translators can all provide that functionality too, at least on some basic level 💅. The catch is that they give you different kinds of reachability 👀.
The topologies that come closest to copying Any-Source IP Multicast, with all-to-all transmission, are the Transport Translator function called "relay" in Section 3.5, plus the Mesh with joint RTP sessions (Section 3.4) 🦋. Media Translators, Mesh with independent RTP Sessions, mixers, SFUs, and the MCU variants don't do fully meshed forwarding at the transport level 😭. They only allow limited forwarding of content from the other session participants 🌸.
Now this part is super important 🚨. The all-to-all media transmission requires that any media-transmitting endpoint considers the path to the least-capable receiving endpoint. Otherwise, the media transmissions may overload that path. So a sending endpoint needs to keep an eye on the path from itself to every receiving endpoint, detect the currently least-capable receiver, and adapt its sending rate to match 🎀. Since multiple endpoints may send at the same time, the available resources may vary. RTCP's receiver reports help with this monitoring, at least on a medium time scale 📊.
Resource use for all-to-all transmission depends on the topology 💜. Both ASM and SSM have the perk that only one copy of each packet traverses a particular link ✨. Using a relay means one copy of a packet gets transmitted per endpoint-to-relay path and per packet sent. But in most cases, the links carrying those multiple copies will be the ones close to the relay (which can be assumed to be part of the network infrastructure with good connectivity to the backbone), not the endpoints (which may be stuck behind slower access links) 🙈. Mesh topologies cause N-1 streams of transmitted packets to traverse the first-hop link from the endpoint, in a mesh with N endpoints. How much those paths overlap is highly situation dependent 🧚.
RTCP is designed to adapt to changes in the number of participants, thanks to the transmission algorithm defined in the RTP specification [RFC3550] and the extensions in AVPF [RFC4585] (when applicable) 🌈. That way, the resources used for RTCP stay within the bounds configured for the session 💅.
4.2. Transport or Media Interoperability 🦋
All translators, mixers, RTCP-terminating MCUs, and Mesh with individual RTP sessions let you change the media encoding, or the transport and other properties, to match the other domain. That gives you extended interoperability when the endpoints don't have a common set of media codecs and/or transport protocols 🫶.
Selective Forwarding Middleboxes can adopt the transport and (at least) selectively forward the encoded streams that match a receiving endpoint's capability ✨. If the encoded streams don't match what the receiving endpoint can handle, it requires an additional translator to change the media encoding 🔧.
4.3. Per-Domain Bitrate Adaptation 🌸
Endpoints are often connected to each other over a heterogeneous set of paths, which makes congestion control in a Point-to-Multipoint set kinda messy 😩. In the ASM, SSM, Mesh with common RTP session, and Transport Relay scenarios, each individual sending endpoint has to adapt to the receiving endpoint behind the least-capable path. The result is suboptimal quality for the endpoints behind the more capable paths 💔.
It's a whole different story once Media Translators, mixers, SFMs, or MCUs are involved 🙌. Each endpoint only needs to adapt to the slowest path within its own domain. The translator, mixer, SFM, and MCU topologies all require their respective outgoing RTP streams to adjust the bitrate, packet rate, and so on, to adapt to the least-capable path in each of the other domains. That way, you avoid dragging everyone in all the domains down to the quality of the least-capable endpoint 🍓. The price is paid in complexity, delay, and equipment at the mixer, SFM, or translator, and potentially at the media sender too (multicast or layered encoding, sending the different representations) 💸.
4.4. Aggregation of Media 🎀
In the all-to-all media property mentioned above, which is provided by ASM, SSM, Mesh with common RTP session, and relay, all the simultaneous media transmissions share the available bitrate 🥲. For endpoints with limited reception capabilities, this can mean that even a minimal, acceptable media quality can't be reached, because multiple RTP streams need to share the same resources 😵.
One solution is to use a mixer, or MCU, to aggregate the multiple RTP streams into a single one that takes up fewer resources in terms of bitrate 🧠. This aggregation can be done in different ways. Mixing and selection are the two common methods. Selection is almost always possible and easy to implement ✅. Mixing needs resources in the mixer, and it may be relatively easy and not impair quality too badly (audio), or it may be quite difficult (video tiling). Video tiling is not only computationally complex but also reduces the pixel count per stream, with a corresponding loss in perceptual quality 📉.
4.5. View of All Session Participants 👀
RTP includes functionality to identify the session participants through the SSRC and CSRC fields 🔍. It can also carry some further identity information about these participants using RTCP SDES 🪪.
In topologies that provide full all-to-all functionality, like ASM, Mesh with common RTP session, and relay, a compliant RTP implementation offers this functionality directly, as specified in RTP 💖. In topologies that don't offer all-to-all communication, RTCP has to be handled correctly in domain bridging functions 🌉. RTP includes explicit specification text for translators and mixers, and for SFMs the required functionality can be derived from that text 📜.
However, the MCU described in Section 3.8 can't offer the full functionality for session participant identification through RTP means 😬. The topologies that create independent RTP sessions per endpoint or per pair of endpoints do not support RTP-based identification of session participants. That includes Back-to-Back RTP session, MESH with independent RTP sessions, and the RTCP terminating MCU (Section 3.9), with the exception of SFM 🙅. In all those cases, other non-RTP-based mechanisms need to be implemented if that knowledge is required or desirable 🛠️.
When it comes to SFM, the SSRC namespace is not necessarily joint 🌀. Instead, identification requires knowledge of the SSRC/CSRC mappings that the SFM performed; see Section 3.7 🗺️.
4.6. Loop Detection 🔁
In complex topologies with multiple interconnected domains, it's possible to form media loops unintentionally 😱. RTP and RTCP support detecting such loops, as long as the SSRC and CSRC identities are maintained and correctly set in forwarded packets 🫡.
Loop detection will work in ASM, SSM, Mesh with joint RTP session, and relay ✅. It's likely that loop detection works for the video-switching MCU (Section 3.8), at least as long as it forwards the RTCP between the endpoints 🤞. However, Back-to-Back RTP sessions, Mesh with independent RTP sessions, and SFMs will definitely break the loop detection mechanism 💀.
4.7. Consistency between Header Extensions and RTCP 🧩
Some RTP header extensions matter not only end to end but also hop to hop 🦋. That means at least some of the middleboxes in the path are aware of their potential presence through signaling, intercept and interpret those header extensions, and potentially also rewrite or generate them 🔄.
Modern header extensions generally follow "A General Mechanism for RTP Header Extensions" [RFC5285], which allows for all of the above 🌟. Examples include the Media ID (MID) in [SDP-BUNDLE]. At the time of writing, there was also a proposal for how to include some SDES into an RTP header extension [RTCP-SDES] 📝.
When these header extensions are in use, any middlebox that understands one must ensure consistency between the extensions it sees and/or generates and the RTCP it receives and generates. Wait, I can't capitalise that. So here's the accurate version 👇
When these header extensions are in use, any middlebox that understands one must ensure consistency between the extensions it sees and/or generates and the RTCP it receives and generates 🔐. For example, the MID of the bundle is sent in an RTP header extension and also in an RTCP SDES message. This apparent redundancy was introduced because unaware middleboxes may choose to discard RTP header extensions 🙃. Obviously, inconsistency between the MID sent in the RTP header extension and the MID in the RTCP SDES message could lead to undesirable results, so consistency is needed 🚫.
Middleboxes that are unaware of the nature of a header extension, as specified in [RFC5285], are free to forward or discard header extensions 🤷♀️.
5. Comparison of Topologies 📊
The table below attempts to summarise the properties of the different topologies 💅. The legend for the topology abbreviations is:
Topo-Point-to-Point (PtP), Topo-ASM (ASM), Topo-SSM (SSM), Topo-Trn-Translator (TT), Topo-Media-Translator (including Transport Translator) (MT), Topo-Mesh with joint session (MJS), Topo-Mesh with individual sessions (MIS), Topo-Mixer (Mix), Topo-Asymmetric (ASY), Topo-Video-switch-MCU (VSM), Topo-RTCP-terminating-MCU (RTM), and Selective Forwarding Middlebox (SFM) 🦋.
In the table below, Y indicates Yes or full support, N indicates No support, (Y) indicates partial support, and N/A indicates not applicable 🔤.
Property PtP ASM SSM TT MT MJS MIS Mix ASY VSM RTM SFM
---------------------------------------------------------------------
All-to-All Media N Y (Y) Y Y Y (Y) (Y) (Y) (Y) (Y) (Y)
Interoperability N/A N N Y Y Y Y Y Y N Y Y
Per-Domain Adaptation N/A N N N Y N Y Y Y N Y Y
Aggregation of Media N N N N N N N Y (Y) Y Y N
Full Session View Y Y Y Y Y Y N Y Y (Y) N Y
Loop Detection Y Y Y Y Y Y N Y Y (Y) N NPlease note that the Media Translator also includes the Transport Translator functionality 📌✨
6. Security Considerations 🔐
okay so security is where things get SERIOUS bestie 😭 mixers, SFMs, and translators all modify packets, which means you can't just rely on integrity and source authentication like normal, unless these devices are trusted and part of the security context. Like, they can send Secure Real-time Transport Protocol (SRTP) and Secure Real-time Transport Control Protocol (SRTCP) [RFC3711] packets to endpoints in the Communication Session 🎀. If encryption is on, the Media Translator, SFM, and mixer need to be able to decrypt the media to do their job.
A Transport Translator can get away without touching the encrypted payload, but only when it's translating stuff that isn't covered by the encryption and integrity protection, like IP address and UDP port numbers in an SRTP media stream [RFC3711]. ngl though, in general the translator, SFM, or mixer needs to be part of the signaling context and get its security associations (e.g., SRTP crypto contexts) set up with its RTP session participants 🦋
🚨 Including the mixer, SFM, or translator in the security context means that if it gets subverted or starts misbehaving, it has full access and can pull off a bunch of very serious attacks. It can do every attack possible (see RFC 3550 and any applicable profiles) as if the media session weren't protected at all, while making the humans think they're protected. Not it. 💀
Transport Translators don't interact with crypto that works above the transport layer, like SRTP, because they leave the RTP header and payload totally unchanged 💅 Media Translators, on the other hand, are super tangled up with crypto since they alter the RTP payload. A Media Translator in a session that uses cryptographic protection needs to do crypto processing on both inbound and outbound packets.
A Media Translator might need different cryptographic keys for inbound and outbound processing. For SRTP, different keys are needed, because an RFC 3550 Media Translator leaves the SSRC unchanged during packet processing, and SRTP key sharing is only allowed when distinct SSRCs can be used to protect distinct packet streams 🌸
When the Media Translator uses different keys for inbound and outbound packets, each session participant has to get the right key, depending on whether they're listening to the translator or the original source 🫶 (Also, there's an architectural difference here that's giving main character energy: with RTP media translation, participants can use the RTP payload type field to figure out how to process a packet. With cryptographically protected media translation, participants have to use info that isn't carried in the packet at all. iykyk 🙈)
When you use security mechanisms with translators, SFMs, and mixers, the translator, SFM, or mixer could make different security associations for the different domains it's working in. That comes with some implications:
First, security might get weaker if the mixer or translator accepts a weaker algorithm or key in one domain compared to another. So be careful that appropriately strong security parameters get negotiated in all domains 👑 In a lot of cases, "appropriate" just means "similar" strength. And if a key-management system does allow negotiating parameters that end up with different security strengths, then that system should let the participants in the other domains know about it 🔥
Second, the number of crypto contexts (keys and security-related state) needed (for example, in SRTP [RFC3711]) can vary between mixers, SFMs, and translators. A mixer normally only needs to represent a single SSRC per domain, so it only needs to create one security association (SRTP crypto context) per domain ✨ A translator, though, needs one security association per participant it translates towards, in the opposite domain. Looking at Figure 11, the translator needs two security associations towards the multicast domain: one for B and one for D. It might be forced to keep a set of totally independent security associations between itself and B and D, respectively, so it avoids two-time pad occurrences. These contexts also have to handle all the sources present in the other domains. So using completely independent security associations (for certain keying mechanisms) could force a translator to handle N*DM keys and related state, where N is the total number of SSRCs used over all domains and DM is the total number of domains 😭
The ASM, SSM, Relay, and Mesh (with common RTP session) topologies each have multiple endpoints that need shared knowledge about the different crypto contexts for the endpoints 🧚 These multiparty topologies have special requirements for key management and security functions. Source authentication in these environments has special requirements too, so it's not just a copy-paste situation.
There are a bunch of different mechanisms for giving keys to the different participants. One example is the choice between group keys and unique keys per SSRC 🌈 The right keying model depends on which topologies you plan to use. The final security properties depend on both the topologies in use and the properties of the keying mechanisms, so the application needs to consider this. Exactly which mechanisms get used is outside the scope of this document, so please check out RTP Security Options [RFC7201] to get a better understanding of most of the available options 💜
7. References
7.1. Normative References
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <http://www.rfc-editor.org/info/rfc3550>.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585,
DOI 10.17487/RFC4585, July 2006,
<http://www.rfc-editor.org/info/rfc4585>.
[RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and
B. Burman, Ed., "A Taxonomy of Grouping Semantics and
Mechanisms for Real-Time Transport Protocol (RTP)
Sources", RFC 7656, November 2015,
<http://www.rfc-editor.org/info/rfc7656>.7.2. Informative References
[MULTI-STREAM-OPT]
Lennox, J., Westerlund, M., Wu, W., and C. Perkins,
"Sending Multiple Media Streams in a Single RTP Session:
Grouping RTCP Reception Statistics and Other Feedback",
Work in Progress, draft-ietf-avtcore-rtp-multi-stream-
optimisation-08, October 2015.
[RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5,
RFC 1112, DOI 10.17487/RFC1112, August 1989,
<http://www.rfc-editor.org/info/rfc1112>.
[RFC3022] Srisuresh, P. and K. Egevang, "Traditional IP Network
Address Translator (Traditional NAT)", RFC 3022,
DOI 10.17487/RFC3022, January 2001,
<http://www.rfc-editor.org/info/rfc3022>.
[RFC3569] Bhattacharyya, S., Ed., "An Overview of Source-Specific
Multicast (SSM)", RFC 3569, DOI 10.17487/RFC3569, July
2003, <http://www.rfc-editor.org/info/rfc3569>.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, DOI 10.17487/RFC3711, March 2004,
<http://www.rfc-editor.org/info/rfc3711>.
[RFC4575] Rosenberg, J., Schulzrinne, H., and O. Levin, Ed., "A
Session Initiation Protocol (SIP) Event Package for
Conference State", RFC 4575, DOI 10.17487/RFC4575, August
2006, <http://www.rfc-editor.org/info/rfc4575>.
[RFC4607] Holbrook, H. and B. Cain, "Source-Specific Multicast for
IP", RFC 4607, DOI 10.17487/RFC4607, August 2006,
<http://www.rfc-editor.org/info/rfc4607>.
[RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman,
"Codec Control Messages in the RTP Audio-Visual Profile
with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104,
February 2008, <http://www.rfc-editor.org/info/rfc5104>.
[RFC5117] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117,
DOI 10.17487/RFC5117, January 2008,
<http://www.rfc-editor.org/info/rfc5117>.
[RFC5285] Singer, D. and H. Desineni, "A General Mechanism for RTP
Header Extensions", RFC 5285, DOI 10.17487/RFC5285, July
2008, <http://www.rfc-editor.org/info/rfc5285>.
[RFC5760] Ott, J., Chesterfield, J., and E. Schooler, "RTP Control
Protocol (RTCP) Extensions for Single-Source Multicast
Sessions with Unicast Feedback", RFC 5760,
DOI 10.17487/RFC5760, February 2010,
<http://www.rfc-editor.org/info/rfc5760>.
[RFC5766] Mahy, R., Matthews, P., and J. Rosenberg, "Traversal Using
Relays around NAT (TURN): Relay Extensions to Session
Traversal Utilities for NAT (STUN)", RFC 5766,
DOI 10.17487/RFC5766, April 2010,
<http://www.rfc-editor.org/info/rfc5766>.
[RFC6285] Ver Steeg, B., Begen, A., Van Caenegem, T., and Z. Vax,
"Unicast-Based Rapid Acquisition of Multicast RTP
Sessions", RFC 6285, DOI 10.17487/RFC6285, June 2011,
<http://www.rfc-editor.org/info/rfc6285>.
[RFC6465] Ivov, E., Ed., Marocco, E., Ed., and J. Lennox, "A Real-
time Transport Protocol (RTP) Header Extension for Mixer-
to-Client Audio Level Indication", RFC 6465,
DOI 10.17487/RFC6465, December 2011,
<http://www.rfc-editor.org/info/rfc6465>.
[RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP
Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014,
<http://www.rfc-editor.org/info/rfc7201>.
[RTCP-SDES]
Westerlund, M., Burman, B., Even, R., and M. Zanaty, "RTP
Header Extension for RTCP Source Description Items", Work
in Progress, draft-ietf-avtext-sdes-hdr-ext-02, July 2015.
[SDP-BUNDLE]
Holmberg, C., Alvestrand, H., and C. Jennings,
"Negotiating Media Multiplexing Using the Session
Description Protocol (SDP)", Work in Progress,
draft-ietf-mmusic-sdp-bundle-negotiation-23, July 2015.Acknowledgements
Huge thanks from the authors to Mark Baugher, Bo Burman, Ben Campbell, Umesh Chandra, Alex Eleftheriadis, Roni Even, Ladan Gharai, Geoff Hunt, Suresh Krishnan, Keith Lantz, Jonathan Lennox, Scarlet Liuyan, Suhas Nandakumar, Colin Perkins, and Dan Wing for helping review and improve this document 🫶✨
Authors' Addresses
Magnus Westerlund
Ericsson
Farogatan 2
SE-164 80 Kista
Sweden
Phone: +46 10 714 82 87
Email: magnus.westerlund@ericsson.com
Stephan Wenger
Vidyo
433 Hackensack Ave
Hackensack, NJ 07601
United States
Email: stewe@stewe.org📦 where this page comes from
- ietf: rfc7667 2015-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms