RFC 6051
Rapid Synchronisation of RTP Flows
RFC 6051: Rapid Synchronisation of RTP Flows 🎀
Okay bestie, this is RFC 6051, "Rapid Synchronisation of RTP Flows," published in November 2010 💜. It's a Standards Track document, so it's a real Internet standard and not just a vibe. Two people wrote it: C. Perkins from the University of Glasgow and T. Schierl from Fraunhofer HHI. The only relationship listed in the title block is that it updates RFC 3550 ✨. Its ISSN is 2070-1721 🧚.
Abstract 🦋
Okay so this memo explains how RTP sessions get synchronised and how fast that can happen ⏱️. Most RTP sessions can sync up immediately, which is great 💖. BUT video switching multipoint conference units (MCUs) or large source-specific multicast (SSM) groups can make the synchronisation delay way longer 😭. That extra wait can be unacceptable for apps that use layered and/or multi-description codecs 💀.
The memo brings in three mechanisms to shrink that delay 💅. First, the RTP Control Protocol (RTCP) timing rules get updated so SSM sessions don't have to wait as long at the start 🌈. Second, there's a new feedback packet for the extended RTP profile for RTCP-based feedback (RTP/AVPF), so video switching MCUs can quickly ask for resynchronisation 🙌. Finally, new RTP header extensions let late joiners sync rapidly and make sure timestamp-based decoding order recovery stays correct for layered codecs even when clock skew shows up 🔥.
Status of This Memo 🌸
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF) 🧡. It represents the consensus of the IETF community, received public review, and was approved for publication by the Internet Engineering Steering Group (IESG) ✅. For more on Internet Standards, check Section 2 of RFC 5741 📚.
For the current status of this document, any errata, and how to give feedback on it, see http://www.rfc-editor.org/info/rfc6051 🔗.
Copyright Notice 💜
Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication of this document. Please read these documents carefully, since they explain your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License.
Table of Contents 🗂️
1. Introduction ....................................................3
2. Synchronisation of RTP Flows ....................................4
2.1. Initial Synchronisation Delay ..............................5
2.1.1. Unicast Sessions ....................................5
2.1.2. Source-Specific Multicast (SSM) Sessions ............6
2.1.3. Any-Source Multicast (ASM) Sessions .................7
2.1.4. Discussion ..........................................8
2.2. Synchronisation for Late Joiners ...........................9
3. Reducing RTP Synchronisation Delays ............................10
3.1. Reduced Initial RTCP Interval for SSM Senders .............10
3.2. Rapid Resynchronisation Request ...........................10
3.3. In-Band Delivery of Synchronisation Metadata ..............11
4. Application to Decoding Order Recovery in Layered Codecs .......14
4.1. In-Band Synchronisation for Decoding Order Recovery .......14
4.2. Timestamp-Based Decoding Order Recovery ...................15
4.3. Example ...................................................16
5. Security Considerations ........................................18
6. IANA Considerations ............................................19
7. Acknowledgements ...............................................19
8. References .....................................................20
8.1. Normative References ......................................20
8.2. Informative References ....................................201. Introduction 🌷
Okay so when you use RTP to deliver multimedia, you usually need to synchronise playout of the audio and video parts of a presentation 🎶🎬. This happens with info in RTCP sender report (SR) packets [RFC3550] 📦. They're sent periodically, and the parts of a multimedia session can't be synchronised until enough RTCP SR packets have arrived for each RTP flow. Only then can the receiver map the media clock used by each RTP flow to the common (NTP-format) reference clock that does the actual syncing 🧠.
Lately people have been worried that this sync delay is a problem for some apps, like ones using layered or multi-description video coding 😬. This memo goes over how RTP synchronisation works, describes the delay you can expect, and proposes three backwards compatible extensions to the basic mechanism 💖:
- The RTCP transmission timing rules get relaxed for source-specific multicast (SSM) senders, so large SSM groups don't wait as long for initial sync. See Section 3.1. 🚀
- The extended RTP profile for RTCP-based feedback (RTP/AVPF) [RFC4585] gets an upgrade so receivers can ask for extra RTCP SR packets, which carry the metadata needed to sync RTP flows. This helps when joining sessions with large RTCP reporting intervals, when there's packet loss, or when video switching MCUs are in use. See Section 3.2. 🙌
- Two RTP header extensions are defined to deliver sync metadata in-band with RTP data packets. That metadata lines up with the RTP data packets, so you don't need to estimate clock skew between flows before syncing. They can also cut down on how many RTCP SR packets you need before flows can sync, though they don't get rid of the need for RTCP entirely. See Section 3.3. 🧚
The immediate use case is cutting the sync delay when joining a layered video session, like an H.264/SVC (Scalable Video Coding) session in Non-Interleaved Timestamp-based (NI-T) mode [AVT-RTP-SVC] 🎥. That said, the extensions aren't specific to layered coding, so they work in any environment where sync latency is a problem 🌈.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] 📖.
2. Synchronisation of RTP Flows 💅
Okay, the key thing here: receivers sync RTP flows using info in RTCP SR packets that senders generate, specifically the NTP-format timestamp and the RTP timestamp ⏰. To sync, a common reference clock MUST be used to generate the NTP-format timestamps in a set of flows that are going to be synchronised. (So when you're syncing several RTP flows, each flow's RTP timestamps come from its own separate, media specific clock, but the NTP-format timestamps in the RTCP SR packets of all the flows you want to sync MUST be sampled from the same clock 🫶.) To get faster and more accurate sync, it's also RECOMMENDED that senders and receivers use a synchronised common NTP-format reference clock with common properties, especially the timebase, where possible. This isn't always doable when RTP is used outside of controlled environments, and how that common reference clock and its properties get signalled and distributed is outside the scope of this memo 🙈.
For multimedia sessions, each type of media (like audio or video) goes in its own RTP session. The receiver links up the RTP flows that should be synced using the canonical end-point identifier (CNAME) item in the RTCP Source Description (SDES) packets the sender generates, or signalled out of band [RFC5576] 🔗. For layered media, different layers can go in different RTP sessions, or use different synchronisation source (SSRC) values inside a single RTP session. Either way, the CNAME is what identifies the flows to sync 💜. To make sure sync actually happens, an RTP sender MUST therefore send periodic compound RTCP packets following Section 6 of RFC 3550 [RFC3550] 📦✨.
The timing of those periodic compound RTCP packets depends on the number of members in each RTP session, the fraction of them that are sending data, the session bandwidth, the configured RTCP bandwidth fraction, and whether the session is multicast or unicast (check RFC 3550, Section 6.2 for the full details 🤓). Basically, RTCP control traffic gets a small slice, generally 5%, of the session bandwidth. Of that slice, one quarter goes to active RTP senders and receivers use the remaining three quarters. (You can configure these fractions via the Session Description Protocol (SDP) [RFC3556] ⚙️.) Each member of an RTP session works out an RTCP reporting interval from these fractions, whether the session is multicast or unicast, how many members it has seen, and whether it's actively sending data. Then it sends a compound RTCP packet on average once per reporting interval. The actual send time is randomised in the range [0.5 ... 1.5] times the reporting interval so reports don't all bunch up together 🎲.
A minimum reporting interval of 5 seconds is RECOMMENDED, except that the delay before sending the initial report "MAY be set to half the minimum interval to allow quicker notification that the new participant is present" [RFC3550] 🙋♀️. Also, for unicast sessions, "the delay before sending the initial compound RTCP packet MAY be zero" [RFC3550] 🦋. And for unicast sessions, plus active senders in a multicast session, the fixed minimum reporting interval MAY be scaled to "360 divided by the session bandwidth in kilobits/second. This minimum is smaller than 5 seconds for bandwidths greater than 72 kb/s" [RFC3550] 🌸.
2.1. Initial Synchronisation Delay ✨
okay so a multimedia session is basically a bunch of RTP sessions hanging out with the same group of participants, with one RTP session for each media type 🎀. a videoconference is a multimedia session, and it might have an audio RTP session AND a video RTP session 💅. so every sender MUST send a compound RTCP packet to each RTP session in the multimedia session, made of an RTCP SR packet and an RTCP SDES packet with a CNAME item. this is how the receiver can line everything up 🫶. the receiver literally cannot sync playout across the whole session until those RTCP packets have arrived on ALL of the component RTP sessions 😭.
if nothing gets lost, the expected initial sync delay is the average time it takes to receive the first RTCP packet in whichever RTP session has the longest RTCP reporting interval. and it'll be different for unicast and multicast RTP sessions 🦋.
layered sessions work pretty much the same way. the layers can't be synchronised until the RTCP SR and CNAME info has been received for each layer in the session 🧚.
2.1.1. Unicast Sessions
for unicast multimedia or layered sessions, senders SHOULD send an initial compound RTCP packet (an RTCP SR packet and an RTCP SDES packet with a CNAME item) immediately when they join each RTP session in the multimedia session 🚀. an RTP session counts as joined once any in-band signalling for NAT traversal (e.g., [RFC5245]) and/or security keying (e.g., [RFC5764], [ZRTP]) has wrapped up, and the media path is open 🔓. that means the initial RTCP packet goes out in parallel with the first data packet, following the RFC 3550 guidance that "the delay before sending the initial compound RTCP packet MAY be zero" 🎀. so if no packets are lost, flows can be synced right away ✨.
it's expected that NAT pinholes, firewall holes, quality-of-service, and media security keys will have been negotiated as part of the signalling (in-band or out-of-band) before that first RTCP packet gets sent 🛡️. this should make sure any middleboxes are ready to accept traffic, and it lowers the chances that the initial RTCP packet gets lost 💖.
2.1.2. Source-Specific Multicast (SSM) Sessions
for multicast sessions, the delay before sending the initial RTCP packet (and therefore the sync delay) depends on the session bandwidth and how many members are in the session 🌈. for a multicast multimedia or layered session, the average sync delay follows the slowest of the component RTP sessions. that's usually the one with the lowest bandwidth (assuming all the RTP sessions have the same number of members) 🐢.
when sending to a multicast group, the reduced minimum RTCP reporting interval of 360 seconds divided by the session bandwidth in kilobits per second [RFC3550] should be used when sync latency is likely to be an issue ⏱️. also, as usual, the reporting interval gets halved for the first RTCP packet. depending on the session bandwidth and the number of members, this gives the average sync delays shown in Figure 1 📊.
Session| Number of receivers:
Bandwidth| 2 3 4 5 10 100 1000 10000
--+------------------------------------------------
8 kbps| 2.73 4.10 5.47 5.47 5.47 5.47 5.47 5.47
16 kbps| 2.50 2.50 2.73 2.73 2.73 2.73 2.73 2.73
32 kbps| 2.50 2.50 2.50 2.50 2.50 2.50 2.50 2.50
64 kbps| 2.50 2.50 2.50 2.50 2.50 2.50 2.50 2.50
128 kbps| 1.41 1.41 1.41 1.41 1.41 1.41 1.41 1.41
256 kbps| 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70
512 kbps| 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35
1 Mbps| 0.18 0.18 0.18 0.18 0.18 0.18 0.18 0.18
2 Mbps| 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09
4 Mbps| 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04
Figure 1: Average Initial Synchronisation Delay in Seconds
for an RTP Session with 1 Senderthese numbers assume a source-specific multicast channel with a single active sender, and an average RTCP packet size of 70 octets 📦. these intervals are good enough for lip-synchronisation without excessive delay, but they might feel like too much latency for syncing parts of a layered video stream 👀.
the RTCP interval is randomised in the usual way, so the minimum sync delay is half these intervals, and the maximum delay is 1.5 times these intervals 🎲. also, these RTCP intervals are calculated assuming perfect knowledge of the number of members in the session 🧠.
2.1.3. Any-Source Multicast (ASM) Sessions
for ASM sessions, the fraction of members who are senders plays a big role, and it causes more variation in the average RTCP reporting interval 🔀. Figure 2 and Figure 3 show this. they use the same session bandwidths and receiver populations as the SSM session in Figure 1, but for sessions with 2 and 10 senders, respectively 📈. the initial sync delay scales with the number of senders (this is so the total RTCP traffic from all group members doesn't grow without bound 🙅♀️), and it can be way bigger than for source-specific groups 😬. even so, the initial sync time is still acceptable for lip-synchronisation in typical small-to-medium video conferencing scenarios 💜.
note that multi-sender groups built with multi-unicast through a central RTP translator (Topo-Translator in the terminology of [RFC5117]) or mixer (Topo-Mixer), or some forms of video switching MCU (Topo-Video-switch-MCU), send RTCP packets to ALL members of the group. so they scale the same way as an ASM group does for initial sync latency 🫧.
Session| Number of receivers:
Bandwidth| 2 3 4 5 10 100 1000 10000
--+------------------------------------------------
8 kbps| 2.73 4.10 5.47 6.84 10.94 10.94 10.94 10.94
16 kbps| 2.50 2.50 2.73 3.42 5.47 5.47 5.47 5.47
32 kbps| 2.50 2.50 2.50 2.50 2.73 2.73 2.73 2.73
64 kbps| 2.50 2.50 2.50 2.50 2.50 2.50 2.50 2.50
128 kbps| 1.41 1.41 1.41 1.41 1.41 1.41 1.41 1.41
256 kbps| 0.70 0.70 0.70 0.70 0.70 0.70 0.70 0.70
512 kbps| 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.35
1 Mbps| 0.18 0.18 0.18 0.18 0.18 0.18 0.18 0.18
2 Mbps| 0.09 0.09 0.09 0.09 0.09 0.09 0.09 0.09
4 Mbps| 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04
Figure 2: Average Initial Synchronisation Delay in Seconds
for an RTP Session with 2 Senders
Session| Number of receivers:
Bandwidth| 2 3 4 5 10 100 1000 10000
--+------------------------------------------------
8 kbps| 2.73 4.10 5.47 6.84 13.67 54.69 54.69 54.69
16 kbps| 2.50 2.50 2.73 3.42 6.84 27.34 27.34 27.34
32 kbps| 2.50 2.50 2.50 2.50 3.42 13.67 13.67 13.67
64 kbps| 2.50 2.50 2.50 2.50 2.50 6.84 6.84 6.84
128 kbps| 1.41 1.41 1.41 1.41 1.41 3.42 3.42 3.42
256 kbps| 0.70 0.70 0.70 0.70 0.70 1.71 1.71 1.71
512 kbps| 0.35 0.35 0.35 0.35 0.35 0.85 0.85 0.85
1 Mbps| 0.18 0.18 0.18 0.18 0.18 0.43 0.43 0.43
2 Mbps| 0.09 0.09 0.09 0.09 0.09 0.21 0.21 0.21
4 Mbps| 0.04 0.04 0.04 0.04 0.04 0.11 0.11 0.11
Figure 3: Average Initial Synchronisation Delay in Seconds
for an RTP Session with 10 Senders2.1.4. Discussion
for unicast sessions, the existing RTCP SR-based mechanism allows immediate synchronisation, as long as the initial RTCP packet isn't lost 🌟.
for SSM sessions, the initial sync delay is fine for lip-synchronisation, but it might be longer than we'd like for some layered codecs 🎞️. the reason multicast groups don't send immediate RTCP packets is to avoid an implosion of requests when a huge number of members join the group at once (the "flash crowd" 🌊). that's not a problem for SSM senders, because there can be at most ONE sender. so it's desirable to let SSM senders send an immediate RTCP SR on joining a session (just like unicast sessions already can, and those don't suffer from the implosion problem either) 🙌. SSM receivers using unicast feedback would not be allowed to send immediate RTCP. for ASM sessions, implosion of responses is a real concern, so no change is proposed to the RTCP timing rules 🚫.
in all cases, the initial RTCP SR packet might get lost. if that happens, the receiver won't be able to synchronise the media until the reporting interval has passed and the next RTCP SR packet is sent. that's not ideal 😩. Section 3.2 defines a new RTP/AVPF transport layer feedback message that requests an RTCP SR be generated, so there's rapid resynchronisation when packets are lost 🔁.
2.2. Synchronisation for Late Joiners
synchronisation between RTP sessions can be slower for late joiners than for people who were there from the start of the session 🕰️. there are three reasons for this:
- Many of the optimisations that allow rapid transmission of RTCP SR packets apply only at the start of a session. this means a new participant may have to wait a complete RTCP reporting interval for each session before getting the data it needs to sync its media streams. depending on the configured session bandwidth and the number of participants, that could take several seconds 🐌.
- extra sync delay comes from the nature of the RTCP timing rules. packets are generated on average once per reporting interval, but the exact transmission times are randomised +/- 50% to avoid synchronisation of reports. this matters for avoiding network congestion in multicast sessions, but it also means the timing of RTCP sender reports for different RTP sessions isn't synchronised. so a receiver has to estimate the skew on the NTP-format clock in order to align RTP timestamps across sessions. this estimation is an essential part of an RTP synchronisation implementation, and it can be done with high accuracy given enough reports. collecting enough RTCP SR data to do that estimation, though, may need several RTCP reports, which adds even more sync delay 🧮.
- lots of media codecs have periodic access points, so a newly joined receiver often can't start decoding a media stream until it has received the packets for an access point. these access points may be sent less often than RTCP SR packets, so they may be the thing that limits when synchronised playout can start for late joiners 🚪. the RTP extension for unicast-based rapid acquisition of multicast RTP sessions [AVT-ACQUISITION-RTP] may be used to cut down the time it takes to receive access points in some scenarios ⚡.
these delays are likely to be an issue when tuning in to an ongoing multicast RTP session, or for video switching MCUs 🚨
3. Reducing RTP Synchronisation Delays ✨
okay bestie, this is the glow-up section 💅 Three backwards compatible RTP extensions are defined to shrink the possible synchronisation delay: a reduced initial RTCP interval for SSM senders, a rapid resynchronisation request message, and RTP header extensions that can carry synchronisation metadata in-band 🦋
3.1. Reduced Initial RTCP Interval for SSM Senders
so in SSM sessions where the initial synchronisation delay really matters 😭, the RTP sender MAY set the delay before its initial compound RTCP packet to zero, and send its first RTCP packet right away when it joins the SSM session 🚀 this is purely a local change on the sender's side, so it can just be a configurable option ✨ RTP receivers in an SSM session that send unicast RTCP feedback MUST NOT send RTCP packets with zero initial delay 🙅♀️ the timing rules defined in [RFC5760] apply unchanged to receivers, so they stay the same.
3.2. Rapid Resynchronisation Request
the general shape of an RTP/AVPF transport layer feedback message is in Figure 4 (check [RFC4585] for the deets) 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P| FMT | PT=RTPFB=205 | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC of packet sender |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC of media source |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: Feedback Control Information (FCI) :
: :Figure 4: RTP/AVPF Transport Layer Feedback Message
one new feedback message type, RTCP-SR-REQ, is defined with FMT = 5 🌸 the Feedback Control Information (FCI) part of the feedback message MUST be empty 🫶 the SSRC of the packet sender tells you which member can't synchronise media streams, and the SSRC of the media source tells you which sender of that media it's stuck on. the length MUST equal 2 💖
if the RTP/AVPF profile [RFC4585] is in use, a receiver MAY send this feedback message to say "hey, I can't synchronise some of my media streams, can you send an RTCP SR packet asap?" 🙏 (asap means within the RTCP timing rules for early feedback, obvi). when a media source that gets RTCP-SR-REQ SHOULD generate an RTCP SR packet as soon as possible while following the RTCP early feedback rules 🔥 if non-compound RTCP [RFC5506] was negotiated earlier, both the feedback request and the RTCP SR response may be sent as non-compound RTCP packets 🧚 the RTCP-SR-REQ packet MAY be repeated once per RTCP reporting interval if no RTCP SR packet is forthcoming. the media source may ignore RTCP-SR-REQ packets if its regular schedule for sending synchronisation metadata can be expected to let the receiver sync the streams within a reasonable time frame 🙈
when you're using SSM sessions with unicast feedback, the feedback target and the media source might not be in the same place 🗺️ if a feedback target gets an RTCP-SR-REQ in that case, the request should be forwarded to the media source. how to forward those requests is not defined here, so that's a later problem 💅
if the feedback target has a network management interface, it might be useful to keep a log of which receivers send RTCP-SR-REQ feedback packets and which ones don't, because the ones that don't will see slower stream synchronisation 👀
3.3. In-Band Delivery of Synchronisation Metadata
the RTP header extension mechanism defined in [RFC5285] can be adapted to carry an OPTIONAL NTP-format timestamp in RTP data packets ✨ if one is included, it MUST correspond to the same time instant as the RTP timestamp in the packet's header, and it MUST be derived from the same clock used to generate the NTP-format timestamps in RTCP SR packets ⏰ the receiver can use it as input to the synchronisation algorithm, exactly as if an extra RTCP SR packet had arrived for the flow, as long as it knows the SSRC to CNAME mapping (from an earlier RTCP CNAME packet or out-of-band signalling [RFC5576]) 🧠
two variants are defined for this header extension 🎀 the first one extends the RTP header with a 64-bit NTP-format timestamp as defined in [RFC5905]. the second one carries the lower 24-bit part of the Seconds of an NTP-format timestamp and the 32 bits of the Fraction of an NTP-format timestamp. the formats of the two variants are shown in Figure 5 and Figure 6 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|1| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+R
| timestamp |T
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+P
| synchronisation source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| 0xBE | 0xDE | length=3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+E
| ID-A | L=7 | NTP timestamp format - Seconds (bit 0-23) |x
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+t
|NTP Sec.(24-31)| NTP timestamp format - Fraction (bit 0-23) |n
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|NTP Frc.(24-31)| 0 (pad) | 0 (pad) | 0 (pad) |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| payload data |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5: Variant A/64-Bit NTP RTP Header Extension
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|1| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+R
| timestamp |T
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+P
| synchronisation source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| 0xBE | 0xDE | length=2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+E
| ID-B | L=6 | NTP timestamp format - Seconds (bit 8-31) |x
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+t
| NTP timestamp format - Fraction (bit 0-31) |n
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| payload data |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: Variant B/56-Bit NTP RTP Header Extension
an NTP-format timestamp MAY be included in any RTP packets the sender picks 🍓 but it is RECOMMENDED when doing timestamp-based decoding order recovery for layered codecs sent across multiple RTP flows, as further specified in Section 4.1 💜 this header extension SHOULD also be sent in the RTP packets for a video random access point, and in the matching audio packets, so late joiners sync up fast in multimedia sessions and in video switching scenarios 🌈
Note: The inclusion of an RTP header extension will reduce the efficiency of RTP header compression, if it is used. Furthermore, middleboxes that do not understand the header extensions may remove them or may not update the content according to this memo.
in all cases, whether or not in-band NTP-format timestamps are included, regular RTCP SR packets MUST be sent 🛡️ this keeps things backwards compatible with receivers that sync RTP flows according to [RFC3550], and keeps things robust when middleboxes (RTP translators) strip RTP header extensions. if the Variant B/56-bit NTP RTP header extension is used, RTCP sender reports MUST be used to derive the upper 8 bits of the Seconds for the NTP-format timestamp 🔢
when SDP is used, the RTP header extensions above MUST be indicated as specified in [RFC5285], so the following URIs MUST be used:
- The URI used for signalling the use of Variant A/64-bit NTP RTP header extension in SDP is
urn:ietf:params:rtp-hdrext:ntp-64. - The URI used for signalling the use of Variant B/56-bit NTP RTP header extension in SDP is
urn:ietf:params:rtp-hdrext:ntp-56.
these header extensions can seriously glow up the user experience in IPTV channel surfing and in some interactive video conferencing scenarios 📺💅 network management tools that try to monitor the user experience may want to log which sessions signal and use these extensions 👀
4. Application to Decoding Order Recovery in Layered Codecs ✨
packets in RTP flows are often predictively coded, so a receiver has to arrange the packets into a particular order before it can decode the media data 🧩 depending on the payload format, the decoding order might be explicitly specified as a field in the RTP payload header, or the receiver might decode the packets in order of their RTP timestamps ⏱️ if a layered encoding is used, where the media data is split across several RTP flows, then it's often necessary to exactly synchronise the RTP flows comprising the different layers before layers other than the base layer can be decoded 🍰 examples of layered encodings are H.264 SVC in NI-T mode [AVT-RTP-SVC] and MPEG surround multi-channel audio [RFC5691] 🎶 as described in Section 2, this synchronisation is possible in RTP, but it can be hard to do rapidly 😤 below, we describe how the extensions defined in Section 3.3 can be used to synchronise layered flows and provide a common timestamp-based decoding order 💖
4.1. In-Band Synchronisation for Decoding Order Recovery
Okay so when a layered, multi-description, or multi-view codec is in use and each piece of the media rides on its own RTP flow, the RTP sender SHOULD keep slipping sync metadata into the stream on a regular basis 💖 That way receivers can line all the pieces up quickly and accurately ✨ There are three parts to this, bestie:
- The sender must negotiate the use of the RTP header extensions described in Section 3.3, and must periodically and synchronously insert such header extensions into all the RTP flows forming the separate components of the layered, multi-description, or multi-view flow. 🫶
- Synchronous insertion means the sender puts these header extensions into packets that correspond to exactly the same sampling instant in every flow. Since the header extensions for each flow go in at that same instant, they end up with identical NTP-format timestamps, so receivers can line up the RTP timestamps for the component flows exactly 🎀 This might mean adding extra data packets to some of the component RTP flows, if some of them have packets for sampling instants that the others don't have. A layered video codec where the layers run at different frame rates is a good example 🦋
- How often the sender inserts the header extensions directly sets the synchronisation latency. More frequent insertion means higher per-flow overhead but lower synchronisation latency, so it's a trade-off 🍓 It is RECOMMENDED that the sender insert the header extensions synchronously into all component RTP flows at least once per random access point of the media, but they MAY be inserted more often.
The sender MUST continue to send periodic RTCP reports, including SR packets, and MUST ensure that the RTP timestamp to NTP-format timestamp mapping in the RTCP SR packets is consistent with the one used in the RTP header extensions 🌸 Receivers should use both the information in the RTCP SR packets and the in-band mapping of RTP and NTP-format timestamps as input to the synchronisation process. Still, it is RECOMMENDED that receivers sanity check the mappings they receive and discard outliers, so they're robust against invalid data. 🙈 You might think the RTCP SR mappings are the ones more likely to be wrong, since they're sent at irregular times and subject to skew. But broken RTP translators could also corrupt the timestamps in the RTP header extensions, so receivers need to be ready for both kinds of failure 💀
4.2. Timestamp-Based Decoding Order Recovery
Once a receiver has synchronised the components of a layered, multi-description, or multi-view flow using the RTP header extensions as described in Section 4.1, it may then derive a decoding order from the synchronised timestamps, like so (or it may use information in the RTP payload header to derive the decoding order, if that's present and wanted) ✨
There may be explicit dependencies between the component flows of a layered, multi-description, or multi-view flow. For example, layered flows are commonly arranged in a hierarchy, where flows from "higher" layers can't be decoded until the matching data in "lower" layer flows has been received and decoded 👑 If a decoding hierarchy like this exists, it MUST be signalled out of band, for example using [RFC5583] when SDP signalling is used.
Each component RTP flow MUST contain packets corresponding to all the sampling instants of the RTP flows it depends on. If those packets aren't naturally present in the RTP flow, the sender MUST generate additional packets as necessary to satisfy this rule. The format of these packets depends on the payload format in use. For H.264 SVC, the Empty Network Abstraction Layer (NAL) unit packet [AVT-RTP-SVC] should be used. Flows may also include packets for additional sampling instants that aren't present in the flows they depend on 🧚
The receiver should decode the packets in all the component RTP flows like this:
- For each RTP packet in each flow, use the mapping contained in the RTP header extensions and RTCP SR packets to derive the NTP-format timestamp that corresponds to its RTP timestamp.
- Group together RTP data packets from all component flows that have identical calculated NTP-format timestamps.
- Process the groups in order of ascending NTP-format timestamps, and decode the RTP packets in each group according to the signalled RTP flow decoding hierarchy. In other words, pass the RTP packet data from the flow that all other flows depend on to the decoder first, then the data from the next dependent flow, and so on. The decoding order of the RTP flow hierarchy may be indicated by mechanisms defined in [RFC5583] or by some other means.
Heads up, this is important 🔥 The decoding order won't necessarily match the packet transmission order. The receiver will need to buffer packets for a codec-dependent amount of time so that all the necessary packets can arrive before decoding.
4.3. Example
The example in Figure 7 is about three RTP flows, A, B, and C, which contain a layered, a multi-view, or a multi-description media stream 🦋 In this example, the dependency signalling defined in [RFC5583] says flow A is the lowest RTP flow. Flow B is the next one up and depends on A. Flow C is the highest of the three and depends on both A and B. The media coding structure produces video access units (that is, coded video frames) that show up in higher flows but not in all the lower ones. Flow A has the lowest frame rate. Flows B and C share the same frame rate, which is higher than Flow A's. The figure shows the full video access units with their RTP timestamps "(x)". The video access units have already been re-ordered by RTP sequence number. Within each RTP flow, the figure shows the received video access unit parts in decoding order, along with the NTP media timestamps ("TS[..]"). As the figure shows, these timestamps can be derived from the NTP-format timestamp in the RTCP sender reports, as indicated by the timestamp in "{x}", or straight from the NTP timestamp in the RTP header extensions, as indicated by the timestamp in "<x>". Notice the timestamps aren't in increasing order, because in this example the decoding order differs from the output/presentation order 💅
The decoding order recovery process first moves ahead to the video access unit parts associated with the first available synchronous insertion of the NTP timestamp into the RTP header extensions, at NTP media timestamp TS=[8]. The receiver starts in the highest RTP flow, C, and removes or ignores all the preceding video access unit parts (in decoding order) up to the video access unit parts with TS=[8] in each of the de-jittering buffers of RTP flows A, B, and C. Then, starting from flow C, the first media timestamp available in decoding order (TS=[8]) is selected. The video access unit parts from RTP flows A, B, and C are placed in order of the RTP flow dependency, as indicated by mechanisms defined in [RFC5583]. (In the example for TS[8], that means flow B first and then flow C, into the video access unit AU(TS[8]) associated with NTP media timestamp TS=[8].) Next, the media timestamp TS=[6] (RTP timestamp=(4)) is processed in order of appearance in the highest RTP flow, C, and the process described above is repeated. Note that some video access units may have no parts present at all, for example in the lowest RTP flow, A (see, for example, TS=[5]) 🌈 The decoding order recovery process could also be started after an RTP sender report containing the mapping between the RTP timestamp and the NTP-format timestamp (indicated as timestamps "(x){y}") has been received, assuming there's no clock skew in the source used for NTP-format timestamp generation.
C:-(0)----(2)----(7)<8>--(5)----(4)----(6)-----(11)----(9){10}-
| | | | | | | |
B:-(3)----(5)---(10)<8>--(8)----(7)----(9){7}--(14)----(12)----
| | | |
A:---------------(3)<8>--(1)-------------------(7){12}-(5)-----
---------------------------------------decoding/transmission order->
TS:[1] [3] [8]=<8> [6] [5] [7] [12] [10]Key:
A, B, C - RTP flows
Integer values in "()" - video access unit with its RTP timestamp as
indicated in its RTP packet.
"|" - indicates the corresponding parts of the
same video access unit AU(TS[..]) in the
RTP flows.
Integer values in "[]" - NTP media timestamp TS, sampling time
as derived from the NTP timestamp
associated with the video access unit
AU(TS[..]), consisting of video access unit
parts in the flows above.
Integer values in "<>" - NTP media timestamp TS as directly
taken from the NTP RTP header extensions.
Integer values in "{}" - NTP media timestamp TS as provided in the
RTCP sender reports.
Figure 7: Example of a Layered RTP Stream5. Security Considerations 🔒
The security considerations of the RTP specification [RFC3550], the extended RTP profile for RTCP-based feedback [RFC4585], and the general mechanism for RTP header extensions [RFC5285] all apply.
The RTP header extensions defined in Section 3.3 include an NTP-format timestamp. When an RTP session using this header extension is protected by the Secure RTP (SRTP) framework [RFC3711], that header extension is not part of the encrypted portion of the RTP data packets or RTCP control packets. However, when SRTP is used without this header extension, these NTP-format timestamps are encrypted. This is a minor information leak, but one that is not believed to be significant. Including this header extension will also reduce the efficiency of RTP header compression, if it's in use. And middleboxes that don't understand the header extensions may remove them, or may not update their content according to this memo. 🙈
Middleboxes that don't speak these header extensions can strip them or leave them stale, so don't assume the metadata survived the trip.
6. IANA Considerations 📝
IANA has registered one new value in the table of FMT Values for RTPFB Payload Types [RFC4585], as follows:
Name: RTCP-SR-REQ
Long name: RTCP Rapid Resynchronisation Request
Value: 5
Reference: RFC 6051IANA has also registered two new RTP Compact Header Extensions [RFC5285], as follows:
Extension URI: urn:ietf:params:rtp-hdrext:ntp-64
Description: Synchronisation metadata: 64-bit timestamp format
Contact: Thomas Schierl <ts@thomas-schierl.de>
IETF Audio/Video Transport Working Group
Reference: RFC 6051
Extension URI: urn:ietf:params:rtp-hdrext:ntp-56
Description: Synchronisation metadata: 56-bit timestamp format
Contact: Thomas Schierl <ts@thomas-schierl.de>
IETF Audio/Video Transport Working Group
Reference: RFC 60517. Acknowledgements ✨
Big thank-you to the whole crew from the IETF AVT working group 🫶 Jonathan Lennox, Magnus Westerlund, Randell Jesup, Gerard Babonneau, Ingemar Johansson, Ali C. Begen, Ye-Kui Wang, Roni Even, Michael Dolan, Art Allison, and Stefan Doehla all talked this through with the authors, and it's giving real collaboration energy 💜 The RTP header extension format of Variant A in Section 3.3 was suggested by Dave Singer, matching a similar mechanism from the Internet Streaming Media Alliance (ISMA) 🎀
8. References 📚
8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July 2006.
[RFC5285] Singer, D. and H. Desineni, "A General Mechanism for RTP Header Extensions", RFC 5285, July 2008.
[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences", RFC 5506, April 2009.
[RFC5583] Schierl, T. and S. Wenger, "Signaling Media Decoding Dependency in the Session Description Protocol (SDP)", RFC 5583, July 2009.
[RFC5760] Ott, J., Chesterfield, J., and E. Schooler, "RTP Control Protocol (RTCP) Extensions for Single-Source Multicast Sessions with Unicast Feedback", RFC 5760, February 2010.
[RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, June 2010.
8.2. Informative References
[AVT-ACQUISITION-RTP] Ver Steeg, B., Begen, A., Van Caenegem, T., and Z. Vax, "Unicast-Based Rapid Acquisition of Multicast RTP Sessions", Work in Progress, October 2010.
[AVT-RTP-SVC] Wenger, S., Wang, Y., Schierl, T., and A. Eleftheriadis, "RTP Payload Format for SVC Video Coding", Work in Progress, October 2010.
[RFC3556] Casner, S., "Session Description Protocol (SDP) Bandwidth Modifiers for RTP Control Protocol (RTCP) Bandwidth", RFC 3556, July 2003.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004.
[RFC5117] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, January 2008.
[RFC5245] Rosenberg, J., "Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols", RFC 5245, April 2010.
[RFC5576] Lennox, J., Ott, J., and T. Schierl, "Source-Specific Media Attributes in the Session Description Protocol (SDP)", RFC 5576, June 2009.
[RFC5691] de Bont, F., Doehla, S., Schmidt, M., and R. Sperschneider, "RTP Payload Format for Elementary Streams with MPEG Surround Multi-Channel Audio", RFC 5691, October 2009.
[RFC5764] McGrew, D. and E. Rescorla, "Datagram Transport Layer Security (DTLS) Extension to Establish Keys for the Secure Real-time Transport Protocol (SRTP)", RFC 5764, May 2010.
[ZRTP] Zimmermann, P., Johnston, A., Ed., and J. Callas, "ZRTP: Media Path Key Agreement for Unicast Secure RTP", Work in Progress, June 2010.
Authors' Addresses 💌
Colin Perkins University of Glasgow School of Computing Science Glasgow G12 8QQ UK
EMail: csp@csperkins.org
Thomas Schierl Fraunhofer HHI Einsteinufer 37 D-10587 Berlin Germany
Phone: +49-30-31002-227 EMail: ts@thomas-schierl.de
📦 where this page comes from
- ietf: rfc6051 2010-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms