RFC 6465
A Real-time Transport Protocol (RTP) Header Extension for Mixer-to-Client Audio Level Indication
RFC 6465 bestie edition 💖🎀
Okay so you know how group calls can be a total mess? 😭 This RFC is basically the IETF saying "what if the mixer could tell everyone who's actually talking, how loud they are, and who's just breathing into their mic" 🫶 It's a standards track doc from December 2011, and it's ISSN 2070-1721. The editors are E. Ivov, Ed. (Jitsi), E. Marocco, Ed. (Telecom Italia), and J. Lennox (Vidyo). The title block doesn't list anything it obsoletes or updates, so it's standing on its own. ✨
Abstract
okay so this doc describes a way for RTP-level mixers in audio conferences to send info about how loud each participant is 🔊 The audio level indicators ride in the same RTP packets as the audio they describe, so it's all bundled together 🎀
Status of This Memo
This is an Internet Standards Track document. 💅
It's a product of the Internet Engineering Task Force (IETF), so it has the consensus of the IETF community. It went through public review and got approved by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check Section 2 of RFC 5741 🍓
For the current status, any errata, and how to give feedback, look at http://www.rfc-editor.org/info/rfc6465 🦋
Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date. Please actually read these, because they spell out your rights and restrictions for this doc 👀 Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License. 💜
Table of Contents
1. Introduction ....................................................2
2. Terminology .....................................................4
3. Protocol Operation ..............................................4
4. Audio Levels ....................................................5
5. Signaling Information ...........................................7
6. Security Considerations .........................................9
7. IANA Considerations ............................................10
8. Acknowledgments ................................................10
9. References .....................................................10
9.1. Normative References ......................................10
9.2. Informative References ....................................11
Appendix A. Reference Implementation ..............................12
A.1. AudioLevelCalculator.java .................................121. Introduction ✨
"A Framework for Conferencing with the Session Initiation Protocol (SIP)" [RFC4353] lays out the overall architecture for multi-party conferencing 🎀 It borrows from RTP [RFC3550] and extends the idea of a mixer, which is "responsible for combining the media streams that make up a conference, and generating one or more output streams that are delivered to recipients". So every participant gets, in one flat stream, the media from everyone else. 🌸
Centralized mixer setups are great for clients, because a conference call ends up feeling basically like a one-to-one chat 💅 BUT there are limitations. Since the mixer's output is one flat stream, it's really hard for users to tell who the original source of what they're hearing is 😭
Mechanisms that let the mixer tell participants who's speaking, like the contributing source (CSRC) fields in RTP [RFC3550], only work for speaking/silent yes-or-no signals 🙈 There are plenty of situations where you'd want more detail. Think background chat, noise, music, or typing, or someone breathing loudly into their mic 💀 Knowing the source would make it easy to fix, like sending a private IM to the person asking them to mute. Another one is a heated discussion between people you don't personally know. Audio level info would help you recognize speakers by things like loudness and speed, and those are still stuff a human can read ✨
One user-friendly way to show this would be for a conferencing client to put audio level indicators next to each participant in the UI. Figure 1 shows the vibe: you can tell Alice is the active speaker, Carol is muted, and Bob and Dave are making some background noise 🎧
________________________
| |
| 00:42 | Weekly Call |
|________________________|
| |
| |
| Alice |====== | (S) |
| |
| Bob |= | |
| |
| Carol | | (M) |
| |
| Dave |=== | |
| |
|________________________|
Figure 1: Displaying Detailed Speaker Information to the User by
Including Audio Level for Every ParticipantBuilding a UI like that means analyzing the media coming from the other participants. In a normal audio conference, only the mixer can really do that, because everyone else gets a single flat audio stream and has no immediate way to figure out individual levels 😤
This document specifies an RTP extension header that lets mixers send audio level info to participants. It goes right inside the RTP packets carrying the matching audio data 🦋
Quick note: the header extension here is different from, but works alongside, the one in [RFC6464]. That one lets clients tell audio mixers how loud the audio in their outgoing packets is 🍓
2. Terminology 🧚
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 📚
3. Protocol Operation 🌈
Per RFC 3550 [RFC3550], a mixer is expected to put a list of identifiers (CSRC IDs) into its outgoing RTP packets. Those identify the sources that contributed to the stream. The CSRC IDs let RTP clients figure out, in a yes-or-no way, who's the active speaker at any given moment 🎤 The RTP Control Protocol (RTCP) also has a basic way to map CSRC IDs to user identities, through the CNAME field. There can be more advanced mechanisms too, depending on the signaling protocol used to set up and control the conference. For example, with the Session Initiation Protocol [RFC3261], "A Session Initiation Protocol (SIP) Event Package for Conference State" [RFC4575] defines a <src-id> tag that binds CSRC IDs to media streams and SIP URIs 💜
This document describes an RTP header extension that lets mixers report the audio level of every contributing conference participant (CSRC), not just whether they're on or off 🔥 The new header extension uses the general mechanism for RTP header extensions described in [RFC5285].
Each instance of this header holds a list of one-octet audio levels expressed in -dBov. Values from 0 to 127 stand for 0 to -127 dBov (see Figures 2 and 3). Appendix A has a reference implementation showing one way to get these values from raw audio samples 🫶
Every audio level value goes with the CSRC identifier in the matching position in the CSRC list. So the first value is the audio level of the participant represented by the first CSRC identifier in that packet, and so on down the line 🌸 The number and order of these values MUST therefore match the number and order of the CSRC IDs in the same packet. 🙌
When encoding audio level info, a mixer SHOULD include information in a packet that matches the audio data carried in that same packet. It's important that these values follow the actual stream as closely as possible. So a mixer SHOULD also calculate the values after the original contributing stream has gone through any processing, like level normalization or noise reduction, for example. 👑
Sometimes a conference has more than one mixer. In that case, each mixer MAY choose to relay the CSRC list and audio level info it gets from peer mixers, as long as the total CSRC count stays below 16 ✨ The maximum audio level isn't precisely defined by this spec, so it's likely that average audio levels will be noticeably different for participants behind different mixers. That's a whole thing, bestie 😭
4. Audio Levels 🎀
okay so the audio level header extension is how a packet tells you how loud the audio in its RTP payload is 🔊 it rides along as an RTP header extension element, the same way [RFC5285] defines it ✨
The payload of this extension element can use either the one-byte or two-byte header from [RFC5285]. Figures 2 and 3 show sample audio level encodings for each header format 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID | len=2 |0| level 1 |0| level 2 |0| level 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2: Sample Audio Level Encoding Using the One-Byte Header Format
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID | len=3 |0| level 1 |0| level 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| level 3 | 0 (pad) | ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3: Sample Audio Level Encoding Using the Two-Byte Header Format
so for the one-byte header, the 4-bit len field is the number of data bytes (the audio level values) in that element, minus one 🧠 which means a len of zero means one byte of data follows. for the two-byte header, the 8-bit len field is the exact number of audio levels in the extension 🙌
here's the bit that matters 👀 RFC 3550 [RFC3550] only lets an RTP packet carry up to 15 CSRC IDs. audio levels refer directly to CSRC IDs, so implementations MUST NOT include more than 15 audio level values 🛑 that means the max len is 14 for the one-byte header and 15 for the two-byte header 💜
Note: Audio levels in this document are defined in the same manner as audio noise level in the RTP Payload Comfort Noise specification [RFC3389]. In [RFC3389], the overall noise level in comfort noise goes in the first byte of the payload, and the spectral info about the noise goes in the bytes after it. This specification's audio level parameter is defined to be identical to that comfort noise payload's noise-level byte. 🫶
the magnitude of the audio level lives in the seven least significant bits of the one byte in the header extension, as shown in Figures 2 and 3 ✨ the least significant bit of the magnitude goes in the least significant bit of that byte. the most significant bit of the byte is unused and always set to 0 🌸
the audio level is written in -dBov, with values from 0 to 127 standing for 0 to -127 dBov. dBov is the level in decibels relative to the overload point of the system, which is the highest-intensity signal the payload format can encode 📈 (fun fact: measuring against the overload point is super handy for digital setups, because you don't need to know how the analog circuitry was calibrated 🤓)
for example, with u-law (audio/pcmu) audio [ITU.G711], the 0 dBov reference would be a square wave with values +/- 8031. (that works out to 6.18 dBm0, relative to u-law's dBm0 definition in Table 6 of [ITU.G711] 🎀)
The audio level for digital silence (like a muted audio source, for example) MUST be represented as 127 (-127 dBov), no matter the dynamic range of the encoded audio format 🤫
the audio level header extension only carries the level of the audio in the RTP payload of its packet. there's no long-term averaging or smoothing applied 🙅♀️ the level is measured as a root mean square of all the samples in the measured range 📊
to make encoding easier, this specification includes a sample Java implementation (see Appendix A) of an audio level calculator. it helps you get these values from raw linear Pulse Code Modulation (PCM) audio samples 💖
5. Signaling Information 📣
the URI for declaring the audio level header extension in a Session Description Protocol (SDP) extmap attribute, and mapping it to a local extension header identifier, is urn:ietf:params:rtp-hdrext:csrc-audio-level 🦋 there's no extra setup info needed for this extension (so no extension attributes) 🙌
here's an example attribute line in the SDP for a conference:
a=extmap:7 urn:ietf:params:rtp-hdrext:csrc-audio-levelthe mapping will most often be given per media stream (in the media-level section(s) of SDP, so after an "m=" line), or globally if there's more than one stream carrying audio level indicators in a session 🧚
when that attribute shows up in the SDP for a media stream, it means RTP packets in that stream that carry the level extension defined here will have it with an ID of 7 💜
conferencing clients that support audio level indicators but can't mix would not be able to provide content for this extension 😭 so they'd always have to include the direction parameter in the "extmap" attribute, set to "recvonly". focus entities that can mix can leave out the direction or set it to "sendrecv" in SDP offers. those entities would need to set it to "sendonly" in SDP answers to offers with a "recvonly" parameter, and to "sendrecv" when answering other "sendrecv" offers 💅
this specification only defines the use of audio level extensions in audio streams. they MUST NOT be advertised with other media types, like video or text, for example 🚫
figures 4 and 5 show two example offer/answer exchanges: one between a conferencing client and a focus, and one between two conference focus entities 🌈
SDP Offer:
v=0
o=alice 2890844526 2890844526 IN IP6 host.example.com
s=-
c=IN IP6 host.example.com
t=0 0
m=audio 49170 RTP/AVP 0 4
a=rtpmap:0 PCMU/8000
a=rtpmap:4 G723/8000
a=extmap:1/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level
SDP Answer:
v=0
i=A Seminar on the session description protocol
o=conf-focus 2890844730 2890844730 IN IP6 focus.example.net
s=-
c=IN IP6 focus.example.net
t=0 0
m=audio 52544 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=extmap:1/sendonly urn:ietf:params:rtp-hdrext:csrc-audio-levelFigure 4: A Client-Initiated Example SDP Offer/Answer Exchange Negotiating an Audio Stream with One-Way Flow of Audio Level Information
SDP Offer:
v=0
i=Un seminaire sur le protocole de description des sessions
o=fr-focus 2890844730 2890844730 IN IP6 focus.fr.example.net
s=-
c=IN IP6 focus.fr.example.net
t=0 0
m=audio 49170 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:csrc-audio-level
SDP Answer:
v=0
i=A Seminar on the session description protocol
o=us-focus 2890844526 2890844526 IN IP6 focus.us.example.net
s=-
c=IN IP6 focus.us.example.net
t=0 0
m=audio 52544 RTP/AVP 0
a=rtpmap:0 PCMU/8000
a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:csrc-audio-levelFigure 5: An Example SDP Offer/Answer Exchange between Two Conference Focus Entities with Mixing Capabilities Negotiating an Audio Stream with Bidirectional Flow of Audio Level Information
6. Security Considerations 🔐💀
- this document defines a way to attribute audio level to a particular participant in a conference 👀 the scary part: an attacker may try to modify the content of RTP packets so audio activity from one participant appears to be coming from a different participant 😱
- and it gets worse 😭 the audio level values would not be protected, even in a Secure Real-time Transport Protocol (SRTP) session [RFC3711]. in some cases, where a particular participant's activity in a conference is confidential, that could be a real problem 🚨 and as discussed in [SRTP-VBR-AUDIO], an attacker might be able to infer information about the conversation, possibly down to phoneme-level resolution 🫣
- both of the above come from the design of the RTP protocol itself, and they would probably also apply when using CSRC identifiers the way RFC 3550 [RFC3550] specifies 🥹 so it's really important that implementors and deployers think about what their scenario needs, and consider header extension encryption [SRTP-ENCR-HDR] or a lower-level security and authentication mechanism such as IPsec [RFC4301], for example 🔑 protect your streams, bestie!!
7. IANA Considerations 📚
this document defines a new extension URI in the RTP Compact Header Extensions subregistry of the Real-Time Transport Protocol (RTP) Parameters registry, using the following data:
Extension URI: urn:ietf:params:rtp-hdrext:csrc-audio-level
Description: Mixer-to-client audio level indicators
Contact: emcho@jitsi.org
Reference: RFC 64658. Acknowledgments 🫶
Lyubomir Marinov contributed level measurement and rendering code 👑
Keith Drage, Roni Even, Miguel A. Garcia, John Elwell, Kevin P. Fleming, Ingemar Johansson, Michael Ramalho, Magnus Westerlund, and several others gave helpful feedback over the avt and avtext mailing lists 💌
Jitsi's participation in this specification is funded by the NLnet Foundation 🌷
9. References 📖
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
[RFC5285] Singer, D. and H. Desineni, "A General Mechanism for RTP Header Extensions", RFC 5285, July 2008.
9.2. Informative References ✨
okay so these are the "good to know" references 📚 they aren't required to implement the spec, but they're the stuff that gives the whole thing context 💅
[ITU.G711] International Telecommunication Union, "Pulse Code
Modulation (PCM) of Voice Frequencies",
ITU-T Recommendation G.711, November 1988.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002.
[RFC3389] Zopf, R., "Real-time Transport Protocol (RTP) Payload for
Comfort Noise (CN)", RFC 3389, September 2002.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, March 2004.
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, December 2005.
[RFC4353] Rosenberg, J., "A Framework for Conferencing with the
Session Initiation Protocol (SIP)", RFC 4353,
February 2006.
[RFC4575] Rosenberg, J., Schulzrinne, H., and O. Levin, Ed., "A
Session Initiation Protocol (SIP) Event Package for
Conference State", RFC 4575, August 2006.
[RFC6464] Lennox, J., Ed., Ivov, E., and E. Marocco, "A Real-time
Transport Protocol (RTP) Header Extension for Client-to-
Mixer Audio Level Indication", RFC 6465, December 2011.
[SRTP-ENCR-HDR]
Lennox, J., "Encryption of Header Extensions in the Secure
Real-Time Transport Protocol (SRTP)", Work in Progress,
October 2011.
[SRTP-VBR-AUDIO]
Perkins, C. and JM. Valin, "Guidelines for the use of
Variable Bit Rate Audio with Secure RTP", Work
in Progress, July 2011.Appendix A. Reference Implementation 🎀
okay this appendix is the Java reference code 🧚 it's totally not normative, and it's definitely not the only way to do it. the whole point is to help implementors add audio level support to mixers and clients, so it's giving very helpful bestie energy 🫶
the Java code has an AudioLevelCalculator class that works out the sound pressure level of a signal from specific samples 🔥 mixers can use it to make the values that go into the level extension headers ✨
and the implementation is written in Java, but it doesn't lean on anything Java-specific, so you can port it to another language pretty easily 🌈
A.1. AudioLevelCalculator.java 💜
<CODE BEGINS>
/*
Copyright (c) 2011 IETF Trust and the persons identified
as authors of the code. All rights reserved.
Redistribution and use in source and binary forms, with
or without modification, is permitted pursuant to, and subject
to the license terms contained in, the Simplified BSD License
set forth in Section 4.c of the IETF Trust's Legal Provisions
Relating to IETF Documents (http://trustee.ietf.org/license-info).
*/
/**
* Calculates the audio level of specific samples of a signal
* relative to overload.
*/
public class AudioLevelCalculator
{
/**
* Calculates the audio level of a signal with specific
* <tt>samples</tt>.
*
* @param samples the samples whose audio level we need to
* calculate. The samples are specified as an <tt>int</tt>
* array starting at <tt>offset</tt>, extending <tt>length</tt>
* number of elements, and each <tt>int</tt> element in the
* specified range representing a sample whose audio level we
* need to calculate. Though a sample is provided in the
* form of an <tt>int</tt> value, the sample size in bits
* is determined by the caller via <tt>overload</tt>.
*
* @param offset the offset in <tt>samples</tt> at which the
* samples start.
*
* @param length the length of the signal specified in
* <tt>samples<tt>, starting at <tt>offset</tt>.
*
* @param overload the overload (point) of <tt>signal</tt>.
* For example, <tt>overload</tt> can be {@link Byte#MAX_VALUE}
* for 8-bit signed samples or {@link Short#MAX_VALUE} for
* 16-bit signed samples.
*
* @return the audio level of the specified signal.
*/
public static int calculateAudioLevel(
int[] samples, int offset, int length,
int overload)
{
/*
* Calculate the root mean square (RMS) of the signal.
*/
double rms = 0;
for (; offset < length; offset++)
{
double sample = samples[offset];
sample /= overload;
rms += sample * sample;
}
rms = (length == 0) ? 0 : Math.sqrt(rms / length);
/*
* The audio level is a logarithmic measure of the
* rms level of an audio sample relative to a reference
* value and is measured in decibels.
*/
double db;
/*
* The minimum audio level permitted.
*/
final double MIN_AUDIO_LEVEL = -127;
/*
* The maximum audio level permitted.
*/
final double MAX_AUDIO_LEVEL = 0;
if (rms > 0)
{
/*
* The "zero" reference level is the overload level,
* which corresponds to 1.0 in this calculation, because
* the samples are normalized in calculating the RMS.
*/
db = 20 * Math.log10(rms);
/*
* Ensure that the calculated level is within the minimum
* and maximum range permitted.
*/
if (db < MIN_AUDIO_LEVEL)
db = MIN_AUDIO_LEVEL;
else if (db > MAX_AUDIO_LEVEL)
db = MAX_AUDIO_LEVEL;
}
else
{
db = MIN_AUDIO_LEVEL;
}
return (int)Math.round(db);
}
}
<CODE ENDS>Authors' Addresses 🌸
and finally, the people who made this happen 🫶 these are the editors and authors, with their addresses and emails, so you know exactly who to slide into the DMs 💌
Emil Ivov (editor)
Jitsi
Strasbourg 67000
France
EMail: emcho@jitsi.org
Enrico Marocco (editor)
Telecom Italia
Via G. Reiss Romoli, 274
Turin 10148
Italy
EMail: enrico.marocco@telecomitalia.it
Jonathan Lennox
Vidyo, Inc.
433 Hackensack Avenue
Seventh Floor
Hackensack, NJ 07601
US
EMail: jonathan@vidyo.comthat's the whole doc, bestie!! 🌈💖 ily for sticking with me through all three parts 😭✨
📦 where this page comes from
- ietf: rfc6465 2011-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms