RFC 6366

Requirements for an Internet Audio Codec

๐Ÿ“… published August 2011, from ietf

Requirements for an Internet Audio Codec ๐ŸŽ€

okay bestie, the title block first ๐Ÿ’… This is RFC 6366, written by J. Valin (Mozilla) and K. Vos (Skype Technologies, S.A.). It's dated August 2011 and its category is Informational ๐Ÿ“š (so it's not a Standards Track spec, just vibes and requirements). Its ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates.

Abstract

this doc gives specific requirements for an Internet audio codec โœจ the requirements cover quality, sampling rate, bit-rate, and packet-loss robustness, plus other properties that are desirable ๐Ÿฆ‹

Status of This Memo

ngl this one is not an Internet Standards Track spec ๐Ÿ™ˆ it's published for informational purposes only ๐Ÿ’–

it's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community ๐Ÿซถ it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you want the deets ๐Ÿ‘‘

for the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6366 ๐ŸŒธ

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. ๐Ÿ”ฅ

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. please read these carefully because they spell out your rights and restrictions for this doc ๐Ÿงš code components pulled out of this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License โญ

Table of Contents

   1. Introduction ....................................................2
   2. Definitions .....................................................3
   3. Applications ....................................................3
      3.1. Point-to-Point Calls .......................................3
      3.2. Conferencing ...............................................4
      3.3. Telepresence ...............................................5
      3.4. Teleoperation and Remote Software Services .................5
      3.5. In-Game Voice Chat .........................................5
      3.6. Live Distributed Music Performances / Internet
           Music Lessons ..............................................6
      3.7. Delay-Tolerant Networking or Push-to-Talk Services .........6
      3.8. Other Applications .........................................7
   4. Constraints Imposed by the Internet on the Codec ................7
   5. Detailed Basic Requirements .....................................8
      5.1. Operating Space ............................................9
      5.2. Quality and Bit-Rate .......................................9
      5.3. Packet-Loss Robustness ....................................10
      5.4. Computational Resources ...................................10
   6. Additional Considerations ......................................12
      6.1. Low-Complexity Audio Mixing ...............................12
      6.2. Encoder Side Potential for Improvement ....................12
      6.3. Layered Bit-Stream ........................................13
      6.4. Partial Redundancy ........................................13
      6.5. Stereo Support ............................................13
      6.6. Bit Error Robustness ......................................13
      6.7. Time Stretching and Shortening ............................14
      6.8. Input Robustness ..........................................14
      6.9. Support of Audio Forensics ................................14
      6.10. Legacy Compatibility .....................................14
   7. Security Considerations ........................................14
   8. Acknowledgments ................................................15
   9. Informative References .........................................15

1. Introduction โœจ

this doc gives requirements for an audio codec built specifically for use over the Internet ๐Ÿ’– the goal is to meet the needs of the most common Internet interactive audio apps and to keep quality good in conditions that are typical for the Internet ๐ŸŒˆ they also cover quality, sampling rate, delay, bit-rate, and packet-loss robustness, and other desirable codec properties get considered too ๐Ÿ’…

2. Definitions ๐ŸŽ€

all through this doc, these conventions refer to the sampling rate of a signal:

  • Narrowband: 8 kilohertz (kHz)
  • Wideband: 16 kHz
  • Super-wideband: 24/32 kHz
  • Full-band: 44.1/48 kHz

bit-rates are measured in bits per second (bit/s) and don't count any overhead like (IP/UDP/RTP) headers, padding, etc. ๐Ÿ“ the codec delay is the total algorithmic delay when you add the codec frame size to the "look-ahead" ๐Ÿฆ‹ so it's the minimum theoretically achievable end-to-end delay of a transmission system that uses the codec โญ

3. Applications ๐ŸŒธ

these apps should be considered for Internet audio codecs, along with their requirements:

  • Point-to-point calls
  • Conferencing
  • Telepresence
  • Teleoperation
  • In-game voice chat
  • Live distributed music performances / Internet music lessons
  • Delay-tolerant networking or push-to-talk services
  • Other applications

3.1. Point-to-Point Calls ๐Ÿ“ž

point-to-point calls are voice over IP (VoIP) calls from two "standard" (fixed or mobile) phones, implemented in hardware or software ๐Ÿ’œ for these, a wideband codec is required, along with narrowband support so it works with a public switched telephone network (PSTN) ๐Ÿซถ the useful bit-rate range is expected to be 12 - 32 kilobits per second (kbit/s) for wideband speech and 8 - 16 kbit/s for narrowband speech ๐Ÿ’–

the codec delay must be less than 40 milliseconds (ms), but no more than 25 ms is desirable โœจ encoding music isn't required, but it's desirable for the codec not to make background (on-hold) music excessively unpleasant to hear ๐Ÿ™ˆ and the codec should be robust to noise (producing intelligible speech and no annoying artifacts) even at lower bit-rates ๐Ÿ”ฅ

3.2. Conferencing ๐Ÿ‘‘

conferencing apps (that support multi-party calls) have extra requirements on top of the point-to-point ones ๐Ÿ’… conferencing systems often have higher-fidelity audio gear and more network bandwidth, especially when video is involved ๐ŸŒˆ so super-wideband audio becomes important, with useful bit-rates in the 32 - 64 kbit/s range ๐Ÿฆ‹

being able to vary the bit-rate based on how "difficult" the audio signal is = a desirable feature ๐Ÿ’– that doesn't just save bandwidth "on average", it also helps conference servers use bandwidth more efficiently by giving more to important audio streams and less to less important ones (like background noise) ๐Ÿงš

conferencing endpoints often run hands-free, which creates acoustic echo problems ๐Ÿ™ˆ so lower delay matters, because it cuts down the quality degradation from any residual echo left after acoustic echo cancellation (AEC) ๐Ÿ˜ญ that means the codec delay must be less than 30 ms for this app ๐Ÿ’… an optional low-delay mode with less than 10 ms delay is desirable, but not required ๐ŸŒธ

most conferencing systems use a bridge that mixes some (or all) of the audio streams and sends them back to everyone ๐Ÿซถ so the codec must not produce annoying artifacts when two voices are present at the same time ๐Ÿ’€ this mixing should also be as easy as possible to do โœจ to figure out which streams need mixing (and which are just noise or silence), it must be possible to measure (or estimate) the voice activity in a packet without fully decoding it, which saves most of the complexity when a packet doesn't need decoding ๐Ÿ“ being able to save on computational complexity while mixing is also desirable, but not required ๐Ÿฆ‹ for example, a transform codec may let you mix streams in the transform domain without going back to the time domain ๐Ÿงš low-complexity up-sampling and down-sampling inside the codec is also desirable when mixing streams with different sampling rates ๐Ÿ’œ

3.3. Telepresence ๐ŸŽฌ

most telepresence apps are basically very high-quality video-conferencing environments, so all the conferencing requirements apply here too ๐Ÿ‘‘ on top of that, telepresence needs super-wideband and full-band audio, with useful bit-rates in the 32 - 80 kbit/s range ๐ŸŒˆ voice is still the most important signal to encode, but it must be possible to get good (even if not transparent) music quality too ๐Ÿ”ฅ

most telepresence apps need more than one audio channel, so stereo and multi-channel support is important ๐ŸŽ€ you can always do that by encoding multiple single-channel streams, but it's preferable to take advantage of the redundancy that exists between channels ๐Ÿ’–

3.4. Teleoperation and Remote Software Services ๐Ÿค–

teleoperation apps are similar to telepresence, except they involve remote physical interactions ๐Ÿ˜ญ for example, a user might control a robot while getting real-time audio feedback from it ๐Ÿซถ for these apps, the delay has to be less than 10 ms ๐Ÿ’… the other telepresence requirements (quality, bit-rate, multi-channel) apply to teleoperation too ๐ŸŒธ the only exception is that mixing isn't an important issue here โœจ

remote software services have requirements similar to teleoperation ๐Ÿงš these include remote desktop apps, remote virtualization, and interactive media apps being rendered remotely (like video games rendered on central servers) ๐Ÿฆ‹ for all of these, full-band audio with an algorithmic delay below 10 ms is important ๐Ÿ”ฅ

3.5. In-Game Voice Chat ๐ŸŽฎ

more and more computer and console games use VoIP so players can talk in real time ๐Ÿ’œ gaming requirements are similar to conferencing, with the main difference that narrowband compatibility isn't necessary ๐Ÿ“ most apps are fine with a codec delay up to 30 ms, but a low-delay (< 10 ms) option is highly desirable, especially for games with rapid interactions ๐ŸŒˆ being able to use variable bit-rate (VBR) (with a maximum allowed bit-rate) is also highly desirable, because it can significantly reduce the bandwidth a game server needs ๐Ÿ‘‘ more of this in the next part, bestie ๐Ÿซถ

3.6. Live Distributed Music Performances / Internet Music Lessons โœจ

Okay bestie, live music over the Internet is literally one of the most demanding things ever ๐Ÿ˜ญ It needs extremely low end-to-end delay, so it's one of the most intense interactive audio use cases out there. Musicians can handle total end-to-end delays up to 25 ms for most setups, and the absolute limit, where none of the scenarios work, is around 50 ms [carot09] ๐ŸŽถ. To get that on the Internet, whether you're in the same city or a nearby one, you have to account for the network propagation time. Then you subtract the audio buffer, jitter buffer, and acoustic path delays, and what's left for the codec is roughly 2 ms to 10 ms ๐ŸŽ€. Light moves through fiber at a fixed speed, so every 1 ms you shave off the codec delay stretches the range where synchronization works by about 200 km ๐Ÿš€

Acoustic echo is expected to be an even bigger problem for network music than it is in conferencing ๐Ÿ™ˆ. The music quality requirements basically rule out using a "non-linear processor" (NLP) with AEC, so that's another reason why super low delay is essential ๐Ÿ’…

Since it's music, the full audio bandwidth (44.1 or 48 kHz sampling rate) has to come through at a bit-rate that gives near-transparent to transparent quality ๐ŸŽง. With current audio coding tech, that's around 64 kbit/s to 128 kbit/s per channel. Just like with telepresence, two or more channels are often wanted, so it would be useful if a codec could take advantage of the redundancy that usually exists between audio channels ๐Ÿซถ

3.7. Delay-Tolerant Networking or Push-to-Talk Services โœจ

Internet transmissions get hit by connectivity interruptions that totally wreck a phone call ๐Ÿ’€. These can come from route changes, handovers, slow fading, or device failures. To get past the distortion, the call can be paused and then picked back up once connectivity is restored ๐Ÿ“ž

Also, if the transmission capacity drops below the minimal coding rate, switching to push-to-talk mode still lets people communicate effectively ๐ŸŽ™๏ธ. In that case, voice gets sent at a slower-than-real-time bit-rate, and conversations are interrupted until the speech has been transmitted ๐Ÿข

Both of these modes mean the audio playout has to be interrupted and then continue after a pause of arbitrary duration ๐Ÿฆ‹

3.8. Other Applications โœจ

This list is by no means complete, obviously, because there are tons of other apps that use interactive audio on the Internet ๐ŸŒˆ. Still, the belief is that if all these different applications get their needs met, then the needs of unlisted applications should be met too ๐Ÿ’œ

4. Constraints Imposed by the Internet on the Codec โœจ

Packet losses are inevitable on the Internet ๐Ÿ˜ค, and dealing with them is one of the most fundamental requirements for an Internet audio codec. Any audio codec can be paired with a good packet-loss concealment (PLC) algorithm, but the real key is what happens on the first packets received after the loss. Specifically:

  • any received packet should be interpretable no matter what was lost before, as laid out in BCP 36 [PAYLOADS]; and
  • the decoder should get back in sync as fast as it can, so its output quickly lands on what it would have been if no loss had happened ๐Ÿ’ซ

Being able to decode any packet leads to these considerations for an audio codec:

  • a compressed frame has to stay smaller than the MTU so it doesn't get fragmented ๐Ÿ“ฆ
  • no parameter encoded in the bit-stream can depend on info from other packets. So a codec can't signal a mode change in one packet and then assume the next frames will be decoded in that mode. Not allowed ๐Ÿ™…

Even though parameters can't depend on other packets, it's still reasonable to use some prediction across frames, as long as the predictors can resynchronize quickly after a lost packet ๐Ÿ”ฎ. Here, the trick is finding the best compromise between better coding efficiency and losing some packet-loss robustness because of inter-frame prediction. It's a desirable property for the codec to let you control that trade-off in real time. That way it can lean on more prediction when losses are rare, but get more robust when the loss rate is high ๐Ÿง 

To make the codec tougher against packet loss, it would be great if it allowed an adaptive amount of side information (depending on the data and the network) to help audio quality when losses happen ๐Ÿ›ก๏ธ. For example, the side information could include retransmitting certain parameters that were encoded in the previous frame(s).

To keep implementation freedom, decoder-side-only error concealment doesn't need to be specified ๐Ÿ•Š๏ธ, though a working PLC algorithm is desirable as part of the codec reference implementation. Obviously, any information signaled in the bit-stream to help PLC does need to be specified.

Another big deal is that the Internet is mostly a best-effort network with no guaranteed bandwidth ๐ŸŒช๏ธ. So the codec has to be able to change its output bit-rate on the fly (in real time), without needing an out-of-band signaling mechanism, and without making audible artifacts at the bit-rate change boundaries. Extra desirable features are:

  • being able to make smooth bit-rate changes at one byte/frame resolution ๐ŸŽš๏ธ
  • letting the codec adapt its bit-rate based on the source signal being encoded (source-controlled VBR), to maximize quality for a certain average bit-rate ๐Ÿ“ˆ

The Internet moves data in bytes, so a codec should produce compressed data in whole numbers of bytes ๐Ÿ”ข. In general, the codec design should take explicit congestion notification (ECN) into account, and it may include features that would make an ECN implementation work even better โœจ

The IETF has defined a bunch of application-layer protocols for real-time transport of multimedia data, voice included ๐ŸŽค. So it's important that the codec is easy to use with these protocols. For example, it must be possible to create an [RTP] payload format that conforms to BCP 36 [PAYLOADS]. If any codec parameters need to be negotiated between endpoints, the negotiation should be as easy as possible to carry over session initiation protocol (SIP) [RFC3261]/ session description protocol (SDP) [RFC4566], or alternatively over extensible messaging and presence protocol (XMPP) [RFC6120] / Jingle [XEP-0167] ๐Ÿ’Œ

5. Detailed Basic Requirements โœจ

This section wraps up all the constraints from the target applications and the Internet into a set of actual requirements for codec development ๐Ÿ“

5.1. Operating Space โœจ

The operating space for the target applications can be split by delay. Most applications need a "medium delay" (20-30 ms), while a few need a "very low delay" (< 10 ms) โฑ๏ธ. Dividing the space by delay makes sense because lowering the delay costs you in quality versus bit-rate.

For medium delay, the resulting codec must be able to efficiently operate within these bit-rates (per channel):

  • Narrowband: 8 kbit/s to 16 kbit/s
  • Wideband: 12 to 32 kbit/s
  • Super-wideband: 24 to 64 kbit/s
  • Full-band: 32 to 80 kbit/s

A lower-delay codec that can operate in the above range is obviously also acceptable ๐Ÿ’–

For very low delay, the resulting codec will need to operate within these bit-rates (per channel):

  • Super-wideband: 32 to 80 kbit/s
  • Full-band: 48 to 128 kbit/s
  • (Narrowband and wideband not required)

5.2. Quality and Bit-Rate โœจ

Quality and bit-rate are directly linked, so you have to think about them together ๐Ÿ‘ฏ. When comparing codec bit-rates, the IP/UDP/RTP header overhead doesn't count, but any extra bits the RTP payload format needs after the header (like required signaling) do count ๐Ÿงพ. For quality versus bit-rate, the codec being developed has to beat these codecs, which are generally considered royalty-free ๐Ÿ‘‘:

  • For narrowband: Speex (NB) [Speex], and internet low bit-rate codec (iLBC)(*) [RFC3951]
  • For wideband: Speex (WB) [Speex], G.722.1(*) [ITU.G722.1]
  • For super-wideband/fullband: G.722.1C(*) [ITU.G722.1]

The codecs marked with (*) have extra licensing restrictions, but the codec being developed should still not perform significantly worse ๐Ÿ™. On top of the quality targets above, a desirable goal is for the codec quality to be no worse than Adaptive Multi-Rate (AMR-NB) and Adaptive Multi-Rate Wideband (AMR-WB) ๐ŸŒŸ. Quality should be measured across multiple languages, including tonal languages ๐ŸŒ. The case of multiple simultaneous voices (like what sometimes happens in conferencing) should be evaluated too ๐Ÿ—ฃ๏ธ

The comparison above assumes the codecs being compared have similar delay characteristics. A much lower delay requirement may push the bit-rate needed for a given quality above what the referenced codecs use. In that case, the increase in bit-rate must be less than the ratio between the delays ๐Ÿ“Š

It's desirable for codecs to support source-controlled variable bit-rate (VBR), so different inputs that need different bit-rates can reach the same quality ๐ŸŽ›๏ธ. Even so, it should still be possible to run the codec at a truly constant bit-rate, so no information leaks out when using an encrypted channel ๐Ÿ”

5.3. Packet-Loss Robustness โœจ

Robustness to packet loss is a huge deal for any codec used on the Internet ๐Ÿซฃ. Codecs must keep acceptable quality at loss rates up to 5%, and keep good intelligibility up to a 15% loss rate ๐Ÿ“‰. At any sampling rate, bit-rate, and packet-loss rate, the quality must be no less than what you'd get from the Speex codec or the Global System for Mobile Communications - Full Rate (GSM-FR) codec under the same conditions ๐Ÿ’ฏ. The actual packet-loss "patterns" used in testing must come from real packet-loss traces collected on the Internet, not from loss models ๐Ÿ•ต๏ธ. These traces should represent the typical environments where the applications of Section 3 operate. For example, traces for VoIP calls should consider the loss patterns seen on typical home broadband and corporate connections ๐Ÿ 

5.4. Computational Resources ๐Ÿ’…

okay so the codec has to run on a super wide range of devices ๐Ÿ“ฑ, which means a fixed-point implementation is ideal, or at the very least we need assurance that a reasonable fixed-point version is possible ๐Ÿ”ง. The complexity numbers below are upper bounds, not targets ๐ŸŽฏ. Even when you're under them, you still want to minimize resources โœจ. And if anyone proposes using more compute (say, to bump up quality), that gets carefully evaluated even when it still fits under the cap ๐Ÿง. Variable complexity would be really useful for this, though it's not required ๐Ÿ’–, because it lets you trade quality or bit-rate for lower complexity ๐Ÿ”.

These are the requirements for real-time encoding and decoding of a mono signal on one core of a recent x86 CPU, measured with the Unix time utility or something equivalent โฑ๏ธ:

  • Narrowband: 40 megahertz (MHz) (2% of a 2 gigahertz (GHz) CPU core)
  • Wideband: 80 MHz (4% of a 2 GHz CPU core)
  • Super-wideband/fullband: 200 MHz (10% of a 2 GHz CPU core)

It's also desirable that those MHz values are achievable on fixed-point digital signal processors that can do single-cycle multiply-accumulate operations, meaning 16x16 multiplication accumulated into 32 bits ๐Ÿค“.

For applications that do mixing, like conferencing ๐ŸŽค, it should be possible to estimate the energy and/or voice activity status of the decoded signal with less than 10% of the complexity figures above ๐Ÿ“Š.

The big goal is maximizing the range of devices the codec can run on ๐ŸŒ. So the reference implementation must not depend on special hardware features or instructions to hit the complexity requirement ๐Ÿšซ. That said, it may be desirable to use such hardware when it's available, like hardware accelerators for operations such as Fast Fourier Transforms (FFT) and convolutions โšก. A codec should also minimize its use of saturating arithmetic so it can run on architectures without hardware saturation, like ARMv4 ๐Ÿ› ๏ธ.

The combined codec size and data read-only memory (ROM) should be small enough not to cause significant implementation problems on typical embedded devices ๐Ÿ”Œ. The codec context/state size required should be no more than 2*R*C bytes in floating-point, where R is the sampling rate and C is the number of channels ๐Ÿง . For fixed-point, that size should be less than R*C ๐ŸŽ€. The scratch space required should also be less than 2*R*C bytes for floating point, or less than R*C bytes for fixed-point ๐Ÿงบ.


6. Additional Considerations โœจ

There are extra features or characteristics that can be really desirable in some situations, but they're not part of the strict requirements ๐Ÿ’ซ. The benefit of meeting them should be weighed against what they cost ๐Ÿ’….

6.1. Low-Complexity Audio Mixing

For lots of applications that need a mixing server, like conferencing or games ๐ŸŽฎ, it's really important to keep the cost of mixing low ๐Ÿ’ธ. As much as possible, mixing should be doable with fewer computations than decoding all the streams, mixing them, and re-encoding the result ๐Ÿงฎ. Things that help cut the mixing complexity include:

  • The ability to derive enough parameters, like loudness and/or spectral envelope, to estimate voice activity for a compressed frame without fully decoding that frame ๐Ÿ‘€;
  • The ability to mix streams in an intermediate representation (e.g., transform domain) instead of fully decoding the signals before mixing ๐ŸŒ€;
  • Using bit-stream layers (Section 6.3) by combining a small number of active streams at lower quality ๐ŸŽš๏ธ.

For conferencing, the total complexity of decoding, voice activity detection (VAD), and mixing should be considered when judging proposals ๐Ÿ“ž.

6.2. Encoder Side Potential for Improvement

With lots of codecs, you can improve quality just by upgrading the encoder without breaking compatibility, meaning without changing the decoder ๐Ÿš€. How much room there is to improve varies from codec to codec ๐Ÿ“ˆ. It's generally low for pulse code modulation (PCM) or adaptive differential pulse code modulation (ADPCM) codecs, and higher for perceptual transform codecs ๐ŸŽถ. All things being equal, being able to improve a codec after the bit-stream exists is a really nice property ๐Ÿฅฐ. However, this shouldn't come at the expense of quality in the reference encoder ๐Ÿ™…โ€โ™€๏ธ. Other possible improvements include signal-adaptive frame size selection and better discontinuous transmission (DTX) algorithms that predict the decoder side's packet loss concealment (PLC) algorithms ๐Ÿ”ฎ.

6.3. Layered Bit-Stream

A layered codec lets you send only a certain subset of the bits and still get a valid bit-stream with quality equivalent to encoding at the corresponding rate ๐Ÿฐ. This isn't necessary for most applications, but it can be desirable when a "mixing server" has to handle a huge number of streams with limited compute ๐Ÿ–ฅ๏ธ.

6.4. Partial Redundancy

One way to boost robustness to packet loss is to include partial redundancy within packets ๐Ÿ›ก๏ธ. You can do this by including the base layer of the previous frame (for a layered codec), or by sending other parameters from the previous frame(s) to help the PLC algorithm when a loss happens ๐Ÿช„. Being able to include partial redundancy for high-loss scenarios is desirable, as long as the feature can be turned on or off dynamically, so no bandwidth gets wasted when there's no loss ๐Ÿ™Œ.

6.5. Stereo Support

Stereo support is highly desirable for the codec ๐ŸŽง. At a minimum, the codec should be able to encode two channels independently without causing significant stereo image artifacts ๐Ÿ‘ฏ. It's also desirable for the codec to take advantage of the inter-channel redundancy in stereo audio to reduce the bit-rate (for equivalent quality) compared to coding the channels independently ๐ŸŒˆ.

6.6. Bit Error Robustness

Most Internet-based applications don't need to be robust to bit errors, because packets either arrive unaltered or don't arrive at all ๐Ÿ“ฆ. So the focus should be on packet-loss robustness and packet-loss concealment ๐Ÿ’œ. That said, extra robustness to bit errors can often be achieved at no cost at all (no increase in size, complexity, or bit-rate, and no decrease in quality or packet-loss robustness) โœจ. In those cases, it's useful to make the change that increases bit error robustness ๐Ÿงฉ. This helps applications that use UDP Lite transmission, like over a wireless LAN ๐Ÿ“ถ. But robustness to packet loss should never be sacrificed to get higher bit error robustness ๐Ÿšจ.

6.7. Time Stretching and Shortening

When adaptive jitter buffers are in use, you often need to stretch or shorten the audio signal so the buffering can change ๐Ÿซง. You can do this directly on the decoder's output, but it's often more computationally efficient to stretch or shorten the signal inside the decoder itself ๐Ÿง . It's desirable for the reference implementation to provide a time stretching/shortening implementation, although it should not be normative ๐Ÿ“.

6.8. Input Robustness

The systems feeding input to the encoder and receiving output from the decoder may be far from ideal in real life ๐Ÿซ . Input and output audio streams may be corrupted by stacked non-linear artifacts from analog hardware and digital processing ๐Ÿ“ผ. The codecs being developed should be tested to make sure they degrade gracefully under bad audio input conditions ๐Ÿ›Ÿ. Digital corruption that may be tested includes tandeming, transcoding, low-quality resampling, and digital clipping ๐Ÿ’ป. Analog corruption that may be tested includes microphones with heavy background noise, analog clipping, and loudspeaker distortion ๐ŸŽค. No specific end-to-end quality requirements are mandated for use with the proposed codec ๐Ÿ™…. It is advisable, though, that several typical in situ environments or processing chains get specified for benchmarking end-to-end quality with the proposed codec ๐Ÿ“‹.

6.9. Support of Audio Forensics

Emergency calls can be analyzed with audio forensics if the context and situation of the caller need to be identified ๐Ÿš‘. So it's important to transmit not only callers' voices well, but also background noise at high quality ๐Ÿ”Š. In these situations, sounds or noises of low volume also should not be compressed or dropped ๐Ÿซฃ. That's why the encoder must allow DTX to be disabled when required, for example for emergency calls ๐Ÿ†˜.

6.10. Legacy Compatibility

To create the best possible codec for the Internet, there's no requirement for compatibility with legacy Internet codecs ๐ŸŒŸ.

7. Security Considerations ๐Ÿ”

Although this document itself doesn't have security considerations, this section describes the security requirements for the codec ๐Ÿ›ก๏ธ.

As with any protocol used over the Internet, security is a really big deal ๐Ÿ˜ค. This goes beyond the obvious stuff like preventing buffer overflows and similar attacks that can lead to denial-of-service (DoS) or remote code execution ๐Ÿ’€. One very important security aspect is making sure the decoders have a bounded and reasonable worst-case complexity ๐Ÿงฑ. This stops an attacker from causing a DoS by sending packets specially crafted to take a very long (or infinite) time to decode ๐Ÿšซ.

There's also a sneakier issue: the information leak that can happen when the codec is used over an encrypted channel, like [SRTP] ๐Ÿ•ต๏ธโ€โ™€๏ธ. For example, it was suggested [wright08] [white11] that source-controlled VBR may reveal some information about a conversation through the size of the compressed packets ๐Ÿ‘‚. So it should be possible to use the codec at a truly constant bit-rate, if needed ๐Ÿ”’.

8. Acknowledgments ๐Ÿซถ

We'd like to thank everyone who contributed directly or indirectly to this document ๐Ÿ’œ, including Slava Borilin, Christopher Montgomery, Raymond (Juin-Hwey) Chen, Jason Fischl, Gregory Maxwell, Alan Duric, Jonathan Christensen, Julian Spittka, Michael Knappe, Christian Hoene, and Henry Sinnreich ๐ŸŒธ. We'd also like to thank Cullen Jennings, Jonathan Rosenberg, and Gregory Lebovitz for their advice ๐ŸŽ€.

9. Informative References โœจ

okay bestie, here's the reading list ๐Ÿ“š๐Ÿซถ these are the informative references, so they're the extra sources the doc points to. Every citation keeps its exact label and details so you can track each one down ๐Ÿ’–

  • [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. ๐Ÿ“ž
  • [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006. ๐ŸŽ€
  • [RFC6120] Saint-Andre, P., "Extensible Messaging and Presence Protocol (XMPP): Core", RFC 6120, March 2011. ๐Ÿ’ฌ
  • [XEP-0167] Ludwig, S., Saint-Andre, P., Egan, S., McQueen, R., and D. Cionoiu, "Jingle RTP Sessions", XSF XEP 0167, December 2009. ๐Ÿฆ‹
  • [RFC3951] Andersen, S., Duric, A., Astrom, H., Hagen, R., Kleijn, W., and J. Linden, "Internet Low Bit Rate Codec (iLBC)", RFC 3951, December 2004. ๐ŸŒธ
  • [ITU.G722.1] International Telecommunications Union, "Low-complexity coding at 24 and 32 kbit/s for hands-free operation in systems with low frame loss", ITU-T Recommendation G.722.1, May 2005. ๐ŸŽง
  • [Speex] Xiph.Org Foundation, "Speex: http://www.speex.org/", 2003. ๐ŸŒˆ
  • [carot09] Carot, A., Werner, C., and T. Fischinger, "Towards a Comprehensive Cognitive Analysis of Delay-Influenced Rhythmical Interaction: http://www.carot.de/icmc2009.pdf", 2009. ๐ŸŽถ
  • [PAYLOADS] Handley, M. and C. Perkins, "Guidelines for Writers of RTP Payload Format Specifications", BCP 36, RFC 2736, December 1999. ๐Ÿ“
  • [RTP] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003. ๐Ÿš€
  • [SRTP] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004. ๐Ÿ”
  • [wright08] Wright, C., Ballard, L., Coull, S., Monrose, F., and G. Masson, "Spot me if you can: Uncovering spoken phrases in encrypted VoIP conversations: http://www.cs.jhu.edu/~cwright/oakland08.pdf", 2008. ๐Ÿ•ต๏ธ
  • [white11] White, A., Matthews, A., Snow, K., and F. Monrose, "Phonotactic Reconstruction of Encrypted VoIP Conversations: Hookt on fon-iks http://www.cs.unc.edu/~fabian/papers/foniks-oak11.pdf", 2011. ๐Ÿ‘€

Authors' Addresses ๐Ÿ’Œ

these are the two people who wrote it, so here's how to reach them ๐Ÿซถ

Jean-Marc Valin
Mozilla
650 Castro Street
Mountain View, CA 94041
USA

EMail: jmvalin@jmvalin.ca
Koen Vos
Skype Technologies, S.A.
Stadsgarden 6
Stockholm, 11645
Sweden

EMail: koen.vos@skype.net

that's the whole doc, bestie ๐ŸŒˆ๐Ÿ‘‘ we made it โœจ

๐Ÿ“ฆ where this page comes from