RFC 6569

Guidelines for Development of an Audio Codec within the IETF

📅 published March 2012, from ietf

RFC 6569 💖 Guidelines for Development of an Audio Codec within the IETF

Okay bestie, so the title block is giving "who's who." 🎀 This is an Informational doc (ISSN 2070-1721) dated March 2012. The authors are JM. Valin from Mozilla, S. Borilin from SPIRIT DSP, K. Vos (no affiliation listed 🤷‍♀️), C. Montgomery from the Xiph.Org Foundation, and R. Chen from Broadcom Corporation. The title block doesn't list anything it obsoletes or updates. ✨

Abstract

Okay so the abstract is the vibe check. 💅 This doc gives general guidelines for working on an interactive audio codec inside the IETF. It covers the development process, evaluation, requirements conformance, and intellectual property issues. Iconic. 🦋

Status of This Memo

Not a standards track spec, so don't get it twisted 🙈 it's published for information only.

This doc is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you're curious. 🧚

For current status, errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6569. ⭐

Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. Please read these carefully, bestie, because they explain your rights and restrictions. Code Components pulled out of this doc must include the Simplified BSD License text described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License describes. 🔥

Table of Contents

   1. Introduction ....................................................2
   2. Development Process .............................................2
   3. Evaluation, Testing, and Characterization .......................5
   4. Specifying the Codec ............................................6
   5. Intellectual Property ...........................................8
   6. Relationship with Other SDOs ...................................10
   7. Security Considerations ........................................12
   8. Acknowledgments ................................................12
   9. References .....................................................12
      9.1. Normative References ......................................12
      9.2. Informative References ....................................12

1. Introduction ✨

So this section sets the scene. 🎬 It describes a process for the IETF codec WG to standardize an audio codec. The codec is optimized for interactive Internet apps, and it needs to be widely implementable and easy to distribute to app developers, service operators, and end users. That's the whole goal, and it's giving main character energy. 👑

2. Development Process 💅

Okay so the whole point of this process is to keep work on a codec inside the IETF transparent, predictable, and well organized. 🦋 It has to be consistent with [PROCESS], so no going rogue. 🙈

The work could look a few different ways. Maybe someone builds a completely new codec. Maybe someone adapts an existing codec to meet the working group's requirements. Or maybe people combine two or more existing codecs into something better that takes the best parts of each. 🫶

For that procedural transparency to work, a contributor of an existing codec has to be willing to hand over change control to the IETF. They also should know the codec well enough to help adapt it or apply its tech to other codecs. 🧚

Also, heads up, bestie: any codec that comes out of IETF work will probably be different from the existing codec that was contributed to the Internet Standards Process. It's a glow-up, not a copy-paste. ✨

Work on developing an interactive audio codec goes like this:

  1. IETF participants identify the requirements an Internet codec needs to meet, written up as an Internet-Draft. 📝
  2. Interested parties are encouraged to propose existing or new codecs, or pieces of them, to the codec WG, as long as the contributions fit the WG's scope. Ideally these come as Internet-Drafts, but other forms of contribution are possible too, as [PROCESS] discusses. 💖
  3. IPR matters a lot to what the working group does, so any IPR disclosures must be made in a timely way. Contributors are required, as [IPR] describes, to disclose any known IPR, both first party and third party. Timely disclosure really matters, because it may be material to the working group's decisions. 🔥
  4. As contributions come in and get discussed, the group should get a clearer picture of what's achievable in the design space. So the requirements document's authors should keep iterating and improving it to reflect where the working group consensus is heading. This will probably mean teaming up with other IETF working groups. For example, they could work with groups in the Transport area to figure out how packets move across the Internet and how much rate adaptation is desirable. They could also work with groups in the RAI area to make sure information about the codec and its negotiation can be represented easily at the signaling layer. Meanwhile, interested parties should look at the contributions at a higher level to see which requirements each codec might meet. 🌸
  5. Once enough proposals are in, interested parties figure out the strengths, weaknesses, and innovative bits of each contributed codec. This step looks at the codecs as a whole, but also at the key features of individual algorithms, like predictors, quantizers, transforms, and so on. 🔍
  6. Interested parties are encouraged to collaborate and combine the best ideas from the different contributions into one consolidated codec definition. Through this back-and-forth, the number of proposals goes down and consensus usually forms around one of them. At that point the working group should adopt that document as a working group item, which becomes the baseline codec. 👑
  7. IETF participants should then keep trying to add to or improve each component of the baseline codec reference implementation. By "component," we mean individual algorithms like predictors, transforms, quantizers, and entropy coders. Participants should try new designs, borrow ideas from the contributed codecs, test "proof of concept" ideas, and use their codec-dev expertise to level up the baseline. Any part of the baseline might change, even the fundamental principles of the codec. Participants can even start over by scrapping the baseline and designing a totally new one. 🧚‍♀️ The overriding goal shall be to design a codec that meets the requirements in the requirements document [CODEC-REQ]. Because the IETF's standards process is open, anyone can contribute to this work, whether or not they submitted an Internet-Draft for one of the contributed codecs. 🌈 The codec itself should be normatively specified, with code, in an Internet-Draft. 💜
  8. While the codec reference implementation is being built, developers and other interested parties should evaluate the codec as described in Section 3. IETF participants should define the codec's payload format for use with the Real-time Transport Protocol [RTP], within the PAYLOAD working group. Ideally, app developers test the codec by implementing it in code and deploying it in real Internet apps. Honestly though, developers often wait to deploy until the codec is published as an RFC or a stable bitstream is guaranteed. 😭 So real deployment is a nice-to-have, not a requirement for this process. Lab implementations are definitely encouraged. 🍓
  9. The group will produce a testing results document. It's a living document that captures testing done before the codec stabilized, after it stabilized, and after the codec specification is issued as an RFC. This document helps the group decide whether the codec meets the requirements. Testing done after the codec RFC is issued helps implementers understand its final performance. The testing process itself is described in Section 3. 🎀

3. Evaluation, Testing, and Characterization ✨

Okay bestie, lab evaluation of the codec being developed should happen the whole way through the process 💅, because that's how you actually know you're getting closer to the requirements. There are so many ways to do continuous evaluation 🌈. For minor, uncontroversial changes, it should usually be enough to use objective measurements (like Perceptual Evaluation of Speech Quality (PESQ) [ITU-T-P.862], Perceptual Evaluation of Audio Quality (PEAQ) [ITU-R-BS.1387-1], and segmental signal-to-noise ratio) that get checked with informal subjective evaluation 🎀. For more complex changes (like when psychoacoustic aspects are involved) or for controversial issues, internal testing should be performed. An example of internal testing would be having individual participants rate the decoded samples using one of the established testing methodologies, such as MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) [ITU-R-BS.1534] 🧚.

Throughout the process, it's going to be super important to use the Internet community at large for real-world distributed testing 🌸. This lets lots of different people, with different equipment and different use cases, test the codec and report any problems they run into. In the same way, third-party developers will be encouraged to integrate the codec into their software (with a warning that the bitstream isn't final yet ⭐) and share feedback on how it performs in real-world use cases 🫶.

Characterization of the final codec must be based on the reference implementation only (and not on any "private implementation") 👑. This can be done by independent testing labs or, if that's not possible, by testing labs of the organizations that contribute to the Internet Standards Process 🦋. Packet-loss robustness should be evaluated using actual loss patterns collected from use over the Internet, rather than theoretical models 💀. The goals of the characterization phase are to:

  • ensure that the requirements have been fulfilled
  • guide the IESG in its evaluation of the resulting work
  • assist application developers in understanding whether the codec is suitable for a particular application

The exact methodology for the characterization phase can be determined by the working group 🍓. Because the IETF doesn't have testing resources of its own, it has to rely on the resources of its participants 🥺. So even if the group agrees that a particular test is important, if no one volunteers to do it, or if volunteers don't finish it in a timely fashion, then that test should be discarded 😭. This makes sure that only important tests get done, and in particular, the tests that matter to participants 💜.

4. Specifying the Codec

Specifying a codec takes careful thought about what is required versus what is left up to the implementation 🧠. The following text gives guidelines for the working group to consider:

  1. Any audio codec specified by the codec working group must include source code for a normative software implementation, documented in an Internet-Draft intended for publication as a Standards Track RFC 🔥. This implementation will be used to verify conformance of an implementation. Although a text description of the algorithm should be provided, its use should be limited to helping the reader understand the source code. Should the description contradict the source code, the latter shall take precedence. For convenience, the source code may be provided in compressed form, with base64 [BASE64] encoding 📦.
  2. Because most codecs are so big and complicated 🙈, it's possible that even after the RFC is published, bugs will be found in the reference implementation, or differences will be found between the implementation and the text description. As usual, an errata list should be maintained for the RFC 📝. Although a public software repository containing the current reference implementation is desirable, the normative implementation would still be the RFC 👑.
  3. It's the group's intention to allow the greatest possible choice of freedom in implementing the specification 🦋. So the number of binding RFC 2119 [KEYWORDS] keywords will be the minimum that still allows interoperable implementations. In practice, this generally means only the decoder needs to be normative, so the encoder can improve over time ✨. This also enables different trade-offs between quality and complexity 💅.
  4. To reduce the risk of bias towards certain CPU/DSP (central processing unit / digital signal processor) architectures 🖥️, ideally the decoder specification should not require "bit-exact" conformance with the reference implementation. In that case, the output of a decoder implementation should only be "close enough" to the output of the reference decoder, and a comparison tool should be provided along with the codec to objectively verify that a decoder's output is likely to be perceptually indistinguishable from that of the reference decoder 🎧. An implementation may still wish to produce output that is bit-exact with the reference implementation to simplify the testing procedure 🌈.
  5. To ensure freedom of implementation, decoder-side-only error concealment does not need to be specified 🫠, although the reference implementation should include the same packet-loss concealment (PLC) algorithm as used in the testing phase. Is it up to the working group to decide whether minimum requirements on PLC quality will be required for compliance with the specification. Obviously, any information signaled in the bitstream intended to aid PLC needs to be specified 🙌.
  6. An encoder implementation should not be required to make use of all the "features" (tools) in the bitstream definition 🧰. However, the codec specification may require that an encoder implementation be able to generate any possible bitrate. Unless a particular "profile" is defined in the specification, the decoder must be able to decode all features of the bitstream. The decoder must also be able to handle any combination of bits, even combinations that the reference encoder can't generate 🫣. It is recommended that the decoder specification shall define how the decoder should react to "impossible" packets (e.g., reject or consider as valid). However, an encoder must never generate packets that don't conform to the bitstream definition 🚫.
  7. Compressed test vectors should be provided as a means to verify conformance with the decoder specification 🎁. These test vectors should be designed to exercise as much of the decoder code as possible 💪.
  8. While the exact encoder won't be specified, it is recommended to specify objective measurement targets for an encoder, below which use of a particular encoder implementation is not recommended 📊. For example, one such specification could be: "the use of an encoder whose PESQ mean opinion score (MOS) is better than 0.1 below the reference encoder in the following conditions is not recommended" 💖.

5. Intellectual Property ✨

okay so let's talk about why an unencumbered codec is the move 🎀 It's giving "everyone gets to use it," and here's why that matters:

  • It's the experience of a bunch of app devs and service providers that encumbrances like licensing and royalties make it hard to implement, deploy, and distribute multimedia apps for the Internet community 💅
  • Low-cost options are super important, bestie, because innovation (the whole vibe of the Internet 🌈) gets blocked when small teams can't ship an app because of usage-based fees and royalties 😭
  • Lots of market segments are moving away from selling hard-coded hardware and toward freely distributing end-user software. That's true for big app providers and even telcos 🦋
  • Open source licensing is a whole thing, so we need to avoid encumbrances, including even having to get a license to implement, deploy, or use something (even when that license doesn't charge a fee) 🫶

So a codec that can be widely implemented and easily distributed to app devs, service operators, and end users is the preferred option 💖 Lowkey, lots of existing codecs that check some or most of the technical boxes above are encumbered in different ways. Like, patent holders might say you need to request a license, enter a business agreement, pay fees or royalties, or follow special conditions or restrictions just to implement the codec in software, deploy it in a service, or distribute it in software or hardware 💀 Because that makes it hard to widely implement and easily distribute high-quality codecs across the whole Internet community, the working group prefers unencumbered tech in a way that's consistent with BCP 78 [IPR] and BCP 79 [TRUST] ✨ Specifically, the working group SHALL heed the preference stated in BCP 79: "In general, IETF working groups prefer technologies with no known IPR claims or, for technologies with claims against them, an offer of royalty-free licensing." 👑 Even though this preference can't guarantee an unencumbered codec, the working group SHALL follow and adhere to the spirit of BCP 79 🫶 The working group can't explicitly rule out encumbered tech, but it will try to avoid encumbered techs that need royalties or other encumbrances that would stop them from being easy to redistribute and use 🙈

When it comes to license terms for technologies with IPR claims against them, some working group members have said they prefer terms that:

  • are available to everyone, worldwide, whether or not they're working group participants 🌸
  • extend to all essential claims owned or controlled by the licensor 💜
  • do not require payment of royalties, fees, or other consideration 🍓
  • do not require licensees to follow usage restrictions (though licenses that only cover implementing the standard are acceptable) 🧚
  • do not otherwise get in the way of the codec being implemented in open-source software projects ⭐

These guidelines help maximize the odds that the codec stays unencumbered:

  1. Following BCP 79 [IPR], contributed codecs SHOULD preferably use technologies with no known IPR claims or technologies with an offer of royalty-free licensing. 💅
  2. As BCP 79 describes, the working group SHOULD use technologies that participants perceive to be safer with regard to IPR issues. 🔥
  3. Contributors MUST disclose IPR as specified in BCP 79. 🫶
  4. If no royalty-free license can be obtained for a patent, BCP 79 suggests that the working group consider alternative algorithms or methods, even if they're lower quality, higher complexity, or otherwise less desirable. 😭
  5. Following BCP 78 [TRUST], the source code for the reference implementation MUST be made available under a BSD-style license (or whatever license the IETF Trust defines as acceptable when the Internet-Draft defining the reference implementation is published). 👑

Lots of IPR licenses only grant rights for technologies the IETF has adopted as a standard. That's reasonable, but it has the unintended side effect of discouraging implementation before RFC status 🙈 Real-world implementation is super useful for evaluating the codec, so entities making IPR license statements are ENCOURAGED to use wording that permits early implementation and deployment 🌈

Also, IETF participants should be aware that, because of how patents work in most countries, the resulting codec can never be guaranteed to be free of patent claims 💀 Some patents might not be known to the contributors, some patent applications might not be disclosed when the codec is developed, and only courts of law can decide the validity and breadth of patent claims. These observations are no different within the Internet Standards Process than they are for codec standardization in other Standards Development Organizations (SDOs) (or codec development outside any SDO). They're also no different for codecs than for other technologies worked on in the IETF 🦋 In all these cases, the best approach is to minimize the risk of unknowingly incurring encumbrance on existing patents. Even with these precautions, participants need to understand that it's practically nearly impossible to _guarantee_ that implementers won't incur encumbrance on existing patents 😭

6. Relationship with Other SDOs 🌸

Other SDOs are also involved in codec development and standardization, including but not necessarily limited to:

  • The Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), in particular Study Group 16 🎀
  • The Moving Picture Experts Group (MPEG) of the International Organization for Standardization and International Electrotechnical Commission (ISO/IEC) 🧚
  • The European Telecommunications Standards Institute (ETSI) 🌈
  • The 3rd Generation Partnership Project (3GPP) 💜
  • The 3rd Generation Partnership Project 2 (3GPP2) ✨

It's important to make sure this work doesn't turn into uncoordinated protocol development of the kind described in [UNCOORD], based on this principle:

   [T]he IAB considers it an essential principle of the protocol
   development process that only one SDO maintains design authority
   for a given protocol, with that SDO having ultimate authority over
   the allocation of protocol parameter code-points and over defining
   the intended semantics, interpretation, and actions associated
   with those code-points.

The work this guidelines document describes is not uncoordinated in the sense of that quote, because the plan is that two possible outcomes might happen:

  1. The IETF adopts an existing audio codec and specifies that it's the "anointed" IETF Internet codec. In that case, codec ownership lies entirely with the SDO that produced the codec, not with the IETF 👑
  2. The IETF produces a new codec. Even if this codec uses concepts, algorithms, or even source code from a codec produced by another SDO, the IETF codec is a specification unto itself and under complete control of the IETF 💅 Any changes or enhancements the original SDO makes to the codecs whose components the IETF used are not applicable to the IETF codec. Such changes would be incorporated as a consequence of a revision or extension of the IETF RFC. In no case should the new codec reuse a name or code point from another SDO. 🔥

Although there's already enough codec expertise among IETF participants to finish the envisioned work, extra contributions are welcome within the Internet Standards Process in these ways:

  • Individuals who are technical contributors to codec work in other SDOs can take part directly in codec work within the IETF 🦋
  • Other SDOs can contribute their expertise (e.g., codec characterization and evaluation techniques) and help with testing a codec produced by the IETF 🫶
  • Any SDO can give input to IETF work through liaison statements 🌸

Just note, though, that final responsibility for the development process and the resulting codec stays with the IETF, as governed by BCP 9 [PROCESS] 💜

Finally, there's precedent for contributing codecs developed elsewhere to the ITU-T (e.g., Adaptive Multi-Rate Wideband (AMR-WB) was originally standardized within 3GPP) ✨ This is a model to explore as the IETF coordinates further with the ITU-T under the collaboration guidelines defined in [COLLAB] 🎀

7. Security Considerations 💀

The procedural guidelines for codec development don't have security considerations 🙈 However, the resulting codec needs to take appropriate security considerations into account, as outlined in [SECGUIDE] and in the security considerations of [CODEC-REQ]. More specifically, the resulting codec MUST avoid being subject to denial of service [DOS] and buffer overflows, and SHOULD take into consideration the impact of variable bitrate (VBR) [SRTP-VBR] 😭🔥

8. Acknowledgments 🫶

We'd like to thank everyone who contributed directly or indirectly to this document, including Jason Fischl, Gregory Maxwell, Alan Duric, Jonathan Christensen, Julian Spittka, Michael Knappe, Timothy B. Terriberry, Christian Hoene, Stephan Wenger, and Henry Sinnreich 💖 We'd also like to thank Cullen Jennings and Gregory Lebovitz for their advice 🌈 Special thanks to Peter Saint-Andre, who originally co-authored this document 👑

9. References 📚

9.1. Normative References 🦋

[IPR]              Bradner, S., "Intellectual Property Rights in IETF
                   Technology", BCP 79, RFC 3979, March 2005.

[PROCESS]          Bradner, S., "The Internet Standards Process --
                   Revision 3", BCP 9, RFC 2026, October 1996.

[TRUST]            Bradner, S. and J. Contreras, "Rights Contributors
                   Provide to the IETF Trust", BCP 78, RFC 5378,
                   November 2008.

9.2. Informative References 📚

okay so these are the bibliography babes 🎀 they're not required reading for the spec itself, but they're the sources the doc points to. go check them out if you're curious 🧚

  • [BASE64] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 4648, October 2006.
  • [CODEC-REQ] Valin, J. and K. Vos, "Requirements for an Internet Audio Codec", RFC 6366, August 2011.
  • [COLLAB] Fishman, G. and S. Bradner, "Internet Engineering Task Force and International Telecommunication Union - Telecommunications Standardization Sector Collaboration Guidelines", RFC 3356, August 2002.
  • [DOS] Handley, M., Rescorla, E., and IAB, "Internet Denial-of-Service Considerations", RFC 4732, December 2006.
  • [ITU-R-BS.1387-1] "Method for objective measurements of perceived audio quality", ITU-R Recommendation BS.1387-1, November 2001.
  • [ITU-R-BS.1534] "Method for the subjective assessment of intermediate quality levels of coding systems", ITU-R Recommendation BS.1534, January 2003.
  • [ITU-T-P.862] "Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs", ITU-T Recommendation P.862, October 2007.
  • [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RTP] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
  • [SECGUIDE] Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations", BCP 72, RFC 3552, July 2003.
  • [SRTP-VBR] Perkins, C. and JM. Valin, "Guidelines for the Use of Variable Bit Rate Audio with Secure RTP", RFC 6562, March 2012.
  • [UNCOORD] Bryant, S., Morrow, M., and IAB, "Uncoordinated Protocol Development Considered Harmful", RFC 5704, November 2009.

Authors' Addresses 💌

these are the humans who made it happen, so give them some love 🫶

Jean-Marc Valin
Mozilla
650 Castro Street
Mountain View, CA  94041
USA

EMail: jmvalin@jmvalin.ca


Slava Borilin
SPIRIT DSP

EMail: borilin@spiritdsp.com


Koen Vos

EMail: koen.vos@skype.net


Christopher Montgomery
Xiph.Org Foundation

EMail: xiphmont@xiph.org


Raymond (Juin-Hwey) Chen
Broadcom Corporation

EMail: rchen@broadcom.com
📦 where this page comes from