RFC 3066

Tags for the Identification of Languages

📅 published January 2001, from ietf

Tags for the Identification of Languages 🎀

Okay bestie, this is RFC 3066, and it's a Best Current Practice (BCP 47) 💅 It was written by H. Alvestrand from Cisco Systems, dated January 2001, and it's part of the Network Working Group's stuff. It obsoletes RFC 1766, so that older one is out and this is the new hotness ✨

Status of This Memo ✨

This document specifies an Internet Best Current Practices for the Internet Community 🌍 and asks people for discussion and suggestions for improvements. Distribution of this memo is unlimited 💖

Copyright (C) The Internet Society (2001). All Rights Reserved.

Abstract

Okay so this doc is all about a language tag 🏷️ It's a little label you stick on information to say what language it's in. It also explains how to register new values for those labels, and it gives a way to match tags against each other 🦋

1. Introduction

Humans on this planet have used a LOT of languages, past and present 🌸 There are tons of reasons you'd want to identify the language used when you're presenting information.

Sometimes information is available in more than one language, or there are tools (like dictionaries) that help you understand a language 📖

Also, lots of information processing needs to know the language the information is written in, otherwise the process just can't happen 🫶 Think spell-checking, computer-synthesized speech, Braille, or high-quality print renderings. Iconic behaviour from all of them 😭

One way to show the language is to label the content with an identifier for the language it uses ✨

This document specifies an identifier mechanism, a registration function for values to be used with that identifier mechanism, and a construct for matching against those values 🔥

Quick heads up on the keywords 💅 The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119].

2. The Language tag

Here's where the fun starts. 🧚

2.1 Language tag syntax

A language tag is made of one or more parts: a primary language subtag, followed by a series of subsequent subtags. That series can be empty, so the tag can be just the primary one 🎀

The syntax of this tag in ABNF [RFC 2234] is:

Language-Tag = Primary-subtag *( "-" Subtag )

Primary-subtag = 1*8ALPHA

Subtag = 1*8(ALPHA / DIGIT)

The productions ALPHA and DIGIT are imported from RFC 2234. They mean the characters A to Z in upper or lower case, and the digits 0 to 9 respectively 🌈 The character - is HYPHEN-MINUS (ABNF: %x2D).

All tags are case insensitive 🦋 There are capitalization conventions for some of them, but those conventions aren't supposed to carry any meaning. For example, [ISO 3166] recommends that country codes are capitalized (MN Mongolia), while [ISO 639] recommends that language codes are written in lower case (mn Mongolian). Iykyk 🙈

2.2 Language tag sources

The namespace of language tags is run by the Internet Assigned Numbers Authority (IANA) [RFC 2860], following the rules in section 3 of this document 💜

Here's the deal for the primary subtag 👑

  • All 2-letter subtags are interpreted according to assignments in ISO standard 639, "Code for the representation of names of languages" [ISO 639], or assignments subsequently made by the ISO 639 part 1 maintenance agency or governing standardization bodies. (Note: A revision is underway, and is expected to be released as ISO 639-1:2000)
  • All 3-letter subtags are interpreted according to assignments in ISO 639 part 2, "Codes for the representation of names of languages -- Part 2: Alpha-3 code [ISO 639-2]", or assignments subsequently made by the ISO 639 part 2 maintenance agency or governing standardization bodies.
  • The value "i" is reserved for IANA-defined registrations.
  • The value "x" is reserved for private use. Subtags of "x" shall not be registered by the IANA.
  • Other values shall not be assigned except by revision of this standard.

The reason for reserving all the other tags is to stay open to new revisions of ISO 639 🌸 Using "i" and "x" is the minimum we can do here, so the mechanism can be extended to cover our immediate requirements. Smart, right? 🧠

Now the second subtag 🎀

  • All 2-letter subtags are interpreted as ISO 3166 alpha-2 country codes from [ISO 3166], or subsequently assigned by the ISO 3166 maintenance agency or governing standardization bodies, denoting the area to which this language variant relates.
  • Tags with second subtags of 3 to 8 letters may be registered with IANA, according to the rules in chapter 5 of this document.
  • Tags with 1-letter second subtags may not be assigned except after revision of this standard.

There are no rules apart from the syntactic ones for the third and subsequent subtags 🤷‍♀️

Tags built entirely from codes that get interpretations in this chapter do not need to be registered with IANA before use. No paperwork, just vibes ✨

The information in a subtag may be things like:

  • Country identification, such as en-US (this usage is described in ISO 639)
  • Dialect or variant information, such as en-scouse
  • Languages not listed in ISO 639 that are not variants of any listed language, which can be registered with the i-prefix, such as i-tsolyani
  • Region identification, such as sgn-US-MA (Martha's Vineyard Sign Language, which is found in the state of Massachusetts, US) 🦋

This document leaves the decision about which tags are appropriate to the registration process in section 3 💅

ISO 639 has a maintenance agency for additions to and changes in its list of languages. That agency is:

International Information Centre for Terminology (Infoterm)
P.O. Box 130
A-1021 Wien
Austria

Phone: +43 1 26 75 35 Ext. 312
Fax:   +43 1 216 32 72

ISO 639-2 has its own maintenance agency for additions to and changes in the ISO 639-2 list. That one is:

Library of Congress
Network Development and MARC Standards Office
Washington, D.C. 20540
USA

Phone: +1 202 707 6237
Fax:   +1 202 707 0115
URL: http://www.loc.gov/standards/iso639

The maintenance agency for ISO 3166 (country codes) is:

ISO 3166 Maintenance Agency Secretariat
c/o DIN Deutsches Institut fuer Normung
Burggrafenstrasse 6
Postfach 1107
D-10787 Berlin
Germany

Phone: +49 30 26 01 320
Fax:   +49 30 26 01 231
URL: http://www.din.de/gremien/nas/nabd/iso3166ma/

ISO 3166 reserves the country codes AA, QM-QZ, XA-XZ and ZZ as user-assigned codes. These MUST NOT be used to form language tags 🚫

Reminder bestie: those reserved country codes are a hard no for building language tags. Do not do it 💀

2.3 Choice of language tag

Sometimes you'll be faced with several possible tags for the same body of text 🤔

Interoperability is best served if all users send the same tag, and use the same tag for the same language across all documents 🫶 If an application has requirements that make the rules here inapplicable, the application protocol specification MUST specify how the procedure varies from the one given here.

The rules below are based on the set of tags known to the tagging entity 💜

  1. Use the most precise tagging known to the sender that can be ascertained and is useful within the application context.
  2. When a language has both an ISO 639-1 2-character code and an ISO 639-2 3-character code, you MUST use the tag derived from the ISO 639-1 2-character code.
  3. When a language has no ISO 639-1 2-character code, and the ISO 639-2/T (Terminology) code and the ISO 639-2/B (Bibliographic) code differ, you MUST use the Terminology code. NOTE: At present, all languages for which there is a difference have 2-character codes, and the displeasure of developers about the existence of 2 code sets has been adequately communicated to ISO. So this situation will hopefully not arise. 🙃
  4. When a language has both an IANA-registered tag (i-something) and a tag derived from an ISO registered code, you MUST use the ISO tag. NOTE: When such a situation is discovered, the IANA-registered tag SHOULD be deprecated as soon as possible. 🍓
  5. You SHOULD NOT use the UND (Undetermined) code unless the protocol in use forces you to give a value for the language tag, even if the language is unknown. Omitting the tag is preferred. 🙈
  6. You SHOULD NOT use the MUL (Multiple) tag if the protocol allows you to use multiple languages, as is the case for the Content-Language: header. 💅

NOTE: To avoid versioning difficulties in applications such as that of RFC 1766, the ISO 639 Registration Authority Joint Advisory Committee (RA-JAC) has agreed on the following policy statement:

"After the publication of ISO/DIS 639-1 as an International Standard, no new 2-letter code shall be added to ISO 639-1 unless a 3-letter code is also added at the same time to ISO 639-2. In addition, no language with a 3-letter code available at the time of publication of ISO 639-1 which at that time had no 2-letter code shall be subsequently given a 2-letter code."

This will ensure that, for example, a user who implements "hwi" (Hawaiian), which currently has no 2-letter code, will not find his or her data invalidated by eventual addition of a 2-letter code for that language. 🌈✨

2.4. Meaning of the language tag ✨

okay so a language tag ALWAYS means a language that human beings speak, write, sign, or otherwise signal to share info with other human beings 🗣️💖 computer languages like programming languages are explicitly excluded, bestie 💅. there's no guaranteed relationship between tags that start with the same subtags. they are NOT guaranteed to be mutually intelligible, although sometimes they totally will be 🙈

the relationship between the tag and the info it describes is defined by the standard that covers the context where the tag shows up 📜 so this section can only give possible examples of how it gets used 🦋

  • For a single information object, it could be taken as the set of languages that you need for a complete understanding of the whole object. Example: plain text documents 📄
  • For an aggregation of information objects, it should be taken as the set of languages used inside the components of that aggregation. Examples: document stores and libraries 📚
  • For information objects whose purpose is to provide alternatives, the set of tags attached to it should be treated as a hint that the content comes in several languages, and that you have to inspect each alternative to find its language or languages. in this case, a tag with multiple languages does NOT mean you need to be multilingual to fully understand the document 🌈 Example: MIME multipart/alternative.
  • In markup languages like HTML and XML, language info can be added to each part of the document that the markup structure identifies, including the whole document itself 🎀 for example, you could write <span lang="FR">C'est la vie.</span> inside a Norwegian document. the Norwegian-speaking user could then open a French-Norwegian dictionary to figure out what the marked section meant 📖 and if the user was listening to that document through a speech synthesis interface, this info could tell the synthesizer to use French text-to-speech pronunciation rules for that span of text, instead of accidentally applying the Norwegian rules 🧚 main character energy for sure ✨

2.5. Language-range 💜

ok so ever since RFC 1766 came out, it's become super clear that we need a name for a set of languages whose tags all start with the same sequence of subtags 🍓

this definition of language-range is borrowed from HTTP/1.1 [RFC 2616], and it's giving simple:

language-range  = language-tag / "*"

so a language-range has the same syntax as a language-tag, or it's just the single character "*" 🫶

a language-range matches a language-tag if it exactly equals the tag, or if it exactly equals a prefix of the tag where the first character after that prefix is "-" 🔥

the special range "" matches any tag 👑 a protocol that uses language ranges can add its own extra rules about what "" means. for instance, HTTP/1.1 says the range "*" only matches languages that aren't matched by any other range inside an "Accept-Language:" header 💅

NOTE: this prefix matching rule doesn't mean language tags are assigned so that, if a user understands a language with a certain tag, they'll also understand every language whose tags have that tag as a prefix 🙈 the prefix rule just lets you use prefix tags when that's actually the case 💀

3. IANA registration procedure for language tags 📝

anyone who wants to use a language tag that isn't given an interpretation in chapter 2.2 of this document or hasn't been previously registered with IANA MUST follow the procedure here 🫶 this is the official process, no shortcuts, babe 💖

this procedure MAY also be used to register info with IANA about a tag defined by this document. like, if you want to make a reference to the definition of a language publicly available, say sgn-US (American Sign Language), you can do that 🦋

tags with a first subtag of "x" don't need to be registered, and they can't be registered either 🚫

the process kicks off by filling out the registration form below 🎀

----------------------------------------------------------------------

LANGUAGE TAG REGISTRATION FORM

Name of requester          :

E-mail address of requester:

Tag to be registered       :

English name of language   :

Native name of language (transcribed into ASCII):

Reference to published description of the language (book or article):

Any other relevant information:

----------------------------------------------------------------------

the language form has to be sent to <ietf-languages@iana.org> for a 2-week review period before it can be submitted to IANA 📬 (this is an open list. if you want to be added, send your request to <ietf-languages-request@iana.org> 🌸)

once the two weeks are up, the language tag reviewer (appointed by the IETF Applications Area Director) either forwards the request to IANA@IANA.ORG, or rejects it because of significant objections raised on the list 👑 the reviewer can also raise objections on the list themselves if they want to. the important part is that any objection has to be made publicly 🔥

the applicant is free to change a rejected application with more info and submit it again. that restarts the 2-week comment period 🔄

decisions made by the reviewer can be appealed to the IESG [RFC 2028], under the same rules as other IETF decisions [RFC 2026]. all registered forms are online in the directory http://www.iana.org/numbers.html under "languages" 🌈

updates to registrations follow the same procedure as new registrations. the language tag reviewer decides whether to let a new registrant update a registration someone else made. normally, objections from the original registrant carry extra weight in that decision ✨

there's no deleting registrations 🙅‍♀️ if a registered tag shouldn't be used anymore, for example because a matching ISO 639 code got registered, the registration should be amended by adding a remark like "DEPRECATED: use <new code> instead" to the "other relevant information" section 💅

Note: the "published description" is meant to help people verify whether a language is registered, or figure out what language a particular tag refers to 🧚 in most cases, a reference to an authoritative grammar or dictionary of the language will be useful. where no such work exists, other well-known works describing that language, or written in it, might be appropriate. the language tag reviewer decides what counts as "good enough" reference material 🎀

4. Security Considerations 🔐

ngl this is the tea 🍵 the only security issue raised with language tags since RFC 1766 came out (which said "Security issues are believed to be irrelevant to this memo") is a worry about language ranges used in content negotiation. they could be used to infer the nationality of the sender, and that could make them a target for surveillance 😭

this is a special case of a bigger problem: anything you send is visible to the receiving party. it's useful to know that concerns like this can show up in some cases 👀

figuring out the exact size of the threat, along with any possible countermeasures, is left up to each application protocol 🛡️

5. Character set considerations 🔤

language tags can always be shown using A-Z, a-z, 0-9 and HYPHEN-MINUS. these characters show up in most character sets, so showing language tags shouldn't cause any character set issues 💖

deciding how to render a character set based on the language tag is NOT covered in this memo 🙈 still, it's thought to be impossible to make that call correctly in every case, unless there are ways to switch languages in the middle of a text. for example, a rendering engine that picks a font based on Japanese or Chinese might give suboptimal output when it hits a mixed Japanese-Chinese text 😵‍💫

6. Acknowledgements 🫶

this document has benefited from many rounds of review and comments across lots of IETF forums and Internet working groups 🌸

any list of contributors is bound to be incomplete, so please treat the following as just a selection of the people who helped make this document what it is today 💜

in alphabetical order:

Glenn Adams, Tim Berners-Lee, Marc Blanchet, Nathaniel Borenstein, Eric Brunner, Sean M. Burke, John Clews, Jim Conklin, Peter Constable, John Cowan, Mark Crispin, Dave Crocker, Mark Davis, Martin Duerst, Michael Everson, Ned Freed, Tim Goodwin, Dirk-Willem van Gulik, Marion Gunn, Paul Hoffman, Olle Jarnefors, Kent Karlsson, John Klensin, Alain LaBonte, Chris Newman, Keith Moore, Masataka Ohta, Keld Jorn Simonsen, Otto Stolz, Rhys Weatherley, Misha Wolf, Francois Yergeau and many, many others 🎉

special thanks must go to Michael Everson, who served as language tag reviewer for almost the whole period since RFC 1766 was published, and who put a great deal of input into this revision 👑🔥

7. Author's Address 🦋

Harald Tveit Alvestrand
Cisco Systems
Weidemanns vei 27
7043 Trondheim
NORWAY

Phone: +47 73 50 33 52
EMail: Harald@Alvestrand.no

8. References 📚

  • [ISO 639] ISO 639:1988 (E/F) - Code for the representation of names of languages - The International Organization for Standardization, 1st edition, 1988-04-01 Prepared by ISO/TC 37 - Terminology (principles and coordination). Note that a new version (ISO 639-1:2000) is in preparation at the time of this writing.
  • [ISO 639-2] ISO 639-2:1998 - Codes for the representation of names of languages -- Part 2: Alpha-3 code - edition 1, 1998-11-01, 66 pages, prepared by a Joint Working Group of ISO TC46/SC4 and ISO TC37/SC2.
  • [ISO 3166] ISO 3166:1988 (E/F) - Codes for the representation of names of countries - The International Organization for Standardization, 3rd edition, 1988-08-15.
  • [RFC 1327] Kille, S., "Mapping between X.400 (1988) / ISO 10021 and RFC 822", RFC 1327, May 1992.
  • [RFC 1521] Borenstein, N., and N. Freed, "MIME Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, September 1993.
  • [RFC 2026] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.
  • [RFC 2028] Hovey, R. and S. Bradner, "The Organizations Involved in the IETF Standards Process", BCP 11, RFC 2028, October 1996.
  • [RFC 2119] Bradner, S. "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC 2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997.
  • [RFC 2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
  • [RFC 2860] Carpenter, B., Baker, F. and M. Roberts, "Memorandum of Understanding Concerning the Technical Work of the Internet Assigned Numbers Authority", RFC 2860, June 2000.

Appendix A: Language Tag Reference Material ✨

okay so the Library of Congress maintains ISO 639-2, and they've put their list of registered languages up on the Internet 📚🦋. At the time of writing, you can find it at http://www.loc.gov/standards/iso639-2/langhome.html

The IANA registration forms for registered language codes live at http://www.iana.org/numbers.html, under the "languages" heading 🌸

The ISO 3166 Maintenance Agency has its own web pages too, over here: http://www.din.de/gremien/nas/nabd/iso3166ma/ 💅

Appendix B: Changes from RFC 1766 🎀

okay, here's the glow-up list. Everything that changed since RFC 1766 🦋

  • The email list address moved from ietf-types@uninett.no to ietf-languages@iana.org 📮
  • The author's address got updated 💖
  • The language-range construct was added, borrowed from HTTP/1.1 ✨
  • ISO 639-2 language codes can now be used 🌈
  • There's a reference to the Library of Congress lists of language codes 📚
  • The examples were switched over to use registered tags 🫶
  • "Any other information" was added to the registration form 💜
  • A procedure for updating registrations was described 🔥
  • The target category changed from standards track to BCP 👑
  • The content-language header definition got moved into another document 🧚
  • Digits are now allowed in the permitted characters of language tags 🍓

okay, legal bit but make it cute 🌈 Copyright (C) The Internet Society (2001). All Rights Reserved. ✨

Anyone can copy this document and share it with others, and anyone can make derivative works that comment on it, explain it, or help people implement it. Those can be prepared, copied, published and distributed, in whole or in part, with no restrictions of any kind, as long as the copyright notice and this paragraph stay on every copy and derivative work 💖 BUT the document itself can't be modified in any way. You can't remove the copyright notice or the references to the Internet Society or other Internet organizations. The only exception is when you need it to develop Internet standards, in which case the copyright procedures from the Internet Standards process must be followed. You're also allowed to translate it into languages other than English 🌸

The limited permissions above last forever and the Internet Society, along with its successors or assigns, will never revoke them 💜

This document and the information in it are provided on an "AS IS" basis, and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 😭

Acknowledgement 🫶

Funding for the RFC Editor function is currently provided by the Internet Society. 👑

📦 where this page comes from