RFC 5645

Update to the Language Subtag Registry

📅 published September 2009, from ietf

Update to the Language Subtag Registry 💖✨

RFC 5645 is a little Informational doc from the Network Working Group 🎀. Its author is D. Ewell, Ed., who was a Consultant, and it's dated September 2009 🌸. The header doesn't list anything it obsoletes or updates, so it doesn't replace or change any older RFC on paper. It's more of a helpful how-to for the registry 🧚.

Abstract ✨

Okay bestie, this memo is the step-by-step for updating the IANA Language Subtag Registry 📚. It works together with RFC 5646, and the whole point is making tags that identify languages 🌈.

Status of This Memo 🦋

This memo is info for the Internet community. It is not an Internet standard of any kind 🙈. You can share it freely, since distribution is unlimited 💅.

Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, as in effect on the date this document was published (http://trustee.ietf.org/license-info). Please read these carefully, bestie, because they explain your rights and restrictions for this document 📖.

Table of Contents 🍓

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  2
   2.  Updating the Registry  . . . . . . . . . . . . . . . . . . . .  2
     2.1.  Starting Point . . . . . . . . . . . . . . . . . . . . . .  2
     2.2.  New Language Subtags . . . . . . . . . . . . . . . . . . .  4
     2.3.  Modified Language Subtags  . . . . . . . . . . . . . . . .  5
     2.4.  New Region Subtags . . . . . . . . . . . . . . . . . . . .  6
     2.5.  Grandfathered and Redundant Tags . . . . . . . . . . . . .  6
     2.6.  Preferred-Value Changes  . . . . . . . . . . . . . . . . .  9
     2.7.  Additional Changes . . . . . . . . . . . . . . . . . . . .  9
   3.  Updated Registry Contents  . . . . . . . . . . . . . . . . . . 10
   4.  Security Considerations  . . . . . . . . . . . . . . . . . . . 10
   5.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 11
   6.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
     6.1.  Normative References . . . . . . . . . . . . . . . . . . . 11
     6.2.  Informative References . . . . . . . . . . . . . . . . . . 12
   Appendix A.  Acknowledgements  . . . . . . . . . . . . . . . . . . 13

1. Introduction 🧚

Okay so, [RFC4646] is where the Language Subtag Registry comes from, and it explains the format 💅. The starting contents and the rules for working them out are in [RFC4645] 📜.

[RFC5646] builds on [RFC4646] and adds support for approximately 7,500 new primary and extended language subtags 🤯. Those are based on the [ISO639-3] and [ISO639-5] alpha-3 code elements. It also adds seven new region subtags, based on [ISO3166-1] code elements that were exceptionally reserved 🌍. This memo explains how to update the registry to include all those extra subtags, plus the secondary changes that come from adding them and from other decisions made by the Language Tag Registry Update (LTRU) Working Group 🎀.

While this document was being written, a complete replacement of the contents of the Language Subtag Registry was given to the Internet Assigned Numbers Authority (IANA) so it could record the necessary updates 📮.

The format of the Language Subtag Registry, and the definition and intended purpose of each of its fields, are described in [RFC5646] 💖.

The registry is expected to change over time, because new subtags get registered and existing subtags get modified or deprecated 🦋. The process for updating it is described in Section 3 of [RFC5646] ✨.

A lot of the subtags in the Language Subtag Registry are based on code elements defined in [ISO639-1], [ISO639-2], [ISO639-3], [ISO639-5], [ISO3166-1], [ISO15924], and [UN_M.49] 🌈. The registry is not a mirror of the code lists defined by these standards, and it should not be used as one 🙅‍♀️.

2. Updating the Registry 🌸

This section is all about how to figure out the updated contents of the Language Subtag Registry 💜.

2.1. Starting Point 🚀

The version of the Language Subtag Registry that was current when this memo was approved by the IESG is the starting point for this update 🫶. That version was made using the process in [RFC4645] and kept up to date using the process in [RFC4646] 💅.

The source data for [ISO639-3] came in three files, all from the official ISO 639-3 Registration Authority site 🗂️. These files get updated from time to time, so just a heads-up: the version used for this memo was the one in place on February 24, 2009 📅.

  • [iso-639-3_20090210] is a list of every language code element in [ISO639-3]. It includes the alpha-3 code element and reference name for each one. For example, the entry for Dari had the code element prs and the name "Dari" (plus other info) 🧁.
  • [iso-639-3_Name_Index_20090210] is a list of all the names linked to each language according to [ISO639-3]. That includes inverted and non-inverted forms where they make sense 🪞. An "inverted" name is one that's been rearranged from the usual English order by moving adjectival qualifiers to the end, after the main language name and separated by a comma 📝. One code element can have more than one entry in this file. The reference name and its inverted form are usually, but not always, in the first entry 👀. For example, this file had an entry for prs with the name "Dari" (twice), and another entry with the names "Eastern Farsi" and "Farsi, Eastern" 🔥.
  • [iso-639-3-macrolanguages_20090120] is a list of every alpha-3 code element for languages that sit inside a macrolanguage in [ISO639-3], along with the alpha-3 code element for that macrolanguage 🌳. For example, a line with fas and prs showed that the macrolanguage "Persian" includes the individual language "Dari" 💫. Heads-up, though: these alpha-3 code elements may not match up directly with subtags in the registry, since the registry uses 2-letter subtags derived from [ISO639-1] whenever possible 🙈.

The source data for [ISO639-5] came in one file, from the official ISO 639-5 Registration Authority site 🗃️. This file also gets updated from time to time, and the version used for this memo was the one in place on February 24, 2009 📅.

  • [iso639-5.tab.txt] is a list of every language code element in [ISO639-5]. It has the alpha-3 code elements and the English name for each one. For example, this file has an entry with the code element ira and the name "Iranian languages" (plus other info) 🌈.

Language code elements that had already been retired in all of the source standards before IESG approval were not listed in these files 🪦. So they weren't considered in this update 🙅‍♀️.

The File-Date field values, the Added date for each new subtag record, and the Deprecated date for each existing grandfathered or redundant tag deprecated by this update were all set to a date as close as practical to the date this memo was approved for publication by the IESG 🗓️💖.

2.2. New Language Subtags 🎀

Okay, this part is kind of the main event 🤩. For each language in [ISO639-3] that wasn't already represented by a language subtag in the registry, a new language subtag was added. The [ISO639-3] code element became the value of the Subtag field, and each non-inverted [ISO639-3] name got its own Description field. The first Description field holds the [ISO639-3] reference name 📛.

If the language was encompassed by one of the [ISO639-3] macrolanguages ar (Arabic), kok (Konkani), ms (Malay), sw (Swahili), uz (Uzbek), or zh (Chinese), as determined by [iso-639-3-macrolanguages_20090120], an extended language subtag was also added 🧚. The Prefix field got the primary language subtag of that macrolanguage as its value 💅. These macrolanguage subtags were already in the registry. The LTRU Working Group picked them because they had been used to represent one dominant language as well as the macrolanguage as a whole. That makes the extended language mechanism a good fit for languages inside the macrolanguage 🫶.

If the language name had the word "Sign" in it, an extended language subtag was added with the string sgn as the value for the Prefix field 🤟. This is a special case. It treats the existing primary language subtag for "Sign languages" as if it were a macrolanguage that covers all sign languages 🌸.

Every extended language subtag was added with a Preferred-Value equal to its corresponding primary language subtag ✨.

If the language was encompassed by a macrolanguage, as determined by [iso-639-3-macrolanguages_20090120], a Macrolanguage field was added for that language. Its value is the subtag of the macrolanguage 🌳. (Note: sgn is defined as a "collection code" by [ISO639-3] and so isn't included in that standard. That's why no Macrolanguage field was added for sign language subtags 🙈.)

If the language was given a "Scope" value of M (Macrolanguage) in [iso-639-3_20090210], a Scope value of "macrolanguage" was added for it 🌈. Otherwise, if it was given a "Scope" value of S (Special), a Scope value of "special" was added 🦋. Most languages in [ISO639-3] have scope I (Individual), so they didn't get a Scope value in the registry 💅.

For each language in [iso639-5.tab.txt] that wasn't already represented by a language subtag in the registry, a new language subtag was added 🎉. The [ISO639-5] code element became the value of the Subtag field, and the "English name" field became the Description field 📝. Every one of these languages got a Scope value of "collection" in the registry 🗂️.

All subtags were added while keeping alphabetical order within each type of subtag: all 2-letter "language" subtags first, then all 3-letter "language" subtags, and finally all "extlang" subtags 🔤. Some existing records were moved so that this order would hold 💜✨.

2.3. Modified Language Subtags ✨

Okay so, for every language in [ISO639-3] that already had a language subtag in the Language Subtag Registry 💖, Description fields got added wherever needed so that every non-inverted name listed for that language in [iso-639-3_Name_Index_20090210] shows up 🌸. Any old Description fields that were inverted, or that only covered a strict subset of the [ISO639-3] name, were deleted 💀. The example is "Ainu" for ain, which said less than the [ISO639-3] name "Ainu (Japan)", so it had to go 🙈.

The Description fields were also reordered. The reference name from [ISO639-3] goes first, then the other names from [ISO639-3] in the order that standard lists them, and then any other names already in the registry 🔀. This sometimes shuffled existing entries even when nothing new got added, so don't be shocked if you spot that 👀.

For each language that falls under a macrolanguage in [ISO639-3], a Macrolanguage field was added, set to the subtag of that macrolanguage 🧚.

For each language in [iso639-5.tab.txt] that was already in the Language Subtag Registry, the Description was adjusted to match the "English name" field in [iso639-5.tab.txt]. Inverted names were rearranged to remove the inversion, and each of these languages got a Scope value of "collection" 📚. Older language subtags whose code elements were assigned before [ISO639-3] or [ISO639-5] were published, and which the [ISO639-3] Registration Authority identified as collections, also got a Scope value of "collection". This happened even though [iso639-5.tab.txt] doesn't list them that way 🫶.

Heads up: switching from [ISO639-2] names like "Afro-Asiatic (Other)" to [ISO639-5] names like "Afro-Asiatic languages" means the scope of some of these subtags got broader. Those are the "remainder groups" in [ISO639-5]. [iso639-5.tab.txt] has a field that says which code elements are "groups" or "remainder groups" in [ISO639-2], but [RFC5646] doesn't make that distinction, so that field wasn't used when updating the registry 🤷‍♀️

A Scope value of "private-use" was added for the one record with Subtag value qaa..qtz 🔐. Its Description was changed from "PRIVATE USE" to "Private use", and it covers a range of code elements that [ISO639-2] reserves for private use. The Description fields for the script and region private-use subtags got the same capitalization, "Private use" 💅.

2.4. New Region Subtags 🌍

[RFC5646] widens the scope of region subtags by adding subtags based on code elements that [ISO3166-1] calls "exceptionally reserved". The ISO 3166 Maintenance Agency reserves these "at the request of national ISO member bodies, governments and international organizations" 🏛️. When the IESG approved this memo, the ISO 3166/MA had defined nine exceptionally reserved code elements. All of them were added to the Language Subtag Registry except for these two:

  • FX (Metropolitan France) was already in the Language Subtag Registry, since it was an assigned [ISO3166-1] code element from 1993 to 1997. It got deprecated with a Preferred-Value of "FR" 🇫🇷
  • UK (United Kingdom) wasn't added, because it shares the UN M.49 code (826) with the existing region subtag GB. [RFC5646], Section 3.4, item 15 (D) says a new region subtag isn't added to the Language Subtag Registry if it carries the same meaning as an existing region subtag 🙅‍♀️

2.5. Grandfathered and Redundant Tags 🎀

As [RFC5646] puts it, "grandfathered" and "redundant" tags are complete tags in the Language Subtag Registry that were registered under [RFC1766] or [RFC3066] and are still valid 🕰️. Grandfathered tags can't be built from a valid combination of subtags. Redundant tags can 🧩.

Sometimes registering a subtag under [RFC5646] can turn a grandfathered tag into a redundant one. It can also make a generative tag possible that means the same thing as a grandfathered or redundant tag. When that happens, the grandfathered or redundant tag gets marked as Deprecated, and the generative tag (new subtag included) becomes its Preferred-Value 💫.

Because of the new subtags in this update, the following grandfathered tags became composable, got reclassified as redundant, and were deprecated with the generative tag as their Preferred-Value:

zh-cmn (Preferred-Value: cmn)
zh-cmn-Hans (Preferred-Value: cmn-Hans)
zh-cmn-Hant (Preferred-Value: cmn-Hant)
zh-gan (Preferred-Value: gan)
zh-wuu (Preferred-Value: wuu)
zh-yue (Preferred-Value: yue)

These grandfathered tags were deprecated, with the generative tag as the Preferred-Value:

i-ami (Preferred-Value: ami)
i-bnn (Preferred-Value: bnn)
i-pwn (Preferred-Value: pwn)
i-tao (Preferred-Value: tao)
i-tay (Preferred-Value: tay)
i-tsu (Preferred-Value: tsu)
zh-hakka (Preferred-Value: hak)
zh-min (no Preferred-Value; see below)
zh-min-nan (Preferred-Value: nan)
zh-xiang (Preferred-Value: hns)

zh-min is the odd one out 🤨. It was registered under [RFC1766] and stands for a small class of Chinese languages, but it isn't a true macrolanguage. The string "min" could never be used to tag these languages, because the [ISO639-3] code element min belongs to an individual language (Minangkabau) that has nothing to do with Chinese (zh). Since it isn't believed to be a useful linguistic entity for tagging, it was deprecated without a Preferred-Value 🙃.

These grandfathered and redundant sign language tags were deprecated, with the generative tag as the Preferred-Value 🤟:

sgn-BE-FR (Preferred-Value: sfb)
sgn-BE-NL (Preferred-Value: vgt)
sgn-BR (Preferred-Value: bzs)
sgn-CH-DE (Preferred-Value: sgg)
sgn-CO (Preferred-Value: csn)
sgn-DE (Preferred-Value: gsg)
sgn-DK (Preferred-Value: dsl)
sgn-ES (Preferred-Value: ssp)
sgn-FR (Preferred-Value: fsl)
sgn-GB (Preferred-Value: bfi)
sgn-GR (Preferred-Value: gss)
sgn-IE (Preferred-Value: isg)
sgn-IT (Preferred-Value: ise)
sgn-JP (Preferred-Value: jsl)
sgn-MX (Preferred-Value: mfs)
sgn-NI (Preferred-Value: ncs)
sgn-NL (Preferred-Value: dse)
sgn-NO (Preferred-Value: nsl)
sgn-PT (Preferred-Value: psr)
sgn-SE (Preferred-Value: swl)
sgn-US (Preferred-Value: ase)
sgn-ZA (Preferred-Value: sfs)

Nothing changed in the Description fields of any grandfathered or redundant tag. For example, the redundant tag sgn-US still says "American Sign Language" 🇺🇸. The sign language tags registered before [RFC4646] are the exception to the general rule that a non-grandfathered tag's meaning can be worked out from its component subtags 🧐.

Older versions of the registry gave grandfathered tags that were deprecated because of an ISO 639-based language subtag a Comments field reading "replaced by ISO code xxx", where xxx was the new language subtag. Those comments just repeated what the Preferred-Value field already said, so they were deleted in this update 🧹. No other Comments fields were changed 💕.

2.6. Preferred-Value Changes 🔄

[RFC5646], Section 3.1.7 lets the Preferred-Value fields be updated when a source standard changes. So the Preferred-Value fields for these deprecated tags were changed ✨:

i-hak (changed from zh-hakka to hak)
zh-guoyu (changed from zh-cmn to cmn)

Now consumers of the Language Subtag Registry don't have to follow a "chain" of Preferred-Values to reach a non-deprecated subtag 🙌.

2.7. Additional Changes 🌈

To match how alternative names are handled for language subtags, Description fields for script subtags taken from [ISO15924] that list alternative names were split into several Description fields. For example, "Han (Hanzi, Kanji, Hanja)" became four separate Description fields 🀄. Some script Descriptions had parenthetical bits that explained things rather than naming alternatives, and those were left alone 🤓.

This doesn't apply to region subtags taken from [ISO3166-1] and [UN_M.49], because those standards don't offer freely available alternative names for their code elements 🗺️.

Inverted Description fields for script and region subtags were rearranged to remove the inversion, matching what happened to language subtags in Sections 2.2 and 2.3. For example, "Korea, Republic of" became "Republic of Korea" 🇰🇷.

The capitalization of the Subtag fields for some grandfathered and redundant tags (sgn-BE-fr, sgn-BE-nl, sgn-CH-de, and yi-latn) was changed to follow the capitalization conventions in [RFC5646], Section 2.1.1. This doesn't change the validity or meaning of these tags 🔤.

The Description for subtag sgn was capitalized as "Sign languages" to match how other languages are capitalized in [ISO639-5]. That doesn't exactly match how code element sgn is capitalized in any ISO 639 part, though 🤷‍♀️.

The Deprecated field for region subtag TP changed from 2002-11-15 to 2002-05-20 to fix a clerical error. The corrected date is when the code element TP was actually withdrawn in [ISO3166-1] 📅.

The order of fields within records was adjusted to match the order [RFC5646], Section 3.1.2 describes. [RFC5646] doesn't require this ordering, and future additions or changes to the registry may not follow it 🗂️.

3. Updated Registry Contents 📋

IANA updated the Language Subtag Registry using the replacement contents provided. Those contents were in the working draft of this document but got deleted before publication as an RFC, so nobody gets confused with the registry itself 🙈. You can find the Language Subtag Registry on the IANA website, <http://www.iana.org> 🌐.

4. Security Considerations 🔐

For security considerations relevant to the Language Subtag Registry and the use of language tags, see [RFC5646] 🛡️.

5. IANA Considerations 🏷️

IANA has updated the Language Subtag Registry, which you can find via <http://www.iana.org>. For the procedures covering the format and ongoing maintenance of this registry, see RFC 5646 📝.

6. References 📚

6.1. Normative References ✨

okay bestie, these are the reference docs the spec leans on hardest 📚 you really need to have read them to follow along 💅

[ISO639-3] International Organization for Standardization, "ISO 639-3:2007. Codes for the representation of names of languages - Part 3: Alpha-3 code for comprehensive coverage of languages", February 2007.

[ISO639-5] International Organization for Standardization, "ISO 639-5:2008. Codes for the representation of names of languages -- Part 5: Alpha-3 code for language families and groups", May 2008.

[RFC5646] Phillips, A., Ed. and M. Davis, Ed., "Tags for Identifying Languages", RFC 5646, September 2009. this one is the big main character 🦋

[iso-639-3-macrolanguages_20090120] International Organization for Standardization, "ISO 639-3 Macrolanguage Mappings", January 2009, <http://www.sil.org/iso639-3/iso-639-3-macrolanguages_20090120.tab>.

[iso-639-3_20090210] International Organization for Standardization, "ISO 639-3 Code Set", February 2009, <http://www.sil.org/iso639-3/iso-639-3_20090210.tab>.

[iso-639-3_Name_Index_20090210] International Organization for Standardization, "ISO 639-3 Language Names Index", February 2009, <http://www.sil.org/iso639-3/iso-639-3_Name_Index_20090210.tab>.

[iso639-5.tab.txt] International Organization for Standardization, "ISO 639-5 code list, Tab-delimited text", February 2009, <http://www.loc.gov/standards/iso639-5/iso639-5.tab.txt>.

6.2. Informative References 🌸

these ones are the extra reading material, the cozy background stuff that gives you more context 🧚

[ISO15924] International Organization for Standardization, "ISO 15924:2004. Information and documentation -- Codes for the representation of names of scripts", January 2004.

[ISO3166-1] International Organization for Standardization, "ISO 3166-1:2006. Codes for the representation of names of countries and their subdivisions -- Part 1: Country codes", November 2006.

[ISO639-1] International Organization for Standardization, "ISO 639-1:2002. Codes for the representation of names of languages -- Part 1: Alpha-2 code", July 2002.

[ISO639-2] International Organization for Standardization, "ISO 639-2:1998. Codes for the representation of names of languages -- Part 2: Alpha-3 code", October 1998.

[RFC1766] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995.

[RFC3066] Alvestrand, H., "Tags for the Identification of Languages", RFC 3066, January 2001.

[RFC4645] Ewell, D., "Initial Language Subtag Registry", RFC 4645, September 2006.

[RFC4646] Phillips, A. and M. Davis, "Tags for Identifying Languages", BCP 47, RFC 4646, September 2006.

[UN_M.49] Statistics Division, United Nations, "Standard Country or Area Codes for Statistical Use", Revision 4 (United Nations publication, Sales No. 98.XVII.9, June 1999.

Appendix A. Acknowledgements 💖

this memo is a team project from the Language Tag Registry Update (LTRU) Working Group 🫶 everyone in the group put in serious work on this memo and on its predecessor, [RFC4645] ✨

big shoutout to these legends for their specific contributions: Stephane Bortzmeyer, John Cowan, Mark Davis, Martin Duerst, Frank Ellermann, Debbie Garside, Kent Karlsson, Gerard Lang, Addison Phillips, Randy Presuhn, and CE Whitehead 👑🌈

Author's Address 🎀

Doug Ewell (editor)
Consultant

EMail: doug@ewellic.org
URI:   http://www.ewellic.org
📦 where this page comes from