RFC 6497

BCP 47 Extension T - Transformed Content

📅 published February 2012, from ietf

RFC 6497 💅 BCP 47 Extension T - Transformed Content 🎀

  • Number: RFC 6497 ✨
  • Authors: M. Davis (Google), A. Phillips (Lab126), Y. Umaoka (IBM), C. Falk (Infinite Automata) 🫶
  • Category: Informational 🌸
  • Date: February 2012 📅
  • ISSN: 2070-1721
  • Obsoletes / Updates: the title block doesn't list anything this one obsoletes or updates 🦋

Abstract ✨

okay so this doc adds an Extension to BCP 47 🧚 that gives you subtags for saying what the source language or script was for content that got transformed. that includes stuff that's been transliterated, transcribed, or translated, or just somehow influenced by the source 🌈 it also has extra info you can use for identification 🔍

Status of This Memo 📝

this one is not an Internet Standards Track spec, it's published for informational purposes 💜

it's a product of the Internet Engineering Task Force (IETF) 👑 it represents the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). not every doc the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 for that 🙈

for the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc6497 🍓

Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date it was published. read these carefully, bestie, because they spell out your rights and restrictions for this document 📜

Code Components pulled out of this doc have to include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License ⚖️

Table of Contents 🗺️

   1. Introduction ....................................................2
      1.1. Requirements Language ......................................4
   2. BCP 47 Required Information .....................................4
      2.1. Overview ...................................................4
      2.2. Structure ..................................................6
      2.3. Canonicalization ...........................................7
      2.4. BCP 47 Registration Form ...................................8
      2.5. Field Definitions ..........................................8
      2.6. Registration of Field Subtags .............................10
      2.7. Registration of Additional Fields .........................11
      2.8. Committee Responses to Registration Proposals .............11
      2.9. Machine-Readable Data .....................................11
   3. Acknowledgements ...............................................14
   4. IANA Considerations ............................................14
   5. Security Considerations ........................................14
   6. References .....................................................14
      6.1. Normative References ......................................14
      6.2. Informative References ....................................15

1. Introduction ✨

okay so [BCP47] lets you define and register language tag extensions "that contain a language component and are compatible with applications that understand language tags" 💖 this doc defines one for saying where transformed content came from 🦋 that covers text that's been transliterated, transcribed, or translated, or otherwise influenced by the source. you can use it in queries too, to ask for content that's been transformed 🔎 the "singleton" identifier for this extension is t 🎀

language tags from [BCP47] are great for saying what language some content is in 🌍 there are variant subtags for special purposes too, but ngl they just aren't enough for content that's been transformed, like transliterated, transcribed, or translated stuff 😭 and figuring out what the content actually means can depend on knowing the conventions that were used for the transformation 🧠

picture this: Italian or Russian city names on a map get transcribed for Japanese users 🗺️ each name has to be transliterated into katakana using rules that fit the specific source and target language 🍓 so when you tag that data, it's important to show not just the resulting content language ("ja" in this case) but also the source language 💅

transforms like transliterations can change depending on more than just the source and target script. they can also depend on the source and target language 🌈 so the Russian <U+041F U+0443 U+0442 U+0438 U+043D> (which corresponds to the Cyrillic <PE, U, TE, I, EN>) becomes "Putin" in English but "Poutine" in French 👑 this identifier can point to a desired mechanical transformation in an API, or it can tag data that got converted (mechanically or by hand) using a transliteration method ✨

and lowkey, there are so many different conventions for transforming text, even between the same languages and scripts 😩 "Gaddafi" is commonly transliterated from Arabic to English as any of (G/Q/K/Kh)a(d/dh/dd/dhdh/th/zz)af(i/y). some standardized conventions for transcribing or transliterating text include:

a. United Nations Group of Experts on Geographical Names (UNGEGN)

b. US Library of Congress (LOC)

c. US Board on Geographic Names (BGN)

d. Korean Ministry of Culture, Sports and Tourism (MCST)

e. International Organization for Standardization (ISO)

and this extension isn't only for formal transformations 💜 it can also cover other cases where the content is somehow influenced by the source. for example, you could use it to ask for a speech recognizer tailored specifically for second-language speakers who are first-language speakers of a particular language (e.g., a recognizer for "English spoken with a Chinese accent") 🎤🫶

1.1. Requirements Language 📖

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 🔥

2. BCP 47 Required Information 🌟

2.1. Overview ✨

Okay bestie, identifying transformed content is done with the 't' extension we're defining here 💅 It's formed by the 't' singleton followed by a sequence of subtags that would form a language tag per [BCP47]. That lets you name the source language or script to exactly the precision you need 🎀 There are restrictions on those subtags, though. They MUST form a regular, valid, canonical language tag, and they MUST neither include extensions nor private use sequences introduced by the singleton 'x'. If only the script matters (like when you're marking a script-to-script transliteration), use 'und' for the primary language subtag 🦋

Here are some examples:

+---------------------+---------------------------------------------+
| Language Tag        | Description                                 |
+---------------------+---------------------------------------------+
| ja-t-it             | The content is Japanese, transformed from   |
|                     | Italian.                                    |
| ja-Kana-t-it        | The content is Japanese Katakana,           |
|                     | transformed from Italian.                   |
| und-Latn-t-und-cyrl | The content is in the Latin script,         |
|                     | transformed from the Cyrillic script.       |
+---------------------+---------------------------------------------+

Note that the sequence of subtags governed by 't' cannot contain a singleton (a single-character subtag), because that would start a new extension. 💀 For example, the tag "ja-t-i-ami" does not indicate that the source is in "i-ami", because "i-ami" is not a regular language tag in [BCP47]. That tag would express an empty 't' extension followed by an 'i' extension. Not the vibe, so avoid it 🙈

The 't' extension is not meant for structured data that already gives you separate source and target language identifiers. Localization interchange formats like XLIFF are the classic example. In those cases it would be inappropriate to use "ja-t-it" for the target language tag, because the source language tag "it" would already be sitting right there in the data. Instead you'd just use the language tag "ja" ✨

Like we said earlier, sometimes you need to share extra info about a transformation. That extra info is optionally supplied after the source, in a series of one or more fields. Each field is a field separator subtag followed by one or more non-separator subtags. Each field separator subtag is a single letter followed by a single digit 🌸

A transformation mechanism is an optional field that tells you which specification was used for the transformation, such as "UNGEGN" for the United Nations Group of Experts on Geographical Names transliterations and transcriptions. It uses the 'm0' field separator followed by certain subtags 🗺️

For example:

+------------------------------------+------------------------------+
| Language Tag                       | Description                  |
+------------------------------------+------------------------------+
| und-Cyrl-t-und-latn-m0-ungegn-2007 | The content is in Cyrillic,  |
|                                    | transformed from Latin,      |
|                                    | according to a UNGEGN        |
|                                    | specification dated 2007.    |
+------------------------------------+------------------------------+

The field separator subtags, like 'm0', were picked because they're short, visually distinctive, and cannot occur in a language subtag (outside of an extension and after 'x'). So there's zero chance of a collision or mix-up with the source language tag 🫶

The field subtags are defined by Section 3 of Unicode Technical Standard #35: Unicode Locale Data Markup Language (LDML) [UTS35], which is the main spec for the Unicode Common Locale Data Repository (CLDR) project. That same section also defines the parallel 'u' extension [RFC6067], and the Unicode Consortium is the maintaining authority for that one too. As required by BCP 47, subtags follow the language tag ABNF and the other rules for forming language tags and subtags. They're restricted to the ASCII letters and digits, they're not case sensitive, and they don't exceed eight characters in length 🔥

The LDML specification is available over the Internet at no cost, and it's available under a royalty-free license at http://unicode.org/copyright.html. LDML is versioned, and every version is numbered, dated, and stable. Extension subtags, once LDML defines them, are never retracted or substantially changed in meaning 👑

The maintaining authority for the 't' extension is the Unicode Consortium:

+---------------+---------------------------------------------------+
| Item          | Value                                             |
+---------------+---------------------------------------------------+
| Name          | Unicode Consortium                                |
| Contact Email | cldr-contact@unicode.org                          |
| Discussion    | cldr-users@unicode.org                            |
| List Email    |                                                   |
| URL Location  | cldr.unicode.org                                  |
| Specification | Unicode Technical Standard #35 Unicode Locale     |
|               | Data Markup Language (LDML),                      |
|               | http://unicode.org/reports/tr35/                  |
| Section       | Section 3 Unicode Language and Locale Identifiers |
+---------------+---------------------------------------------------+

2.2. Structure 🧩

The subtags in the 't' extension look like this:

t-ext     = "t"                      ; Extension
          (("-" lang *("-" field))   ; Source + optional field(s)
          / 1*("-" field))           ; Field(s) only (no source)

lang      = language                 ; BCP 47, with restrictions
          ["-" script]
          ["-" region]
          *("-" variant)

field     = fsep 1*("-" 3*8alphanum) ; With restrictions

fsep      = ALPHA DIGIT              ; Subtag separators
alphanum  = ALPHA / DIGIT

where <language>, <script>, <region>, and <variant> rules are specified in [BCP47], and <ALPHA> and <DIGIT> rules in [RFC5234].

Description and restrictions:

a. The 't' extension MUST have at least one subtag. 🫠

b. The 't' extension normally starts with a source language tag, which MUST be a regular, canonical language tag as specified by [BCP47]. Tags described by the 'irregular' production in BCP 47 MUST NOT be used to form the language tag. The source language tag MAY be omitted, since some field values don't require it 🌈

c. There is optionally a sequence of fields, where each field has a separator followed by a sequence of one or more subtags. Two identical field separators MUST NOT be present in the language tag. Don't double up, bestie 🙅‍♀️

d. The order of the fields in a 't' extension is not significant. The order of subtags within a field is significant. See Section 2.3 ("Canonicalization") 💖

e. The 't' subtag fields are defined by Section 3 of Unicode Technical Standard #35: Unicode Locale Data Markup Language [UTS35] ✨

2.3. Canonicalization 🪄

As required by [BCP47], uppercase and lowercase letters don't matter in the subtags used in this extension. The canonical form for all subtags in the extension is lowercase, with the fields ordered by their separators, alphabetically. The order of subtags within a field is significant, and it MUST NOT be changed while canonicalizing. Hands off the order, periodt 💅

2.4. BCP 47 Registration Form 📝

Per RFC 5646, Section 3.7 [BCP47]:

%%
Identifier: t
Description: Specifying Transformed Content
Comments: Subtags for the identification of content that has been
   transformed, including but not limited to:
   transliteration, transcription, and translation.
Added: 2011-12-16
RFC: RFC 6497
Authority: Unicode Consortium
Contact_Email: cldr-contact@unicode.org
Mailing_List: cldr-users@unicode.org
URL: http://www.unicode.org/Public/cldr/latest/core.zip
%%

2.5. Field Definitions ✨

okay so who gets to assign the 't' field subtags? The Unicode CLDR Technical Committee 💅, following the procedures at http://www.unicode.org/consortium/tc-procedures.html, and it's all subject to the Unicode Consortium Policies at http://www.unicode.org/policies/policies.html 🫶

Here's what the Unicode Consortium can add through newer versions of LDML [UTS35] without needing a whole new RFC 🌈

  • allocating new field separator subtags for use after the 't' extension
  • allocating subtags that are valid after a field separator subtag
  • adding subtag aliases and descriptions
  • modifying subtag descriptions
Changes to the syntax or meaning of the 't' extension would need a brand new RFC that obsoletes this document. That would break stability, which is a no-go under Unicode Consortium policy 😭

When this document was published, [UTS35] specified one field separator subtag: the transform mechanism 🦋 here's the tea:

  • a. The transform mechanism is a sequence of subtags that starts with the m0 separator, followed by one or more mechanism subtags. Each mechanism subtag is 3 to 8 alphanumeric characters long. The whole sequence identifies the specification for the transform. For example, the mechanism subtag ungegn in und-Cyrl-t-und-latn-m0-ungegn 🌸 In lots of cases one mechanism subtag is enough, but multiple subtags MAY be defined in [UTS35] where necessary ✨
  • b. Any purely numeric subtag stands for a date in the Gregorian calendar. It MAY show up in any mechanism field, but it SHOULD only be used where it's needed 🙈 If it does show up:
    • it MUST be the final subtag in the field
    • it MUST NOT be the only subtag in the field
    • it MUST only consist of digits in the form YYYY, YYYYMM, or YYYYMMDD
    • it SHOULD be as short as possible
Note: This format is related to the one in [RFC3339], but it's not the same. The RFC 3339 full-date won't work here because it uses hyphens. The offset ("Z") isn't used because the date is a publication date (aka 'floating date'). For more info, see Section 3.3 ("Floating Time") of [W3C-TimeZones] 🧚
  • c. Examples, because examples are everything 💖
    • 20110623 means June 23, 2011.
    • There are three dated versions of the UNGEGN transliteration spec for Hebrew to Latin, and they can be written as these language tags:
      • und-Hebr-t-und-latn-m0-ungegn-1972
      • und-Hebr-t-und-latn-m0-ungegn-1977
      • und-Hebr-t-und-latn-m0-ungegn-2007
    • Okay, now imagine the BGN transliteration spec for Cyrillic to Latin had three versions, dated June 11, 1999; Dec 30, 1999; and May 1, 2011. In that case the first two DATE subtags would need months to tell them apart (199906 and 199912), but the last one would only need the year (2011) 👑
  • d. Some mechanisms use a versioning system that isn't based on date, or isn't based on date alone. In the second case, the version takes whatever form [UTS35] specifies for that mechanism. For example, if the mechanism xxx uses versions like v21a, a tag could look like ja-t-it-m0-xxx-v21a. If there are sub-versions that are distinguished by date, a tag could look like ja-t-it-m0-xxx-v21a-2007 💅

A language tag with the 't' extension MAY be used to request a specific transform of content. In that case, the recipient SHOULD return content that matches the requested transform as closely as feasible, including the specification of the mechanism. For example, say the request is ja-t-it-m0-xxx-v21a-2007, and the recipient has content that matches both ja-t-it-m0-xxx-v21a and ja-t-it-m0-xxx-v21b-2009. Then the v21a version would be preferred 🎀 As is the case for language matching as discussed in [BCP47], different implementations MAY have different measures of "closeness" 🌈

2.6. Registration of Field Subtags 📝

Registering a transform mechanism means filing a ticket at http://cldr.unicode.org/. The proposal in the ticket MUST contain the following information 🫶

+-------------+-----------------------------------------------------+
| Item        | Description                                         |
+-------------+-----------------------------------------------------+
| Subtag      | The proposed mechanism subtag (or subtag sequence). |
| Description | A description of the proposed mechanism; that       |
|             | description MUST be sufficient to distinguish it    |
|             | from other mechanisms in use.                       |
| Version     | If versioning for the mechanism is not done         |
|             | according to date, then a description of the        |
|             | versioning conventions used for the mechanism.      |
+-------------+-----------------------------------------------------+

You can also file a ticket to ask for clarifications of descriptions or extra aliases 🦋 And the committee MAY define a template for submissions that asks for more information, if they find that the extra info would help them evaluate proposals 💜

2.7. Registration of Additional Fields 🌸

What if we need another field, like 'm2'? That's totally possible! You can request it by filing a ticket at http://cldr.unicode.org/. The proposal in the ticket MUST contain a full description of the proposed field semantics and subtag syntax, and MUST conform to the ABNF syntax for "field" presented in Section 2.2 🧚

2.8. Committee Responses to Registration Proposals 🔥

The committee MUST post each proposal publicly within 2 weeks after reception, so people can comment 🫶 The committee must respond publicly to each proposal within 4 weeks after reception ⏰

The response MAY:

  • request more information or clarification
  • accept the proposal, optionally with modifications to the subtag or description
  • reject the proposal, because of significant objections raised on the mailing list or due to problems with constraints in this document or in [UTS35] 💀

Accepted tickets result in a new entry in the machine-readable CLDR BCP 47 data or, in the case of a clarified description, modifications to the description attribute value for an existing entry 🌈

2.9. Machine-Readable Data 📚

okay so, starting with CLDR 1.7.2, there are machine-readable files that list the data defined for BCP 47 extensions for each version of [UTS35] 🫶. This data is used to test whether subtags are valid for the 't' extension and the 'u' extension [RFC6067], and the Unicode Consortium is the maintaining authority for both 💅. The releases are listed at http://cldr.unicode.org/index/downloads. Each release has its own data directory of the form "http://unicode.org/Public/cldr/<version>", where <version> gets swapped for the release number. For example, for version 1.7.2, the core.zip file lives at http://unicode.org/Public/cldr/1.7.2/core.zip ✨. The newest version is always labelled "latest", and you can get to it with the URL in Section 2.4 🌈.

Inside core.zip, the directory common/bcp47 holds the data files listing the valid attributes, keys, and types for each version of [UTS35]. Each file lists the keys and types for its own topic 🌸.

The XML structure lists keys, like <key extension="t" name="m0" description="Transliteration extension mechanism">, with subelements for types, like <type name="ungegn" description="United Nations Group of Experts on Geographical Names"/>. The attributes that are currently defined for the mechanisms are:

+-------------+-------------------------------+---------------------+
| Attribute   | Description                   | Examples            |
+-------------+-------------------------------+---------------------+
| name        | The name of the mechanism,    | UNGEGN, ALALC       |
|             | limited to 3-8 characters (or |                     |
|             | sequences of them).           |                     |
| description | A description of the name,    | United Nations      |
|             | with all and only that        | Group of Experts on |
|             | information necessary to      | Geographical Names; |
|             | distinguish one name from     | American Library    |
|             | others with which it might be | Association-Library |
|             | confused.  Descriptions are   | of Congress         |
|             | not intended to provide       |                     |
|             | general background            |                     |
|             | information.                  |                     |
| since       | Indicates the first version   | 1.9, 2.0.1          |
|             | of CLDR where the name        |                     |
|             | appears.  (Required for new   |                     |
|             | items.)                       |                     |
| alias       | Alternative name of the key   |                     |
|             | or type, not limited in       |                     |
|             | number of characters.         |                     |
|             | Aliases are intended for      |                     |
|             | backwards compatibility, not  |                     |
|             | to provide all possible       |                     |
|             | alternate names or            |                     |
|             | designations.  (Optional.)    |                     |
+-------------+-------------------------------+---------------------+

The file for the transform extension is "transform.xml" 🦋. Its initial version has the following information:

<keyword>
  <key extension="t" name="m0" description=
      "Transliteration extension mechanism">
    <type name="ungegn" description=
        "United Nations Group of Experts on Geographical Names"
        since="21"/>
    <type name="alaloc" description=
        "American Library Association-Library of Congress"
        since="21"/>
    <type name="bgn" description=
        "US Board on Geographic Names"
        since="21"/>
    <type name="mcst" description=
        "Korean Ministry of Culture, Sports and Tourism"
        since="21"/>
    <type name="iso" description=
        "International Organization for Standardization"
        since="21"/>
    <type name="din" description=
        "Deutsches Institut fuer Normung"
        since="21"/>
    <type name="gost" description=
        "Euro-Asian Council for Standardization, Metrology
         and Certification"
        since="21"/>
  </key>
</keyword>

To get the version information in XML when you're working with the data files, the XML parser must be validating 🧐. When the 'core.zip' file is unzipped, the 'dtd' directory has to sit at the same level as the 'bcp47' directory; this is required for correct validation 💜. For each release after CLDR 1.8, any types introduced in that release are also marked in the data files with the XML attribute "since", like in this example:

<type name="adp" since="1.9"/>

The data is also kept in a source code repository, with each release tagged, so you can view it directly without unzipping 🍓. For example, see:

For more information, see http://cldr.unicode.org/index/bcp47-extension 🌸.

3. Acknowledgements 🎀

Huge thanks to John Emmons and the rest of the Unicode CLDR Technical Committee for their work developing the BCP 47 subtags for LDML 🙌.

4. IANA Considerations 📋

IANA has inserted the record of Section 2.4 into the Language Extensions Registry, following Section 3.7 ("Extensions and the Extensions Registry") of "Tags for Identifying Languages" [BCP47] 🗂️. Per Section 5.2 of [BCP47], there might be occasional (rare) requests by the Unicode Consortium (the "Authority" listed in the record) to maintain this record. Changes that can be submitted to IANA without publishing a new RFC are limited to modifying the Comments, Contact_Email, Mailing_List, and URL fields. Any such requested change MUST use the domain 'unicode.org' in any new addresses or URIs, MUST explicitly cite this document (so that IANA can reference these requirements), and MUST originate from the 'unicode.org' domain. The domain or authority can only be changed via a new RFC 🫣.

5. Security Considerations 🔒

The security considerations for this extension are the same as those for [BCP47]. See RFC 5646, Section 6, Security Considerations [BCP47] 🛡️. Please go read that section, because security is not something to skim ‼️.

6. References 📚

6.1. Normative References

[BCP47]    Phillips, A., Ed., and M. Davis, Ed., "Tags for
           Identifying Languages", BCP 47, RFC 5646, September 2009.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC5234]  Crocker, D., Ed., and P. Overell, "Augmented BNF for
           Syntax Specifications: ABNF", STD 68, RFC 5234,
           January 2008.

[UTS35]    Davis, M., "Unicode Technical Standard #35: Locale Data
           Markup Language (LDML)", February 2012,
           <http://www.unicode.org/reports/tr35/>.

6.2. Informative References

[RFC3339]  Klyne, G. and C. Newman, "Date and Time on the Internet:
           Timestamps", RFC 3339, July 2002.

[RFC6067]  Davis, M., Phillips, A., and Y. Umaoka, "BCP 47
           Extension U", RFC 6067, December 2010.

[W3C-TimeZones]
           Phillips, Ed., "W3C Working Group Note: Working with Time
           Zones", July 2011,
           <http://www.w3.org/TR/2011/NOTE-timezone-20110705/>.

Authors' Addresses ✉️

Mark Davis
Google

EMail: mark@macchiato.com


Addison Phillips
Lab126

EMail: addison@lab126.com


Yoshito Umaoka
IBM

EMail: yoshito_umaoka@us.ibm.com


Courtney Falk
Infinite Automata

EMail: court@infiauto.com
📦 where this page comes from