RFC 6365
Terminology Used in Internationalization in the IETF
RFC 6365: Terminology Used in Internationalization in the IETF 🎀
Okay bestie, let's start with the title block 💅 This is RFC 6365 from the Internet Engineering Task Force (IETF), written by P. Hoffman (VPN Consortium) and J. Klensin, and dated September 2011. It's a Best Current Practice, which makes it BCP 166 👑 Its ISSN is 2070-1721. It obsoletes RFC 3536, so 3536 is officially retired 🫡 That's the whole title block, no cap 🧚
Abstract ✨
Okay so the abstract says this doc gives you a list of terms the IETF uses when it talks about internationalization 🌈 The goal is to help frame those chats across all the IETF areas and to introduce the main concepts to IETF people 🫶
Status of This Memo 💜
This memo documents an Internet Best Current Practice.
This doc is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community 🌸 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on BCPs, check out Section 2 of RFC 5741 ✨
For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc6365 🦋
Copyright Notice 🎀
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions with respect to this document 🙈 Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 💖
Table of Contents 🧚
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Purpose of this Document . . . . . . . . . . . . . . . . . 3
1.2. Format of the Definitions in This Document . . . . . . . . 4
1.3. Normative Terminology . . . . . . . . . . . . . . . . . . 4
2. Fundamental Terms . . . . . . . . . . . . . . . . . . . . . . 5
3. Standards Bodies and Standards . . . . . . . . . . . . . . . . 10
3.1. Standards Bodies . . . . . . . . . . . . . . . . . . . . . 11
3.2. Encodings and Transformation Formats of ISO/IEC 10646 . . 13
3.3. Native CCSs and Charsets . . . . . . . . . . . . . . . . . 15
4. Character Issues . . . . . . . . . . . . . . . . . . . . . . . 16
4.1. Types of Characters . . . . . . . . . . . . . . . . . . . 20
4.2. Differentiation of Subsets . . . . . . . . . . . . . . . . 23
5. User Interface for Text . . . . . . . . . . . . . . . . . . . 24
6. Text in Current IETF Protocols . . . . . . . . . . . . . . . . 27
7. Terms Associated with Internationalized Domain Names . . . . . 31
7.1. IDNA Terminology . . . . . . . . . . . . . . . . . . . . . 31
7.2. Character Relationships and Variants . . . . . . . . . . . 32
8. Other Common Terms in Internationalization . . . . . . . . . . 33
9. Security Considerations . . . . . . . . . . . . . . . . . . . 36
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 37
10.1. Normative References . . . . . . . . . . . . . . . . . . . 37
10.2. Informative References . . . . . . . . . . . . . . . . . . 37
Appendix A. Additional Interesting Reading . . . . . . . . . . . 41
Appendix B. Acknowledgements . . . . . . . . . . . . . . . . . . 42
Appendix C. Significant Changes from RFC 3536 . . . . . . . . . . 42
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431. Introduction 🌸
So the IETF Character Set Policy spec [RFC2277] sums it up perfectly: "Internationalization is for humans. This means that protocols are not subject to internationalization; text strings are." Iconic 😭
Tons of IETF protocols use text strings that humans type in or look at. Within the limits of their own knowledge and tools, anyone is meant to be able to enter or read these strings. That means Internet users need to type text with normal input methods and see it displayed in any human language 🌍 Plus, text with any character should be able to zip between Internet apps easily ✨ That's the internationalization challenge, and it's a big one 💅
1.1. Purpose of this Document 💖
This doc gives you a glossary of terms the IETF uses when it talks about internationalization. The point is to help frame those discussions across the different IETF areas and to introduce the main concepts to IETF participants 🦋
Internationalization comes up in lots of IETF working groups, but few of them have internationalization experts 🙈 When people design or update protocols, the question pops up constantly: "Do we internationalize this?" (or, more likely, "Do we have to internationalize this?").
This doc gives a light overview of internationalization terminology as it applies to IETF standards work. It covers many aspects of internationalization and the vocabulary attached to them. Some of the overview is kind of tutorial in nature 📚 It is not meant to be a complete description of internationalization. The definitions here SHOULD be used by IETF standards. IETF standards that explicitly want to create different definitions for these terms can do so, but unless an alternate definition is provided, the definitions in this document apply. IETF standards that need different definitions are encouraged, for clarity's sake, to pick terms that are different from the ones defined here 🌈 Some of the definitions in this document come from earlier IETF documents and books.
As in many fields, the internationalization community disagrees on the definitions of lots of words 🔥 The topic of language brings out super passionate opinions from experts and non-experts alike. This document tries to define terms in the way that's most useful to the IETF audience 💅
This doc uses definitions from many documents, both inside and outside the IETF. The main documents it uses are:
- ISO/IEC 10646 [ISOIEC10646]
- The Unicode Standard [UNICODE]
- W3C Character Model [CHARMOD]
- IETF RFCs, including the Character Set Policy specification [RFC2277] and the domain name internationalization standard [RFC5890]
1.2. Format of the Definitions in This Document 🎀
In the body of this doc, the source of each definition is shown in angle brackets, like <ISOIEC10646>. Lots of definitions are shown as <RFC6365>, which means they were originally written for this document ✨ The angle bracket notation for definition sources is different from the square bracket notation used for references to documents, like the ones in the paragraph above. Those references are listed in the reference sections of this document 🦋
Some terms come with commentary and examples after the definitions. In those cases, the part before the angle brackets is the definition from the original source. The part after the angle brackets is commentary, not a definition. It might be an example or some extra explanation 🌸
Examples in this doc use the notation for code points and names from the Unicode Standard [UNICODE] and ISO/IEC 10646 [ISOIEC10646]. For example, the letter "a" may be represented as either U+0061 or LATIN SMALL LETTER A. See RFC 5137 [RFC5137] for a description of this notation 🧚
1.3. Normative Terminology 💜
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
2. Fundamental Terms ✨
okay bestie, this section is the vocab foundation 🎀 Basically everyone who wants IETF protocols to play nicely with non-ASCII text (see Section 4.2) needs these terms down, plus the rest of the internationalization world. Let's go 💅
language
A language is a way that humans communicate 🗣️ It shows up in a bunch of forms, and the most common are speech, writing, and signing. <RFC6365>
Some languages have a super close relationship between the written and spoken forms, and others are more loosey-goosey about it 🌸 The LTRU (Language Tag Registry Update) standards [RFC5646] [RFC4647] go deeper on languages and give you identifiers to use in Internet protocols. Also, computer languages are explicitly excluded from this definition, so no, Python doesn't count here 💀
script
A set of graphic characters used for the written form of one or more languages. <ISOIEC10646>
Examples of scripts are Latin, Cyrillic, Greek, Arabic, and Han (the characters, often called ideographs after a subset of them, used in writing Chinese, Japanese, and Korean) 🌈 RFC 2277 covers scripts in detail if you want the full tea ☕
It's super common for internationalization newbies to mix up "language" and "script" 🙈 and that can cause real problems in protocols that tell the two apart. Almost every protocol designed (or redesigned) to handle non-ASCII text deals with scripts (the written systems) or characters, while way fewer actually deal with languages.
A single name can mean either a language or a script. "Arabic," for example, is both a language name and a script name 💜 Plenty of scripts borrow their names from languages, too. Also, lots of scripts are used to write more than one language. Russian and Bulgarian, for instance, are both written in the Cyrillic script ✨
Some languages can be written in different scripts, or switched scripts over time. Mongolian can be written in either the Mongolian or Cyrillic scripts 🦋 Malay is mostly written in Latin script today, but the older Arabic-script-based Jawa form is still around. And a bunch of languages got converted from other scripts to Cyrillic in the first half of the last century, and some of them have switched again more recently 😭
Then there are languages that use more than one script at the same time. Japanese is normally written with Kanji (Han), Katakana, and Hiragana all in a single string of text 🎌 that's iconic behavior, honestly.
writing system
A set of rules for using one or more scripts to write a particular language. Examples include the American English writing system, the British English writing system, the French writing system, and the Japanese writing system. <UNICODE>
character
A member of a set of elements used for the organization, control, or representation of data. <ISOIEC10646>
There are at least three common definitions of "character," and they're all kind of a vibe 🧚
- a general description of a text entity
- a unit of a writing system, often synonymous with "letter" or similar terms, but generalized to include digits and symbols of various sorts
- the encoded entity itself
When people talk about characters, they usually mean one of the first two definitions 💖 "Character" is often shortened to "char."
A particular character is identified by its name, not by its shape 🍓 A name might suggest a meaning, but the character can be used for other meanings too. And a name might suggest a shape, but that doesn't mean that's the only shape you'll see in print, or that the shape belongs only to that name 🙃
coded character
A character together with its coded representation. <ISOIEC10646>
coded character set
A coded character set (CCS) is a set of unambiguous rules that establishes a character set and the relationship between the characters of the set and their coded representation. <ISOIEC10646>
character encoding form
A character encoding form is a mapping from a coded character set (CCS) to the actual code units used to represent the data. <UNICODE>
repertoire
The collection of characters included in a character set. Also called a character repertoire. <UNICODE>
glyph
A glyph is an image of a character that can be displayed after being imaged onto a display surface. <RFC6365>
The Unicode Standard has a different definition, one that refers to an abstract form that may show up as different images when the same character is rendered under different circumstances 🎨
glyph code
A glyph code is a numeric code that refers to a glyph. Usually, the glyphs contained in a font are referenced by their glyph code. Glyph codes are local to a particular font; that is, a different font containing the same glyphs may use different codes 🔤
transcoding
Transcoding is the process of converting text data from one character encoding form to another. Transcoders work only at the level of character encoding and do not parse the text. Note: Transcoding may involve one-to-one, many-to-one, one-to-many, or many-to-many mappings. Because some legacy mappings are glyphic, they may not only be many-to-many, but also unordered: thus XYZ may map to yxz. <CHARMOD>
Quick aside: in this definition, "many-to-one" means a sequence of characters mapped to a single character. The "many" does not mean alternative characters that map to the single character 🧠
character encoding scheme
A character encoding scheme (CES) is a character encoding form plus byte serialization. There are many character encoding schemes in Unicode, such as UTF-8 and UTF-16BE. <UNICODE>
Some CESs are tied to a single CCS. For example, UTF-8 [RFC3629] applies only to the identical CCSs of ISO/IEC 10646 and Unicode 💅 Other CESs, like ISO 2022, are associated with many CCSs.
charset
A charset is a method of mapping a sequence of octets to a sequence of abstract characters. A charset is, in effect, a combination of one or more CCSs with a CES. Charset names are registered by the IANA according to procedures documented in [RFC2978]. <RFC6365>
Lots of protocol definitions use the term "character set" in their descriptions 👀 But "charset," or "character encoding scheme" and "coded character set," are strongly preferred over "character set" because "character set" has other definitions in other contexts, particularly outside the IETF. When you're reading IETF standards that use "character set" without defining it, they usually mean "a specific combination of one CCS with a CES," especially when they're talking about the "US-ASCII character set" 🎀
internationalization
In the IETF, "internationalization" means to add or improve the handling of non-ASCII text in a protocol. <RFC6365> A different perspective, more appropriate for protocols designed for global use from the beginning, is the definition used by W3C:
"Internationalization is the design and development of a product, application or document content that enables easy localization for target audiences that vary in culture, region, or language." [W3C-i18n-Def]
Many protocols that handle text only handle one charset (US-ASCII), or they leave the question of what CCS and encoding get used up to local guesswork, which of course leads to interoperability problems 😤 If multiple charsets are permitted, they must be explicitly identified [RFC2277]. Adding non-ASCII text to a protocol lets it handle more scripts, hopefully all of the useful ones in the world 🌍 In today's world, that's normally best done by allowing Unicode encoded in UTF-8 only, which shifts conversion issues away from individual choices 🫶
localization
The process of adapting an internationalized application platform or application to a specific cultural environment. In localization, the same semantics are preserved while the syntax may be changed. [FRAMEWORK]
Localization is the act of tailoring an application for a different language or script or culture 🌸 Some internationalized applications can handle a wide variety of languages. Typical users only understand a small number of languages, though, so the program has to be tailored to interact with users in just the languages they know.
The major work of localization is translating the user interface and documentation. It also involves other relevant changes, like how numbers, dates, currency, and so on are displayed 💸 The better internationalized an application is, the easier it is to localize for a particular language and character encoding scheme.
Localization is rarely an IETF matter 🙌 Protocols that are merely localized, even if they're serially localized for several locations, are generally considered unsatisfactory for the global Internet.
Don't confuse "localization" with "locale," which is described in Section 8 of this document ⚠️
i18n, l10n
These are abbreviations for "internationalization" and "localization". <RFC6365>
"18" is the number of characters between the "i" and the "n" in "internationalization," and "10" is the number of characters between the "l" and the "n" in "localization" 🔢 iconic counting, no notes.
multilingual
The term "multilingual" has many widely varying definitions and thus is not recommended for use in standards 🙅♀️ Some definitions relate to the ability to handle international characters, others relate to the ability to handle multiple charsets, and still others relate to the ability to handle multiple languages. <RFC6365>
displaying and rendering text
To display text, a system puts characters on a visual display device such as a screen or a printer. To render text, a system analyzes the character input to determine how to display the text. The terms "display" and "render" are sometimes used interchangeably. Note, however, that text might be rendered as audio and/or tactile output, such as in systems designed for people with visual disabilities 🦻 <RFC6365>
Combining characters modify the display of the character (or, in some cases, characters) that precede them 💫 When rendering this kind of text, the display engine must either find the glyph in the font that represents the base character and all of the combining characters, or it must render the combination itself. That can be straightforward, but it gets messy when the combining marks interact with each other, like when two combining marks would both appear above the same character 😵 Formatting characters can also change the way a renderer displays text. And rendering can be genuinely difficult for scripts with complex display rules for base characters, such as Arabic and Indic scripts 🌙
3. Standards Bodies and Standards ✨
okay so this section is about the standards bodies and standards that keep popping up when people talk about internationalization in the IETF 🌍 it's an incomplete list and maybe a little overstuffed, but listing too few bodies or standards is just as politically dangerous as listing too many 🫣 also there are loads of other bodies working on internationalization, but few if any of them show up much in IETF standards work 💅
3.1. Standards Bodies
ISO and ISO/IEC JTC 1
okay bestie, the International Organization for Standardization has been dealing with character standards since before the IETF even existed 🎀 ISO is a non-governmental group made up of national bodies. Most of ISO's IT work gets done jointly with a similar body, the International Electrotechnical Commission (IEC), through a joint committee called "JTC 1" 🤝 The one the IETF uses most is usually called "ISO/IEC 10646", sometimes with a specific date attached. ISO/IEC 10646 describes a CCS that covers almost every written character in use today, so it's giving main character energy 👑
ISO/IEC 10646 is controlled by the group known as "ISO/IEC JTC 1/SC 2 WG2", which people usually shorten to "SC2/WG2" or just "WG2" 🧚 ISO standards go through a lot of steps before they're finished, and years can pass between changes to the base ISO/IEC 10646 standard. Amendments are issued now to track Unicode changes though, which is a nice glow-up ✨ You can find info on WG2 and its work products at http://www.dkuug.dk/JTC1/SC2/WG2/. You can find info on SC2 and its work products at http://www.iso.org/iso/standards_development/technical_committees/list_of_iso_technical_committees/iso_technical_committee.htm?commid=45050
The standard comes as a base part plus a series of attachments or amendments. It's available as a PDF to download or as a CD-ROM version 💿 One example of how to cite the standard is in [RFC3629]. Any standard that cites ISO/IEC 10646 needs to figure out how to handle the versioning problem that matters for the protocol's needs, so don't skip that part, bestie 🙈
ISO is responsible for other standards that protocol developers who care about internationalization might be into too 💖 ISO 639 [ISO639] specifies the names of languages, and it forms part of the basis for the IETF's Language Tag work [RFC5646]. ISO 3166 [ISO3166] specifies the names and code abbreviations for countries and territories. It's used in several protocols and databases, including names for country-code top level domain names 🌏 The responsibilities of ISO TC 46 on Information and Documentation include a series of standards for transliterating various languages into Latin characters, which is so useful for anyone who's ever typed a name without the fancy accents 🫶 You can find TC 46 at http://www.iso.org/iso/standards_development/technical_committees/list_of_iso_technical_committees/iso_technical_committee.htm?commid=48750
Another ISO group that was relevant was JTC 1/SC22/WG20, which was in charge of internationalization in JTC 1, like for international string ordering 🔤 You can find info on WG20 and its work products at http://www.dkuug.dk/jtc1/sc22/wg20/. The specific tasks of SC22/WG20 were moved from SC22 into SC2, and there hasn't been much significant activity since that happened 😴
Unicode Consortium
The second big group for international character standards is the Unicode Consortium 🌈 It's a trade association of companies, governments, and other groups who are into promoting the Unicode Standard [UNICODE]. The Unicode Standard is a CCS whose repertoire and code points are identical to ISO/IEC 10646. Unicode has added features on top of the base CCS that make it more useful in protocols, like defining attributes for each character. Examples of these attributes include case conversion and numeric properties 🔥
The actual technical and definitional work of the Unicode Consortium is done in the Unicode Technical Committee (UTC). "UTC" and "Unicode Consortium" get treated as synonyms in the IETF all the time, which is imprecise, but we all do it 🙈
The Unicode Consortium publishes addenda to the Unicode Standard as Unicode Technical Reports. There are a bunch of different types of technical reports at different stages of maturity 📚 The Unicode Standard and its affiliated technical reports live at http://www.unicode.org/.
There's a reciprocal agreement between the Unicode Consortium and ISO/IEC JTC 1/SC 2 so that ISO/IEC 10646 and The Unicode Standard track each other for character definitions and code point assignments 🤝 Updates to the former, often as amendments, sometimes lag behind updates to the latter for a short while, but the gap has rarely been significant in recent years ✨
When the IETF character set policy [RFC2277] was established, and when the first version of this terminology specification was published, the IETF community had a strong preference for referencing ISO/IEC 10646 rather than Unicode whenever possible 🎀 That preference mostly reflected a broader IETF love for established open international standards over consortium specs. BUT the Unicode definitions of character properties and classes are not part of ISO/IEC 10646, and IETF specs are increasingly depending on those definitions (see the explanation in Section 4.2) 👀 The Unicode specs are also freely available online in a super convenient machine-readable form, so the IETF's preference has shifted to referencing the Unicode Standard. This matters most when version consistency between code points (either standard) and Unicode properties (Unicode only) is required 💅
World Wide Web Consortium (W3C)
This group created and maintains the standard for XML, the markup language for text that became super popular 💜 XML has always been fully internationalized, so there's no need for a new version just to handle international text 🌸 Still, in some circumstances, XML files may be sensitive to differences among Unicode versions, so keep an eye on that 👀
local and regional standards organizations
Just like there are lots of native CCSs and charsets, there are lots of local and regional standards organizations to create and support them 🗺️ Common examples include ANSI (United States), CEN/ISSS (Europe), JIS (Japan), and SAC (China) 🧋
3.2. Encodings and Transformation Formats of ISO/IEC 10646
okay so characters in the ISO/IEC 10646 CCS can be expressed in a bunch of different ways 🦋 Historically, "encoding forms" were direct addressing methods, while "transformation formats" were methods for expressing encoding forms as bits on the wire. That distinction has mostly disappeared in recent years, so don't stress too hard about it 💅
Documents that talk about characters in the ISO/IEC 10646 CCS often need to list specific characters. RFC 5137 describes the common methods for doing that in IETF documents, and lots of other communities have adopted these practices too 🫶
Basic Multilingual Plane (BMP)
The BMP is made up of the first 2^16 code points in ISO/IEC 10646 and holds almost all characters in contemporary use 💖 It's also called "Plane 0" 🧚
UCS-2 and UCS-4
UCS-2 and UCS-4 are the two encoding forms that were historically defined for ISO/IEC 10646 🕰️ UCS-2 only addresses the BMP. Because lots of useful characters (like many Han characters) were defined outside the BMP, lots of people consider UCS-2 to be obsolete 💀
UCS-4 addresses the entire range of code points from ISO/IEC 10646 (by agreement between ISO/IEC JTC 1 SC2 and the Unicode Consortium, a range from 0..0x10FFFF) as 32-bit values with zero padding to the left 🔢 UCS-4 is identical to UTF-32BE (without use of a BOM (see below)), and UTF-32BE is now the preferred term 🎀
UTF-8
UTF-8 [RFC3629] is the preferred encoding for IETF protocols, and it's giving literally everything 👑 Characters in the BMP are encoded as one, two, or three octets. Characters outside the BMP are encoded as four octets. Characters from the US-ASCII repertoire have the same on-the-wire representation in UTF-8 as they do in US-ASCII, so old-school stuff stays compatible 🥰 The IETF-specific definition of UTF-8 in RFC 3629 is identical to the one in recent versions of the Unicode Standard (for example, Section 3.9 of Version 6.0 [UNICODE]) ✨
UTF-16, UTF-16BE, and UTF-16LE
UTF-16, UTF-16BE, and UTF-16LE are three transformation formats described in [RFC2781] and defined in The Unicode Standard (Sections 3.9 and 16.8 of Version 6.0). They're not required by any IETF standards, so they show up way less in protocols than UTF-8 does 🙃 Characters in the BMP are always encoded as two octets, and characters outside the BMP are encoded as four octets using a "surrogate pair" arrangement. The surrogate pair thing isn't part of UCS-2, and that's exactly the difference between UTF-16 and UCS-2 🧩 The three UTF-16 formats differ based on the order of the octets and on whether there's a special lead-in ordering identifier called the "byte order mark" or "BOM" 🔍
UTF-32
The Unicode Consortium and ISO/IEC JTC 1 defined UTF-32 as a transformation format that puts the integer code point value right-justified in a 32-bit field 📐 Like UTF-16, the byte order mark (BOM) can be used, and UTF-32BE and UTF-32LE are defined too. UTF-32 and UCS-4 are essentially equivalent, and people often use the terms interchangeably 🦄
SCSU and BOCU-1
The Unicode Consortium defined an encoding, SCSU [UTR6], that's designed to offer good compression for typical text 🗜️ A different encoding meant to be MIME-friendly, BOCU-1, is described in [UTN6]. Compression is attractive compared to UTF-8, but at the time of this writing, neither of these has attracted much interest 😬
The compression that comes as a side effect of the Punycode algorithm [RFC3492] is heavily used in some contexts, especially IDNA [RFC5890], but it does come with some restrictions 🫣 (See also Section 7.)
3.3. Native CCSs and Charsets 🌸
Okay so, before ISO/IEC 10646 even existed, lots of countries were making their own CCSs and charsets 🌍 Some of them got adopted into international standards for their scripts or writing systems, and dozens are still super popular on the Internet today 💅 Examples include ISO 8859-5 for Cyrillic and Shift-JIS for Japanese scripts 🍓
The official list of registered charset names for IETF protocols is kept by IANA, and you can find it at <http://www.iana.org/assignments/character-sets> ✨ It has preferred names and aliases, but ngl the list has a history of errors 🙈 like names that aren't actually charsets, or references that don't give enough detail to reliably map a name to a charset 💀
Probably the most famous native CCS is ASCII [US-ASCII] 👑 It's the basis for keywords and parameter names in tons of IETF protocols, and it's the only CCS in plenty of IETF protocols that haven't been internationalized yet. ASCII also became the basis for ISO/IEC 646, which then formed the basis for many national and international standards, like the ISO 8859 series, that mix Basic Latin characters with characters from another script 🌈
This next bit is important 🫶 Strictly speaking, "ASCII" is a CCS and repertoire, not an encoding! 🤯 The encoding used for ASCII in IETF protocols puts the 7-bit integer ASCII code point right-justified in an 8-bit field, and it's sometimes called the "Network Virtual Terminal" or "NVT" encoding [RFC5198] 🦋 Less formally, people use "ASCII" and "NVT" interchangeably. But "non-ASCII" only refers to characters outside the ASCII repertoire and isn't tied to any specific encoding. See Section 4.2.
A Unicode publication covers the issues involved in mapping character data between charsets, and it includes an XML format for mapping table data [UTR22] 📚
4. Character Issues 💜
This section is all about terms and topics that come up constantly in character handling, so they matter if you're adding non-ASCII text handling to a protocol 🎀 These topics are standardized outside the IETF, so think of this as a friendly tour 🧚
code point ✨
A code point is a value in the codespace of a repertoire. For all the common repertoires developed in recent years, code point values are integers 🔢 (Code points for ASCII and its immediate descendants were defined in terms of column and row positions of a table 📊)
combining character 🌸
A combining character is a member of an identified subset of the coded character set of ISO/IEC 10646. It's meant to be combined with the preceding non-combining graphic character, or with a sequence of combining characters that follows a non-combining character. Combining characters are inherently non-spacing. <ISOIEC10646>
composite sequence or combining character sequence 🦄
A composite sequence is a sequence of graphic characters made up of a non-combining character followed by one or more combining characters 🧩 A graphic symbol for a composite sequence generally combines the graphic symbols of each character in the sequence. The Unicode Standard often uses the term "combining character sequence" for these. A composite sequence isn't a character, so it isn't a member of the repertoire of ISO/IEC 10646 <ISOIEC10646> 😳 However, Unicode now assigns names to some of these sequences, especially when the names need to match terminology in other standards [UAX34].
In some CCSs, some characters are made of combinations of other characters 🧃 For example, the letter "a with acute" might be a combination of the two characters "a" and "combining acute", or it might be a combination of the three characters "a", a non-destructive backspace, and an acute 😵 In the same or other CCSs, it might be available as a single code point 🙌 The rules for combining two or more characters are called "composition rules", and the rules for taking a character apart into other characters are called "decomposition rules". The result of decomposition is called a "decomposed character", and the result of composition is usually a "precomposed character" 🍰
normalization 🧼
Normalization is the transformation of data to a normal form, for example, to unify spelling. <UNICODE>
Quick note: the phrase "unify spelling" above does not mean unifying different strings that have the same meaning as words, like "color" and "colour" 🇬🇧🇺🇸 Instead, it means unifying different character sequences that are meant to form the same composite characters, like <n><combining tilde> and <n with tilde> (where <n> is U+006E, <combining tilde> is U+0303, and <n with tilde> is U+00F1) 🦋
The whole point of normalization is letting two strings be compared for equivalence 👯 The strings <a><n><combining tilde><o> and <a><n with tilde><o> would look identical on a text display device. If a protocol designer wants those two strings to count as equivalent during comparison, the protocol must define where normalization occurs.
"Normalization" and "canonicalization" get used interchangeably a lot 🙃 Generally, they both mean converting a string of one or more characters into another string based on standardized rules. But in Unicode, "canonicalization" or similar terms refer to a particular type of normalization equivalence ("canonical equivalence" as opposed to "compatibility equivalence"), so you've gotta use the term with some care ⚠️ Some CCSs allow multiple equivalent representations for a written string, and normalization picks one of them as a base for reference when comparing strings. In text, these rules are usually based on decomposing combined characters or composing characters with combining characters. Unicode Standard Annex #15 [UTR15] describes the process and many forms of normalization in detail. Normalization matters when you're comparing strings to see if they're the same ✅
The Unicode NFC and NFD normalizations support canonical equivalence, and NFKC and NFKD support canonical and compatibility equivalence 🌟
case 🔠
Case is the feature of certain alphabets where letters have two (or occasionally more) distinct forms 💫 These forms can differ a lot in shape and size, and they're called the uppercase letter (also known as capital or majuscule) and the lowercase letter (also known as small or minuscule). Case mapping is the association of the uppercase and lowercase forms of a letter. <UNICODE>
There's usually (but not always) a one-to-one mapping between the same letter in the two cases 😅 But there are plenty of characters that exist in one case with no corresponding character in the other, or that have a special mapping rule, like the Turkish dotless "i", some Greek characters with modifiers, and characters like the German Sharp S (Eszett) and the Greek Final Sigma, which traditionally don't have uppercase forms 🤪 Case mapping can even depend on locale or language. Converting text to have only a single case, mainly for comparison purposes, is called "case folding". Because of all these unusual cases, case mapping can be pretty controversial, and some case folding algorithms even more so 🥴 For example, some programming languages, like Java, have case-folding algorithms that are locale-sensitive. That makes those algorithms incredibly resource-intensive, and they act differently depending on where the system is located at the time the algorithm runs 🌍💀
sorting and collation 📚
Collating is the process of ordering units of textual information. Collation is usually specific to a particular language or even to a particular application or locale. It's sometimes known as alphabetizing, although alphabetization is just a special case of sorting and collation. <UNICODE>
Collation is all about figuring out the relative order of any particular pair of strings 🧠 Algorithms for collation focus on providing appropriate weighted keys for string values, so that binary comparison of the key values can determine the relative ordering of the strings.
The relative orders of letters in collation sequences can differ wildly depending on what the system or protocol defining the collation order needs 🎢 For example, even within ASCII characters there are two common and very different collation orders: A, a, B, b,... and A, B, C, ..., Z, a, b,..., with extra variations for lowercase first and for digits before or after letters.
In practice, it's rarely necessary to define a collation sequence for characters drawn from different scripts, but arranging those sequences so they don't surprise users is usually a particularly tricky problem 😬
Sorting is the process of actually putting data records into a specified order, based on criteria for comparing the records 🗂️ Sorting can apply to any kind of data, including textual data, as long as an ordering criterion can be defined. Algorithms for sorting focus on performance (in terms of time, memory, or other resources) when they're actually putting the data records into the desired order ⏱️
A sorting algorithm for string data can be internationalized by giving it the appropriate collation-weighted keys for the strings to be ordered 🌐
Lots of processes need to order strings in a consistent (sorted) sequence 📋 For just a few CCS/CES combinations, there's an obvious sort order that can be applied without reference to the linguistic meaning of the characters. The code point order is enough for sorting there, which means it's also the order a person would use when sorting those characters 🙌 For many CCS/CES combinations, though, the code point order would make no sense to a person, so it isn't useful for sorting if the results are going to be displayed to someone 🫠
Code point order is usually not how any human educated by a local school system expects strings to be ordered. If you order to the expectations of a human, you get a "language-specific" or "human language" sort 🎓 Sorting by code point order will look inconsistent if the strings aren't normalized before sorting, because different representations of the same character will sort differently 😤 This problem may be smaller with a language-specific sort ✨
code table 🗺️
A code table is a table showing the characters allocated to the octets in a code. <ISOIEC10646>
Code tables are also commonly called "code charts" 🎀
4.1. Types of Characters 🎀
Okay so this section is basically a tour of all the vibes text can give 💅 The definitions here do NOT neatly put every character into one box, and you can't predict a character's type just by looking at it 🙈 They're meant for app designers to think about the many (sometimes confusing!) properties of text 💖
alphabetic
An informative Unicode property ✨ These are the main building blocks of alphabets and/or syllabaries, whether they're combining or non-combining. It includes composite characters that are canonical equivalents to a combining character sequence made of an alphabetic base character plus one or more combining characters: letter digraphs, contextual variants of alphabetic characters, ligatures of alphabetic characters, contextual variants of ligatures, modifier letters, letterlike symbols that are compatibility equivalents of single alphabetic letters, and miscellaneous letter elements. [UNICODE]
ideographic
Any symbol that mainly shows an idea (or meaning) rather than a sound (or pronunciation) 🧚 Think of a symbol showing a telephone, or the Han characters used in Chinese, Japanese, and Korean. [UNICODE]
Unicode and lots of other systems use this word for every Han character, but strictly speaking, not all of them are actually ideographic 🙈 Some are pictographic (like the telephone example), some are used phonetically, and so on. Still, the convention is to call the script ideographic when you're contrasting it with alphabetic, so that's the vibe we go with 🌸
digit or number
Every modern writing system uses decimal digits in some form 🔢 Some older ones use non-positional or other systems, and different scripts can have their own digits. Unicode tells numbers apart from other characters by giving them a special General Category value, then splitting that value further into decimal digits, letter digits, and other digits. [UNICODE]
punctuation
Characters that separate units of text, like sentences and phrases, so the meaning stays clear 📖 Punctuation isn't just for prose either. It shows up in mathematical and scientific formulae too, for example. [UNICODE]
symbol
One of a set of characters that aren't used for letters, digits, or punctuation, and that stand for various concepts not usually tied to written language use itself 🦋 [RFC6365]
Examples of symbols include characters for mathematical operators, symbols for optical character recognition (OCR), symbols for box-drawing or graphics, and symbols for dingbats, arrows, faces, and geometric shapes 🌈 Unicode has a property that identifies symbol characters too.
nonspacing character
A combining character whose position in presentation depends on its base character 🫶 By itself, it generally doesn't take up any space along the visual baseline. [UNICODE]
For example, a combining acute accent (U+0301) is a nonspacing character ✨
diacritic
A mark that's applied or attached to a symbol to make a new symbol that stands for a modified or new value 💜 Diacritics can also be marks applied to a symbol whether or not they change that symbol's value. In that latter case, the diacritic usually stands for an independent value, like an accent, a tone, or some other linguistic information. Also called diacritical mark or diacritical. [UNICODE]
control character
The 65 characters in the ranges U+0000..U+001F and U+007F..U+009F 🙈 The basic space character, U+0020, is often counted as a control character too, which makes the total 66. They're also known as control codes. Unicode picked up this terminology from ASCII and the ISO 8859 standards, where these codes fall into three ranges: "C0" (for U+0000..U+001F), "C1" (for U+0080...U+009F), and the single control character "DEL" (U+007F). [UNICODE]
Sometimes, in other vocabularies, "control character" is used for any character that doesn't normally have an associated glyph, and it's also sometimes used for device control sequences [ISO6429]. Neither of those usages is appropriate to internationalization terminology in the IETF 🚫 So no mixing those up, bestie!
formatting character
Characters that are invisible by nature but still have an effect on the characters around them 🪄 [UNICODE]
Examples include characters that specify the direction of text, and characters that specify how to join multiple characters 🧵
compatibility character or compatibility variant
A graphic character included as a coded character of ISO/IEC 10646 mainly for compatibility with existing coded character sets 🎀 [ISOIEC10646)]
The Unicode definition of compatibility character also covers characters that were added for other reasons 💅 The list includes several separate groups of characters added for compatibility purposes: halfwidth and fullwidth characters used with East Asian scripts, Arabic contextual forms (e.g., initial or final forms), some ligatures, deprecated formatting characters, variant forms of characters (or even copies of them) for particular uses (e.g., phonetic or mathematical applications), font variations, CJK compatibility ideographs, and so on. For more info and the separate term "compatibility decomposable character," check out the Unicode standard 📚
For example, U+FF01 (FULLWIDTH EXCLAMATION MARK) was included for compatibility with Asian charsets that include full-width and half-width ASCII characters 🍓
Some IETF efforts have concluded that it would be useful to support mapping some groups of compatibility equivalents but not others. For instance, you could support or map width variations while preserving or rejecting mathematical variations 🌸 See the IDNA Mapping document [RFC5895] for one example.
4.2. Differentiation of Subsets 🦋
Especially as existing IETF standards get internationalized, it becomes necessary to describe collections of characters, especially various subsets of Unicode 🌈 Unicode can code substantially all characters in contemporary use, so subsets of the Unicode repertoire are a handy tool for defining these collections as repertoires that don't depend on any specific Unicode coding ✨
However a specific collection gets defined, remember this: older CCSs like ASCII and the ISO 8859 family are close-ended and fixed, but Unicode is open-ended 🚀 New character definitions, and often new scripts, get added about every year. So an ASCII subset like "uppercase letters" can be specified as a range of code points (4/1 to 5/10 for that example), but similar definitions for Unicode either have to be written in terms of Unicode properties, or they depend heavily on Unicode versions, and the relevant version has to be identified in any specification 🔖 See the IDNA code point specification [RFC5892] for an example of specifying things with combinations of properties 💜
Some terms are commonly used in the IETF to define character ranges and subsets 🫶 A few of them are imprecise and can cause confusion if you don't use them carefully.
non-ASCII
The term "non-ASCII" strictly refers to characters other than those in the ASCII repertoire, no matter what CCS or encoding is used for them 🙈 In practice, if a repertoire like Unicode is established as the context, "non-ASCII" means the characters in that repertoire that don't appear in the ASCII repertoire. "Outside the ASCII repertoire" and "outside the ASCII range" are practical, and more precise, synonyms for "non-ASCII" 🎀
letters
The term "letters" doesn't have an exact equivalent in the Unicode standard 🥺 Letters are generally characters used to write words, but what counts as a word means very different things in different languages and cultures 🌍
5. User Interface for Text ✨
okay so the IETF doesn't standardize user interfaces 🙈 but lowkey, a LOT of protocols quietly assume how a user will type or see text that goes through them. internationalization is literally the moment those assumptions get wrecked 💀 because input and output devices aren't the same everywhere. so it's worth thinking about how real people actually interact with text that has one or more non-ASCII characters in it 🌸
input methods
An input method is a mechanism for a person to enter text into an application. <RFC6365>
okay so text can get into a computer in a bunch of ways 💅 keyboards are the most common by FAR, but a lot of characters can't be typed on a normal keyboard in a single stroke. so most operating systems ship with system software that lets users input characters outside what the keyboard can do natively 🫶
for example, there are dozens of input methods for Han characters in Chinese, Japanese, and Korean 🧚 some start with phonetic typing on the keyboard, while others use the number of strokes in the character. input methods are also needed for scripts with lots of diacritics, like European or Vietnamese characters that stack two or three diacritics on one single letter 😭
the term "input method editor" (IME) gets used generically to describe the tools and software that handle character input on a particular system. iconic 👑
rendering rules
A rendering rule is an algorithm that a system uses to decide how to display a string of text. <RFC6365>
some scripts can be displayed straight up with fonts ✨ each character from the input stream gets copied from a glyph set and dropped onto the screen or printed page. other scripts need rules based on the context of the characters around them to render text for display 🦋
some examples of these rendering rules:
- scripts like Arabic (and many others), where the shape of a letter changes depending on its neighbours. the letter could be standing alone, or at the beginning, middle, or end of a word. the rendering rules must choose between two or more glyphs 💖
- scripts like the Indic scripts, where consonants can change their form when they sit next to certain other consonants, or get displayed in a different order from how they're stored and pronounced. the rendering rules must choose between two or more glyphs 🌈
- Arabic and Hebrew scripts, where the display order of characters gets flipped by the bidirectional properties of the letters and other characters, plus right-to-left and left-to-right ordering marks. the rendering rules must choose the order that characters get displayed in 🔥
- some writing systems can't have their rendering rules defined with the mechanisms that are currently in the Unicode Standard. none of those languages are in active non-scholarly use today 💀
- lots of systems use a special rendering rule when they don't have a font or other way to render a particular character correctly. that rule usually swaps in a small open box or a question mark for the missing character. see "undisplayable character" below 🙈
graphic symbol
A graphic symbol is the visual representation of a graphic character or of a composite sequence. <ISOIEC10646>
font
A font is a collection of glyphs used for the visual depiction of character data. A font is often associated with a set of parameters (for example, size, posture, weight, and serifness), which, when set to particular values, generates a collection of imagable glyphs. <UNICODE>
the word "font" gets used interchangeably with "typeface" 🎀 but historically in typography, a typeface is a family of one or more fonts that share a common general design. like, "Times Roman" is actually a typeface, and it contains fonts such as "Times Roman Bold", "Times Roman Medium", "Times Roman Italic", and so on. some sources even count different type sizes within a typeface as different fonts 🍓 those distinctions rarely matter for internationalization, but there are exceptions. people writing specs should be very careful with definitions when those exceptions could cause ambiguity 🫶
bidirectional display
The process or result of mixing left-to-right oriented text and right-to-left oriented text in a single line is called bidirectional display, often abbreviated as "bidi". <UNICODE>
most of the world's written languages are displayed left-to-right, but a lot of super widely used languages, like ones based on the Hebrew or Arabic scripts, are displayed mainly right-to-left 🌸 (numerals are a common exception in the modern scripts). right-to-left text really messes with protocol writers 😭 because they have to keep thinking in terms of the order of characters in memory, which can be totally different from what shows up on the screen. (some languages are written both horizontally and vertically, and some historical ones use other display orderings too 🧚)
bidi text can also be confusing because ISO/IEC 10646 has formatting characters that change the order of display. these explicit formatting characters change the display no matter what the implicit left-to-right or right-to-left properties of the characters are 💅 text that might contain those characters usually needs careful processing before it gets sorted or compared for equality ⭐
it's super common to see strings with text in both directions, like strings with both text and numbers, or strings that mix scripts 🦋
Unicode has a long and incredibly detailed algorithm for displaying bidirectional text [UAX9]. it's giving main character energy 👑
undisplayable character
A character that has no displayable form. <RFC6365>
for instance, the zero-width space (U+200B) can't be displayed because it takes up no horizontal space 🙈 formatting characters, like the ones for setting the direction of text, are undisplayable too. BUT note that every character in [UNICODE] has a glyph associated with it, and the glyphs for undisplayable characters are enclosed in a dashed square to show that the actual character can't be displayed 💜
the property that makes a character undisplayable is intrinsic to its definition. undisplayable characters can never be displayed in normal text, and the dashed square notation only shows up in special circumstances 🎀 printable characters whose Unicode definitions are tied to glyphs that a particular system can't render are NOT, in this sense, undisplayable 🌈
writing style
Conventions of writing the same script in different styles. <RFC6365>
different communities using the same script may find text in different writing styles hard to read, and possibly unintelligible 🥺 for example, the Perso-Arabic Nastalique writing style and the Arabic Naskh writing style both use the Arabic script but look super different and are not mutually comprehensible 💀 writing styles can have a big impact on internationalization too. for example, Nastalique needs significantly more line height than Naskh does 📏✨
6. Text in Current IETF Protocols ✨
Lots of IETF protocols started out fully internationalized, and others got internationalized when they were revised. Along the way, IETF folks noticed patterns in how protocols handle text. This section walks through some specific ways protocols interact with text 💅
protocol elements
Protocol elements are uniquely named parts of a protocol. <RFC6365>
Almost every protocol has named elements, like "source port" in TCP 🔌 In some protocols, the names of the elements (or text tokens for the names) get sent right inside the protocol. For example, in SMTP and loads of other IETF protocols, the names of the verbs are part of the command stream. So the names are literally part of the protocol standard 📜 End users don't normally see the names of protocol elements, and it's rarely appropriate to internationalize them (even though the elements themselves can be internationalized).
name spaces
A name space is the set of valid names for a particular item, or the syntactic rules for generating those valid names. <RFC6365>
Lots of items in Internet protocols use names to identify specific instances or values. The names might be generated by prescribed rules, registered centrally (for example, with IANA), or handled by a distributed registration and control mechanism, like the names in the DNS 🌐
on-the-wire encoding
The encoding and decoding used before and after transmission over the network is often called the "on-the-wire" format (or sometimes just "wire" format). <RFC6365>
Characters are identified by code points. Before they're transmitted in a protocol, they have to be encoded as bits and octets first 🧚 Same deal in reverse: when characters arrive in a transmission, they've been encoded, so a protocol that needs to process the individual characters has to decode them before processing.
parsed text
Text strings that have been analyzed for subparts. <RFC6365>
In some protocols, free text in text fields might get parsed. For example, lots of mail user agents (MUAs) will parse the words in the text of the Subject: field to try to thread messages based on what shows up after the "Re:" prefix 🧵
Honestly, these conventions are super sensitive to localization 😭 If something like a "Re:" form gets changed by an MUA to match the language of the sender or recipient, a system that later does threading may not recognize the replacement term as a delimiter string. Total plot twist.
charset identification
Specification of the charset used for a string of text. <RFC6365>
Protocols that allow more than one charset in the same place should require that the text be identified with the appropriate charset. Without that identification, a program looking at the text can't definitively tell which charset it is 🤷♀️ Charset identification is also called "charset tagging" 🏷️
language identification
Specification of the human language used for a string of text. <RFC6365>
Some protocols (like MIME and HTTP) let text meant for machine processing be identified with the language used in the text. This matters a lot for machine processing, like systems that render text by speaking it out loud 🔊 Language identification is also called "language tagging". The IETF "LTRU" standards [RFC5646] and [RFC4647] give a comprehensive model for language identification 🌸
MIME
MIME (Multipurpose Internet Mail Extensions) is a message format that allows textual message bodies and headers in character sets other than US-ASCII, in formats that require ASCII (most notably RFC 5322, the standard for Internet mail headers [RFC5322]). MIME is described in RFCs 2045 through 2049, plus more recent RFCs. <RFC6365> 💌
transfer encoding syntax
A transfer encoding syntax (TES) (sometimes called a transfer encoding scheme) is a reversible transform of already encoded data that is represented in one or more character encoding schemes. <RFC6365>
TESs are useful for encoding types of character data into another format, usually so new kinds of data can be sent over legacy protocols 🦋 The main TES examples used in the IETF are Base64 and quoted-printable. MIME identifies the transfer encoding syntax for body parts as a Content-transfer-encoding, occasionally abbreviated C-T-E.
Base64
Base64 is a transfer encoding syntax that lets binary data be represented by the ASCII characters A through Z, a through z, 0 through 9, +, /, and =. It is defined in [RFC2045]. <RFC6365> 🍓
quoted printable
Quoted printable is a transfer encoding syntax that lets strings with non-ASCII characters mixed in with mostly ASCII printable characters be somewhat human readable. It is described in [RFC2047]. <RFC6365>
The quoted printable syntax is generally considered to be a failure at being readable 💀 It's jokingly called "quoted unreadable" 🙈
XML
XML (which is an approximate abbreviation for Extensible Markup Language) is a popular method for structuring text. XML text that isn't encoded as UTF-8 is explicitly tagged with charsets, and all text in XML consists only of Unicode characters 👑 The XML specification can be found at http://www.w3.org/XML/. <RFC6365>
ASN.1 text formats
The ASN.1 data description language has many formats for text data. The formats allow for different repertoires and different encodings 🎀 Some of the formats that show up in IETF standards based on ASN.1 include:
IA5String(all ASCII characters)PrintableString(most ASCII characters, but missing many punctuation characters)BMPString(characters from ISO/IEC 10646 plane 0 in UTF-16BE format)UTF8String(just as the name implies)TeletexString(also called T61String)
ASCII-compatible encoding (ACE)
Starting in 1996, a bunch of ASCII-compatible encoding schemes (which are actually transfer encoding syntaxes) were proposed as possible solutions for internationalizing host names and some other purposes ✨ Their goal is to be able to encode any string of ISO/IEC 10646 characters using the preferred syntax for domain names (as described in STD 13). At the time of this writing, only the ACE produced by Punycode [RFC3492] has become an IETF standard 🏆
The choice of ACE forms to internationalize legacy protocols must be made with care, because it can cause some difficult side effects [RFC6055]. Be careful, bestie 🫶
LDH label
The classical label form used in the DNS, and in most applications that call on it, reflects the early syntax of "hostnames" [RFC0952] and limits those names to ASCII letters, digits, and embedded hyphens. Some additional restrictions apply too, which is giving very strict vibes 😤 The hostname syntax is identical to the "preferred name syntax" described in Section 3.5 of RFC 1034 [RFC1034], as modified by RFC 1123 [RFC1123]. LDH labels are defined in a more restrictive and precise way for internationalization contexts, as part of the IDNA2008 specification [RFC5890] 💖
7. Terms Associated with Internationalized Domain Names 🌈
7.1. IDNA Terminology
The current specification for Internationalized Domain Names (IDNs), formally known as Internationalized Domain Names for Applications or IDNA, is referred to in the IETF and parts of the wider community as "IDNA2008", and it consists of several documents 📚 Section 2.3 of the first of those documents, commonly known as "IDNA2008 Definitions" [RFC5890], gives definitions and introduces some specialized terms for telling apart types of DNS labels in an IDN context. Those terms are listed in the table below. If you need the exact definitions, see RFC 5890 🤓
ACE Prefix
A-label
Domain Name Slot
IDNA-valid string
Internationalized Domain Name (IDN)
Internationalized Label
LDH Label
Non-Reserved LDH label (NR-LDH label)
U-labelTwo more terms joined the IETF's vocabulary as part of the earlier IDN effort [RFC3490] (IDNA2003) 🦄
- Stringprep: Stringprep [RFC3454] provides a model and character tables for preparing and handling internationalized strings. It was used in the original IDN specification (IDNA2003) via a profile called "Nameprep" [RFC3491]. It is no longer in use in IDNA, but a number of other protocols still use it in their own profiles.
<RFC6365> - Punycode: This is the name of the algorithm [RFC3492] used to convert otherwise-valid IDN labels from native-character strings expressed in Unicode to an ASCII-compatible encoding (ACE). Strictly speaking, the term applies only to the algorithm. In practice, it's widely used, if erroneously, to refer to the strings that the algorithm encodes 🍭
7.2. Character Relationships and Variants
okay so, the word "variant" entered the IETF i18n vocab with the JET recommendations [RFC3743] 💅 At first it meant ONE specific thing: the relationship between Traditional Chinese characters and their Simplified equivalents 🇨🇳✨ JET gave a model for spotting those character pairs and the labels that used them. Then a follow-up doc [RFC4713] gave specific variant handling guidance for the Chinese language 🎀
in more recent years, people started using "variant" for other collections of characters or strings that might look equivalent 👀 bestie, it's giving scope creep 😭 Those collections have involved one or more of these categories of characters and labels containing them:
- "visually similar" or "visually confusable" characters 🙈 These may be limited to characters in different scripts, characters in a single script, or both. They may be ones that can look alike even with high-distinguishability reference fonts, or under various circumstances involving malicious typeface choices or other ways to trick user perception. Trivial examples include ASCII "l" and "1" and Latin and Cyrillic "a". 💀
- characters assigned more than one Unicode code point because of some special property 🌈 These might be "the same" for some purposes and different for others (or for other users). The most commonly cited example is the Arabic YEH, which is encoded more than once because some of its shapes differ across languages. Another is the Greek lowercase sigma and final sigma: if the final sigma were seen purely as a positional presentation variation of the lowercase sigma, it should not have been given a separate code point. ng, that's a mood 🫶
- numerals and labels including them 🔢 Unlike letters, the "meaning" of decimal digits is clear and unambiguous no matter what script they're tied to. Some scripts are routinely used almost interchangeably with European digits and digits native to that script. The Arabic script has two sets of digits (
U+0660..U+0669andU+06F0..U=06F9), written identically for zero through three and seven through nine, but differently for four through six. European digits predominate in other areas. Substituting digits with the same numeric value in labels may give rise to another type of variant. 🧚 - orthographic differences within a language ✍️ Many languages have alternate spellings, or spellings that differ by locale. Users of those languages generally recognize the spellings as equivalent, at least as much as the variations described above. Examples include "color" and "colour" in English, German words spelled with o-umlaut or "oe", and so on. Some of these relationships may also create other language-specific perceived differences that don't exist for other languages using the same script. For example, in Arabic at the end of words, ARABIC LETTER TEH MARBUTA (
U+0629) and ARABIC LETTER HEH (U+0647) are shaped differently (one has 2 dots on top), but they're used interchangeably in writing. They "sound" similar when pronounced at the end of a phrase, so LETTER TEH MARBUTA is sometimes written as LETTER HEH, and the two are considered "confusable" in that context. this is so specific and I love it 🍓
The term "variant" as used in this section should also not be confused with other uses of the term in this document or in Unicode terminology (e.g., those in Section 4.1 above). If the term is to be used at all, context should clearly distinguish among these different uses and, in particular, between variant characters and variant labels. Local text should identify which meaning, or combination of meanings, are intended. 🚨
8. Other Common Terms in Internationalization ✨
this is a hodge-podge of other terms that have shown up in internationalization discussions in the IETF 🧺💖
locale
Locale is the user-specific location and cultural information managed by a computer. <RFC6365> 🌍
Because languages and orthographic conventions differ from country to country (and even region to region within a country), the user's locale can often be a really important factor 🗺️ Typically, the locale information for a user includes the language(s) used.
Locale issues go beyond character use, and can include things like the display format for currency, dates, and times 💸🗓️ Some locales (especially the popular "C" and "POSIX" locales) don't include language information. 🙃
it should be noted that there are a lot of thorny, unsolved issues with locale 😩 For example, should text be viewed using the locale information of the person who wrote it, the information that would apply to the location of the system storing or providing the text, or the person viewing it? What if the person viewing it is traveling to different locations? Should only some of the locale information affect creation and editing of text? the questions are endless bestie 🌀
Latin characters
"Latin characters" is a not-precise term for characters historically related to ancient Greek script as modified in the Roman Republic and Empire, and currently used throughout the world. <RFC6365> 🏛️
The base Latin characters are a subset of the ASCII repertoire, and they've been augmented by many single and multiple diacritics and quite a few other characters. ISO/IEC 10646 encodes the Latin characters in, including the ranges U+0020..U+024F and U+1E00..U+1EFF. ✨
Because "Latin characters" gets used in different contexts to refer to the letters from the ASCII repertoire, the subset of those characters used late in the Roman Republic period, or the different subset used to write Latin in medieval times, the entire ASCII repertoire, all of the code points in the extended Latin script as defined by Unicode, and other collections, the term should be avoided in IETF specifications when possible. 🙅♀️ Similarly, "Basic Latin" should not be used as a synonym for "ASCII". no mixing those up 🙏
romanization
The transliteration of a non-Latin script into Latin characters. <RFC6365> 🔤
Because of their widespread use, Latin characters (or graphemes constructed from them) are often used to try to write text in languages that didn't previously have writing systems, or whose writing systems were originally based on different scripts. For example, there are two popular romanizations of Chinese: Wade-Giles and Pinyin, and the latter is by far more common today. 🥡 Many romanization systems are inexact and don't give perfect round-trip mappings between the native script and the Latin characters.
CJK characters and Han characters
The ideographic characters used in Chinese, Japanese, Korean, and traditional Vietnamese writing systems are often called "CJK characters" after the initial letters of the language names in English. They're also called "Han characters", after the term in Chinese that is often used for these characters. <RFC6365> 🀄
Note that Han characters do not include the phonetic characters used in the Japanese and Korean languages. Users of the term "CJK characters" may or may not assume those additional characters are included. 🤷♀️
In ISO/IEC 10646, the Han characters were "unified", meaning that each set of Han characters from Japanese, Chinese, and/or Korean that had the same origin was assigned a single code point. 🧩 The positive result was that many fewer code points were needed to represent Han. The negative result is that characters that people who write the three languages think are different have the same code point. There is a great deal of disagreement about the nature, the origin, and the severity of the problems caused by Han unification. this debate is so intense 😭
translation
The process of conveying the meaning of some passage of text in one language, so that it can be expressed equivalently in another language. <RFC6365> 🌐
Many language translation systems are inexact and cannot be applied repeatedly to go from one language to another to another. 🔁💀
transliteration
The process of representing the characters of an alphabetical or syllabic system of writing by the characters of a conversion alphabet. <RFC6365> 🔁
Many script transliterations are exact, and many have perfect round-trip mappings. The notable exception is romanization, described above. Transliteration involves converting text expressed in one script into another script, generally on a letter-by-letter basis. There are many official and unofficial transliteration standards, most notably those from ISO TC 46 and the U.S. Library of Congress. 📚✨
transcription
The process of systematically writing the sounds of some passage of spoken language, generally with the use of a technical phonetic alphabet (usually Latin-based) or other systematic transcriptional orthography. Transcription also sometimes refers to converting written text into a transcribed form, based on the sound of the text as if it had been spoken. <RFC6365> 🎙️
Unlike transliterations, which are generally designed to be round-trip convertible, transcriptions of written material are almost never round-trip convertible to their original form, at least without some supplemental information. 🫠
regular expressions
Regular expressions provide a mechanism to select specific strings from a set of character strings. They're a language used to search for text within strings, and possibly to modify the text found with other text. <RFC6365> 🔍💅
Pattern matching for text involves being able to represent one or more code points in an abstract notation, such as searching for all capital Latin letters or all punctuation. The most common mechanism in IETF protocols for naming such patterns is regular expressions. There's no single regular expression language, but there are numerous very similar dialects that aren't quite consistent with each other. 😵💫
The Unicode Consortium has a good discussion about how to adapt regular expression engines to use Unicode. [UTR18] 📖
private use character
ISO/IEC 10646 code points from U+E000 to U+F8FF, U+F0000 to U+FFFFD, and U+100000 to U+10FFFD are available for private use. This refers to code points of the standard whose interpretation is not specified by the standard, and whose use may be determined by private agreement among cooperating users. <UNICODE> 🔐
The use of these "private use" characters is defined by the parties who transmit and receive them, and is thus not appropriate for standardization. (The IETF has a long history of private use names for things such as "x-" names in MIME types, charsets, and languages. Most of the experience with these has been quite negative, with many implementors assuming that private use names are in fact public and long-lived. 🫣 we have all been there 💜)
9. Security Considerations 🔐
okay so bestie, security isn't actually discussed directly in this document 💀 but the definitions here have no direct effect on security. They still get used in SO many security contexts. Like, authentication usually means comparing two tokens, and one or both of those tokens might be text. So some ways of comparing them might use the internationalization concepts we defined in this doc. Lowkey wild, right? 🫣
If you want the security tea, other RFCs about internationalization have security considerations sections that are super helpful. Specifically, RFC 3454, RFC 3629, RFC 4013 [RFC4013], and RFC 5890 go into a fair amount of detail. Go read those, they're so worth it 🧠✨
10. References 📚
10.1. Normative References
These are the sources I'm treating as must-reads, no cap 💅
[ISOIEC10646] ISO/IEC, "ISO/IEC 10646:2011. International Standard
-- Information technology - Universal Multiple-Octet
Coded Character Set (UCS)", 2011.
[RFC2047] Moore, K., "MIME (Multipurpose Internet Mail
Extensions) Part Three: Message Header Extensions for
Non-ASCII Text", RFC 2047, November 1996.
[UNICODE] The Unicode Consortium, "The Unicode Standard,
Version 6.0", (Mountain View, CA: The Unicode
Consortium, 2011. ISBN 978-1-936213-01-6).
<http://www.unicode.org/versions/Unicode6.0.0/>.10.2. Informative References
These ones are extra reading for when you're obsessed and need more 🤓🫶
[CHARMOD] W3C, "Character Model for the World Wide Web 1.0",
2005, <http://www.w3.org/TR/charmod/>.
[FRAMEWORK] ISO/IEC, "ISO/IEC TR 11017:1997(E). Information
technology - Framework for internationalization,
prepared by ISO/IEC JTC 1/SC 22/WG 20", 1997.
[ISO3166] ISO, "ISO 3166-1:2006 - Codes for the representation
of names of countries and their subdivisions -- Part
1: Country codes", 2006.
[ISO639] ISO, "ISO 639-1:2002 - Code for the representation of
names of languages - Part 1: Alpha-2 code", 2002.
[ISO6429] ISO/IEC, "ISO/IEC, "ISO/IEC 6429:1992. Information
technology -- Control functions for coded character
sets"", ISO/IEC 6429:1992, 1992.
[RFC0952] Harrenstien, K., Stahl, M., and E. Feinler, "DoD
Internet host table specification", RFC 952,
October 1985.
[RFC1034] Mockapetris, P., "Domain names - concepts and
facilities", STD 13, RFC 1034, November 1987.
[RFC1123] Braden, R., "Requirements for Internet Hosts -
Application and Support", STD 3, RFC 1123,
October 1989.
[RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet
Mail Extensions (MIME) Part One: Format of Internet
Message Bodies", RFC 2045, November 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2277] Alvestrand, H., "IETF Policy on Character Sets and
Languages", BCP 18, RFC 2277, January 1998.
[RFC2781] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of
ISO 10646", RFC 2781, February 2000.
[RFC2978] Freed, N. and J. Postel, "IANA Charset Registration
Procedures", BCP 19, RFC 2978, October 2000.
[RFC3454] Hoffman, P. and M. Blanchet, "Preparation of
Internationalized Strings ("stringprep")", RFC 3454,
December 2002.
[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello,
"Internationalizing Domain Names in Applications
(IDNA)", RFC 3490, March 2003.
[RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
Profile for Internationalized Domain Names (IDN)",
RFC 3491, March 2003.
[RFC3492] Costello, A., "Punycode: A Bootstring encoding of
Unicode for Internationalized Domain Names in
Applications (IDNA)", RFC 3492, March 2003.
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, November 2003.
[RFC3743] Konishi, K., Huang, K., Qian, H., and Y. Ko, "Joint
Engineering Team (JET) Guidelines for
Internationalized Domain Names (IDN) Registration and
Administration for Chinese, Japanese, and Korean",
RFC 3743, April 2004.
[RFC4013] Zeilenga, K., "SASLprep: Stringprep Profile for User
Names and Passwords", RFC 4013, February 2005.
[RFC4647] Phillips, A. and M. Davis, "Matching of Language
Tags", BCP 47, RFC 4647, September 2006.
[RFC4713] Lee, X., Mao, W., Chen, E., Hsu, N., and J. Klensin,
"Registration and Administration Recommendations for
Chinese Domain Names", RFC 4713, October 2006.
[RFC5137] Klensin, J., "ASCII Escaping of Unicode Characters",
BCP 137, RFC 5137, February 2008.
[RFC5198] Klensin, J. and M. Padlipsky, "Unicode Format for
Network Interchange", RFC 5198, March 2008.
[RFC5322] Resnick, P., Ed., "Internet Message Format",
RFC 5322, October 2008.
[RFC5646] Phillips, A. and M. Davis, "Tags for Identifying
Languages", BCP 47, RFC 5646, September 2009.
[RFC5890] Klensin, J., "Internationalized Domain Names for
Applications (IDNA): Definitions and Document
Framework", RFC 5890, August 2010.
[RFC5892] Faltstrom, P., "The Unicode Code Points and
Internationalized Domain Names for Applications
(IDNA)", RFC 5892, August 2010.
[RFC5895] Resnick, P. and P. Hoffman, "Mapping Characters for
Internationalized Domain Names in Applications (IDNA)
2008", RFC 5895, September 2010.
[RFC6055] Thaler, D., Klensin, J., and S. Cheshire, "IAB
Thoughts on Encodings for Internationalized Domain
Names", RFC 6055, February 2011.
[UAX34] The Unicode Consortium, "Unicode Standard Annex #34:
Unicode Named Character Sequences", 2010,
<http://www.unicode.org/reports/tr34>.
[UAX9] The Unicode Consortium, "Unicode Standard Annex #9:
Unicode Bidirectional Algorithm", 2010,
<http://www.unicode.org/reports/tr9>.
[US-ASCII] ANSI, "Coded Character Set -- 7-bit American Standard
Code for Information Interchange, ANSI X3.4-1986",
1986.
[UTN6] The Unicode Consortium, "Unicode Technical Note #5:
BOCU-1: MIME-Compatible Unicode Compression", 2006,
<http://www.unicode.org/notes/tn6/>.
[UTR15] The Unicode Consortium, "Unicode Standard Annex #15:
Unicode Normalization Forms", 2010,
<http://www.unicode.org/reports/tr15>.
[UTR18] The Unicode Consortium, "Unicode Standard Annex #18:
Unicode Regular Expressions", 2008,
<http://www.unicode.org/reports/tr18>.
[UTR22] The Unicode Consortium, "Unicode Technical Standard
#22: Unicode Character Mapping Markup Language",
2009, <http://www.unicode.org/reports/tr22>.
[UTR6] The Unicode Consortium, "Unicode Technical Standard
#6: A Standard Compression Scheme for Unicode", 2005,
<http://www.unicode.org/reports/tr6>.
[W3C-i18n-Def] W3C, "Localization vs. Internationalization",
September 2010, <http://www.w3.org/International/
questions/qa-i18n.en>.Appendix A. Additional Interesting Reading 🦋
if you're hooked on writing systems now (same), these books are the moodboard 🌸
Barry, Randall, ed. ALA-LC Romanization Tables. Washington: U.S.
Library of Congress, 1997. ISBN 0844409405
Coulmas, Florian. Blackwell Encyclopedia of Writing Systems.
Oxford: Blackwell Publishers, 1999. ISBN 063121481X
Dalby, Andrew. Dictionary of Languages: The Definitive Reference to
More than 400 Languages. New York: Columbia University Press, 2004.
ISBN 978-0231115698
Daniels, Peter, and William Bright. The World's Writing Systems.
New York: Oxford University Press, 1996. ISBN 0195079930
DeFrancis, John. The Chinese Language: Fact and Fantasy. Honolulu:
University of Hawaii Press, 1984. ISBN 0-8284-085505 and
0-8248-1058-6
Drucker, Joanna. The Alphabetic Labyrinth: The Letters in History
and Imagination. London: Thames & Hudson, 1995. ISBN 0-500-28068-1
Fazzioli, Edoardo. Chinese Calligraphy. New York: Abbeville Press,
1986, 1987 (English translation). ISBN 0-89659-774-1
Hooker, J.T., et al. Reading the Past: Ancient Writing from
Cuneiform to the Alphabet. London: British Museum Press, 1990. ISBN
0-7141-8077-7
Lunde, Ken. CJKV Information Processing. Sebastopol, CA: O'Reilly &
Assoc., 1999. ISBN 1-56592-224-7
Nakanishi, Akira. Writing Systems of the World. Rutland, VT:
Charles E. Tuttle Company, 1980. ISBN 0804816549
Robinson, Andrew. The Story of Writing: Alphabets, Hieroglyphs, &
Pictograms. London: Thames & Hudson, 1995, 2000. ISBN 0-500-28156-4
Sacks, David. Language Visible. New York: Broadway Books (a
division of Random House, Inc.), 2003. ISBN 0-7679-1172-5Appendix B. Acknowledgements 🫶
the definitions in this doc came from SO many places, like a wide variety of IETF documents. Huge shoutout to everyone who helped out 💜
James Seng contributed to the initial outline of RFC 3536. Harald Alvestrand and Martin Duerst made extensive useful comments on early versions. Others who contributed to the development of RFC 3536 include Dan Kohn, Jacob Palme, Johan van Wingen, Peter Constable, Yuri Demchenko, Susan Harris, Zita Wenzel, John Klensin, Henning Schulzrinne, Leslie Daigle, Markus Scherer, and Ken Whistler. 👑
Abdulaziz Al-Zoman, Tim Bray, Frank Ellermann, Antonio Marko, JFC Morphin, Sarmad Hussain, Mykyta Yevstifeyev, Ken Whistler, and others identified important issues with, or made specific suggestions for, this new version. Thank you all, you're literally the reason this is so good 🌈✨
Appendix C. Significant Changes from RFC 3536 💅✨
okay bestie, this one is a changelog for RFC 3536, so basically what changed between the old version and this new one 🦋 This doc is mostly stuff added on top of RFC 3536. Here's the tea on the biggest changes:
- Changed the document's status to BCP. 👑
- Commonly used synonyms added to several descriptions and indexed. 🌸
- A list of terms defined and used in IDNA2008 was added, with a pointer to RFC 5890. Those definitions have not been repeated in this document. 📚
- The much-abused term "variant" is now discussed in some detail. 🙈
- A discussion of different subsets of the Unicode repertoire was added as Section 4.2 and associated definitions were included. 🧚
- Added a new term, "writing style". 💜
- Discussions of case-folding and mapping were expanded. ✨
- Minor edits were made to some section titles and a number of other editorial improvements were made. 🎀
- The discussion of control codes was updated to include additional information and clarify that "control code" and "control character" are synonyms. 🍓
- Many terms were clarified to reflect contemporary usage. 🌈
- The index to terms by section in RFC 3536 was replaced by an index to pages containing considerably more terms. 📖
- The acknowledgments were updated. 🫶
- Some of the references were updated. 🔗
- The supplemental reading list was expanded somewhat. 📝
Index 📇
A
A-label 31
ACE 30, 31
ACE Prefix 31
alphabetic 20
ANSI 13
ASCII 15
ASCII-compatible encoding 30, 31
ASN.1 text formats 30
B
Base64 29
Basic Multilingual Plane 13
bidi 26
bidirectional display 26
BMP 13
BMPString 30
BOCU-1 14
BOM 14
byte order mark 14
C
C-T-E 29
case 18
CCS 7
CEN/ISSS 13
character 6
character encoding form 7
character encoding scheme 8
character repertoire 7
charset 8
charset identification 28
CJK characters 34
code chart 19
code point 16
code table 19
coded character 6
coded character set 7
collation 18
combining character 16
combining character sequence 16
compatibility character 22
compatibility variant 22
composite sequence 16
content-transfer-encoding 29
control character 21
control code 21
control sequence 22
D
decomposed character 16
diacritic 21
displaying and rendering text 10
Domain Name Slot 31
E
encoding forms 13
F
font 25
formatting character 22
G
glyph 7
glyph code 7
graphic symbol 25
H
Han characters 34
I
i18n 9
IA5String 30
ideographic 20
IDN 31
IDNA 31
IDNA-valid string 31
IDNA2003 31
IDNA2008 31
IME 24
input method editor 24
input methods 24
internationalization 8
Internationalized Domain Name 31
Internationalized Label 31
ISO 11
ISO 639 11
ISO 3166 11
ISO 8859 15
ISO TC 46 11
J
JIS 13
JTC 1 11
L
l10n 9
language 5
language identification 29
Latin characters 34
LDH Label 30
letters 23
Local and regional standards organizations 13
locale 33
localization 9
M
MIME 29
multilingual 10
N
name spaces 28
Nameprep 31
NFC 17
NFD 17
NFKC 17
NFKD 17
non-ASCII 23
nonspacing character 21
normalization 17
NR-LDH label 31
NVT 15
O
on-the-wire encoding 28
P
parsed text 28
precomposed character 16
PrintableString 30
private use charater 36
protocol elements 27
punctuation 21
Punycode 30, 31
Q
quoted-printable 29
R
regular expressions 36
rendering rules 24
repertoire 7
romanization 34
S
SAC 13
script 5
SCSU 14
sorting 18
Stringprep 31
surrogate pair 14
symbol 21
T
T61String 30
TeletexString 30
TES 29
transcoding 7
transcription 35
transfer encoding syntax 29
transformation formats 13
translation 35
transliteration 34, 35
typeface 25
U
U-label 31
UCS-2 13
UCS-4 13
undisplayable character 26
Unicode Consortium 12
US-ASCII 15
UTC 12
UTF-8 14
UTF-16 14
UTF-16BE 14
UTF-16LE 14
UTF-32 14
UTF8String 30
V
variant 32
W
W3C 13
World Wide Web Consortium 13
writing style 27
writing system 6
X
XML 13, 30Authors' Addresses 🫶
Paul Hoffman
VPN Consortium
EMail: paul.hoffman@vpnc.org
John C Klensin
1770 Massachusetts Ave, Ste 322
Cambridge, MA 02140
USA
Phone: +1 617 245 1457
EMail: john+ietf@jck.comThat's the whole doc, bestie!! 🌈💖
📦 where this page comes from
- ietf: rfc6365 2011-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms