RFC 3536

Terminology Used in Internationalization in the IETF

📅 published May 2003, from ietf

Terminology Used in Internationalization in the IETF 🎀

okay bestie, so this is RFC 3536, a glossary of i18n terms for the IETF crew 📚. Written by P. Hoffman (IMC & VPNC), filed as Informational in May 2003. It's not a standard, just a super helpful reference 💖. The header doesn't list anything it obsoletes or updates, so it's a solo vibe 🦋.

Status of this Memo

This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright (C) The Internet Society (2003). All Rights Reserved.

Abstract

okay so this doc is a glossary of terms the IETF uses when talking about internationalization 🌈. The whole point is to help frame i18n discussions across all the different IETF areas and to introduce the main concepts to participants 🫶.

Table of Contents

   1. Introduction...................................................  2
     1.1 Purpose of this document....................................  2
     1.2 Format of the definitions in this document..................  3
   2. Fundamental Terms..............................................  3
   3. Standards Bodies and Standards.................................  8
     3.1 Standards bodies............................................  8
     3.2 Encodings and transformation formats of ISO/IEC 10646....... 10
     3.3 Native CCSs and charsets.................................... 11
   4. Character Issues............................................... 12
     4.1 Types of characters......................................... 15
   5. User interface for text........................................ 17
   6. Text in current IETF protocols................................. 19
   7. Other Common Terms In Internationalization..................... 22
   8. Security Considerations........................................ 25
   9. References..................................................... 25
     9.1 Normative References........................................ 25
     9.2 Informative References...................................... 26
   10. Additional Interesting Reading................................ 27
   11. Index......................................................... 27
   A. Acknowledgements............................................... 29
   B. Author's Address............................................... 29
   Full Copyright Statement.......................................... 30

1. Introduction ✨

As [RFC2277] summarizes: "Internationalization is for humans. This means that protocols are not subject to internationalization; text strings are." Honestly iconic 😭. Lots of IETF protocols use text strings that humans type in or read 💅. Anyone should be able to enter or read these strings, which means Internet users need to be able to type text with their usual input methods and see it displayed in any human language 🌸. On top of that, text with any character should be able to get passed between Internet applications easily 🧚. That's the whole challenge of internationalization, and it's giving main character energy 👑.

1.1 Purpose of this document

okay so this section is basically the doc explaining itself 💜. It's a glossary of terms the IETF uses when talking about internationalization, meant to help frame i18n discussions in every area and to introduce the main concepts to IETF folks 🦋.

Internationalization gets discussed in tons of working groups, but ngl very few of them have i18n experts 🙈. When people design or update protocols, the question pops up constantly: "should we internationalize this" (or, more likely, "do we have to internationalize this") 💀.

This document gives an overview of internationalization as it applies to IETF standards work. It lightly covers the many aspects of i18n and the vocabulary that goes with them 🍓. It is not meant to be a complete description of internationalization, so don't expect everything 😅. The definitions in this document are not normative for IETF standards. However, they're useful, and standards may make informative reference to this document after it becomes an RFC ✨. Some of the definitions come from earlier IETF documents and books 📚.

Like in lots of fields, the internationalization community doesn't fully agree on definitions for many words 🌈. The topic of language brings up especially passionate opinions from experts and non-experts alike 🔥. This document tries to define terms in the way that's most useful for the IETF audience 💖.

This document uses definitions from many documents developed outside the IETF. The primary documents are:

  • ISO/IEC 10646 [ISOIEC10646]
  • The Unicode Standard [UNICODE]
  • W3C Character Model [CHARMOD]
  • IETF RFCs, including [RFC2277]

1.2 Format of the definitions in this document

So here's how the definitions are laid out 🎀. In the body of the doc, the source of each definition is shown in angle brackets, like <ISOIEC10646>. A lot of definitions show <NONE>, which means they were written originally for this document 🫶. The angle bracket notation for definition sources is different from the square bracket notation used for document references, like the [RFC2277] up above. Those references are listed in Section 9 🦋.

Some terms have commentary and examples after the definitions. In those cases, the part before the angle brackets is the definition from the original source. The part after the angle brackets is commentary that isn't a definition, like examples or extra explanation 💅.

Examples in this document use the code point and name notation from the Unicode Standard [UNICODE] and ISO/IEC 10646 [ISOIEC10646]. For example, the letter "a" may be represented as either U+0061 or LATIN SMALL LETTER A 🌸.

Part 1 of 7 done 💖 more coming, bestie! ✨

2. Fundamental Terms ✨

okay bestie, this section is the vocab list you need before anything else 🎀 it covers the basics for anyone trying to make IETF protocols friendlier to non-ASCII text and other internationalization stuff 💅

language

A language is a way that humans interact 🗣️ Language shows up in lots of forms, and the most common are speech, writing, and signing. <NONE>

Some languages have a close relationship between the written and spoken forms, while others are looser 🌸 [RFC3066] goes deeper and gives identifiers for languages to use in Internet protocols 🧚 ngl, computer languages are explicitly excluded from this definition, so no coding languages here 💀

script

A set of graphic characters used for the written form of one or more languages. <ISOIEC10646>

Examples of scripts are Latin, Cyrillic, Greek, Arabic, and Han (the ideographs used in writing Chinese, Japanese, and Korean). [RFC2277] discusses scripts in detail 📜

It's super common for internationalization newbies to mix up "language" and "script" 🙈 This can be a problem in protocols that differentiate the two. Almost all protocols designed (or redesigned) to handle non-ASCII text deal with scripts (the written systems) or characters, while fewer actually deal with languages 💜

A single name can mean either a language or a script. For example, "Arabic" is both the name of a language and the name of a script 🌈 In fact, many scripts borrow their names from languages. Also, many scripts are used for many languages. For example, the Russian and Bulgarian languages are both written in the Cyrillic script 👑 Some languages can be expressed using different scripts. The Mongolian language can be written in either the Mongolian or Cyrillic scripts, and the Serbo-Croatian language is written using both the Latin and Cyrillic scripts 🦋 Further, some languages are normally expressed with more than one script at the same time. For example, the Japanese language is normally expressed in the Kanji (Han), Katakana, and Hiragana scripts in a single string of text 🍓

character

A member of a set of elements used for the organization, control, or representation of data. <ISOIEC10646>

There are at least three common definitions of the word "character" 🧠

  • a general description of a text entity
  • a unit of a writing system, often synonymous with "letter" or similar terms
  • the encoded entity itself

When people talk about characters, they're mostly using one of the first two definitions 💖

A particular character is identified by its name, not by its shape ✨ A name may suggest a meaning, but the character may be used for representing other meanings as well. A name may suggest a shape, but that doesn't mean only that shape is commonly used in print, nor that the particular shape is associated only with that name. Main character energy, but make it a glyph 💅

coded character

A character together with its coded representation. <ISOIEC10646>

coded character set

A coded character set (CCS) is a set of unambiguous rules that establishes a character set and the relationship between the characters of the set and their coded representation. <ISOIEC10646>

character encoding form

A character encoding form is a mapping from a character set definition to the actual code units used to represent the data. <UNICODE>

repertoire

The collection of characters included in a character set. Also called a character repertoire. <UNICODE>

glyph

A glyph is an abstract form that represents one or more glyph images. The term "glyph" is often a synonym for glyph image, which is the actual, concrete image of a glyph representation that has been rasterized or otherwise imaged onto some display surface. In displaying character data, one or more glyphs may be selected to depict a particular character. These glyphs are selected by a rendering engine during composition and layout processing. <UNICODE> 🎨

glyph code

A glyph code is a numeric code that refers to a glyph. Usually, the glyphs contained in a font are referenced by their glyph code. Glyph codes are local to a particular font; that is, a different font containing the same glyphs may use different codes. <UNICODE> 🔤

transcoding

Transcoding is the process of converting text data from one character encoding form to another. Transcoders work only at the level of character encoding and do not parse the text. Note: Transcoding may involve one-to-one, many-to-one, one-to-many, or many-to-many mappings. Because some legacy mappings are glyphic, they may not only be many-to-many, but also discontinuous: thus XYZ may map to yxz. <CHARMOD> 🔁

In this definition, "many-to-one" means a sequence of characters mapped to a single character. The "many" does not mean alternative characters that map to the single character 🙃

character encoding scheme

A character encoding scheme (CES) is a character encoding form plus byte serialization. There are many character encoding schemes in Unicode, such as UTF-8 and UTF-16. <UNICODE> 💾

Some CESs are associated with a single CCS. For example, UTF-8 [RFC2279] applies only to ISO/IEC 10646. Other CESs, such as ISO 2022, are associated with many CCSs 🌟

charset

A charset is a method of mapping a sequence of octets to a sequence of abstract characters. A charset is, in effect, a combination of one or more CCSs with a CES. Charset names are registered by the IANA according to procedures documented in [RFC2278]. <NONE> 🗂️

Many protocol definitions use the term "character set" in their descriptions. The terms "charset" or "character encoding scheme" are strongly preferred over "character set", because "character set" has other definitions in other contexts and this can be confusing 🙄

internationalization

In the IETF, "internationalization" means to add or improve the handling of non-ASCII text in a protocol. <NONE> 🌍

Many protocols that handle text only handle one script (often the one that contains the letters used in English text), or they leave the question of what character set is used up to local guesswork. Of course, that leads to interoperability problems 😭 Adding non-ASCII text to such a protocol lets it handle more scripts, hopefully all of the ones useful in the world 🫶

localization

The process of adapting an internationalized application platform or application to a specific cultural environment. In localization, the same semantics are preserved while the syntax may be changed. [FRAMEWORK]

Localization is the act of tailoring an application for a different language, script, or culture. Some internationalized applications can handle a wide variety of languages. Typical users only understand a small number of languages, so the program must be tailored to interact with users in just the languages they know 💬

The major work of localization is translating the user interface and documentation. Localization involves not only changing the language interaction, but also other relevant changes, such as display of numbers, dates, currency, and so on. The better internationalized an application is, the easier it is to localize for a particular language and character encoding scheme 🧁

Localization is rarely an IETF matter. Protocols that are merely localized, even if they are serially localized for several locations, are generally considered unsatisfactory for the global Internet 🌐

Do not confuse "localization" with "locale", which is described in Section 7 of this document ⚠️

i18n, l10n

These are abbreviations for "internationalization" and "localization". <NONE> ✨

"18" is the number of characters between the "i" and the "n" in "internationalization", and "10" is the number of characters between the "l" and the "n" in "localization" 🔢

multilingual

The term "multilingual" has many widely-varying definitions and thus is not recommended for use in standards. Some of the definitions relate to the ability to handle international characters, other definitions relate to the ability to handle multiple charsets, and still others relate to the ability to handle multiple languages. <NONE> 🚫

displaying and rendering text

To display text, a system puts characters on a visual display device such as a screen or a printer. To render text, a system analyzes the character input to determine how to display the text. The terms "display" and "render" are sometimes used interchangeably. Note, however, that text might be rendered as audio and/or tactile output, such as in systems designed for people with visual disabilities 👀 <NONE>

Combining characters modify the display of the character (or, in some cases, characters) that precede them. When rendering such text, the display engine must either find the glyph in the font that represents the base character and all of the combining characters, or it must render the combination itself. Such rendering can be straightforward, but it is sometimes complicated when the combining marks interact with each other, such as when there are two combining marks that would appear above the same character 😤 Formatting characters can also change the way that a renderer would display text. Rendering can also be difficult for some scripts that have complex display rules for base characters, such as Arabic and Indic scripts 🪄

3. Standards Bodies and Standards ✨

This section describes some of the standards bodies and standards that appear in discussions of internationalization in the IETF. This is an incomplete and possibly over-full list, and listing too few bodies or standards can be just as politically dangerous as listing too many 🫣 Note that there are many other bodies that deal with internationalization. However, few if any of them appear commonly in IETF standards work 💅

3.1 Standards bodies 🌸

ISO

Okay so ISO (the International Organization for Standardization) has been doing standards for characters since before the IETF even existed 🧚 It's a non-governmental group made up of national bodies. ISO has SO many diverse standards in the international characters area 💅 The one the IETF uses most is commonly called "ISO/IEC 10646", although its official name has more qualifications. (IEC is the International Electrotechnical Commission, fyi 🤓) ISO/IEC 10646 describes a CCS that covers almost all known written characters in use today 🌈

ISO/IEC 10646 is controlled by the group "ISO/IEC JTC 1/SC 2 WG2", usually just called "WG2" 🫶 ISO standards go through many steps before they're finished, so years can go by between changes to ISO/IEC 10646 ⏳ You can find info on WG2 and its work products at <http://www.dkuug.dk/JTC1/SC2/WG2/>.

The standard comes in multiple parts and can be bought in both print and CD-ROM versions 💿 One example of how to cite it is given in [RFC2279]. Any standard that cites ISO/IEC 10646 needs to evaluate how to handle the versioning problem that's relevant to the protocol's needs. This is a big deal, bestie 😭

ISO is responsible for other standards that might be of interest to protocol developers too. [ISO 639] specifies the names of languages, and [ISO 3166] specifies the abbreviations of countries 🌍 Character work is done in the groups known as ISO/IEC JTC1/SC22 and ISO TC46, plus other ISO groups.

Another relevant ISO group is JTC 1/SC22/WG20, which handles internationalization in JTC1, like international string ordering 🔥 Info on WG20 and its work products can be found at <http://www.dkuug.dk/jtc1/sc22/wg20/>.

Unicode Consortium

The second important group for international character standards is the Unicode Consortium 👑 It's a trade association of companies, governments, and other groups who are into promoting the Unicode Standard [UNICODE]. The Unicode Standard is a CCS whose repertoire and code points are identical to ISO/IEC 10646. The Unicode Consortium added features to the base CCS that make it more useful in protocols, like defining attributes for each character. Examples of these attributes include case conversion and numeric properties ✨

The Unicode Consortium publishes addenda to the Unicode Standard as Unicode Technical Reports. There are many types of technical reports at various stages of maturity 📚 The Unicode Standard and affiliated technical reports can be found at <http://www.unicode.org/>.

World Wide Web Consortium (W3C)

This group created and maintains the standard for XML, the markup language for text that became super popular 🦋 XML has always been fully internationalized, so there's no need for a new version to handle international text. Iconic behaviour 💖

local and regional standards organizations

Just like there are many native CCSs and charsets, there are many local and regional standards organizations to create and support them 🌏 Common examples are ANSI (United States) and CEN/ISSS (Europe).

3.2 Encodings and transformation formats of ISO/IEC 10646 🧁

Characters in the ISO/IEC 10646 CCS can be expressed in many ways. Encoding forms are direct addressing methods, while transformation formats are methods for expressing encoding forms as bits on the wire 🔌

Basic Multilingual Plane (BMP)

The BMP is made up of the first 2^16 code points in ISO/IEC 10646. It's also called "plane 0" 🍓

UCS-2 and UCS-4

UCS-2 and UCS-4 are the two encoding forms defined for ISO/IEC 10646. UCS-2 only addresses the BMP. Because many useful characters (like many Han characters) have been defined outside of the BMP, many people would consider UCS-2 to be dead 💀 Theoretically, UCS-4 addresses the entire range of 2^31 code points from ISO/IEC 10646 as 32-bit values. However, for interoperability with UTF-16, ISO 10646 restricts the range of characters that will actually be allocated to the values 0..0x10FFFF 🎀

UTF-8

UTF-8, a transformation format specified in [RFC2279], is the preferred encoding for IETF protocols 👑 Characters in the BMP are encoded as one, two, or three octets. Characters outside the BMP are encoded as four octets. Characters from the US-ASCII repertoire have the same on-the-wire representation in UTF-8 as they do in US-ASCII. Iykyk, it's the main character of encodings 💅

UTF-16, UTF-16BE, and UTF-16LE

UTF-16, UTF-16BE, and UTF-16LE are three transformation formats defined in [RFC2781]. They're not required by any IETF standards, so they get used a lot less than UTF-8 🥲 Characters in the BMP are always encoded as two octets, and characters outside the BMP are encoded as four octets. The three formats differ based on the order of the octets and the presence of a special lead-in mark called the "byte order mark" or "BOM" 🔍

UTF-32

The Unicode Consortium has defined UTF-32 as a transformation format for UCS-4 in [UTR19] 🌸

SCSU and BOCU-1

The Unicode Consortium has defined an encoding, SCSU, which is designed to offer good compression for typical text. SCSU is described in [UTR6]. A different encoding that's meant to be MIME-friendly, BOCU-1, is described in [UTN6]. Compression is attractive, as opposed to UTF-8 , but neither of these has (at the time of this writing) attracted much interest in the IETF 🙈

3.3 Native CCSs and charsets 🦋

Before ISO/IEC 10646 was developed, lots of countries made their own CCSs and charsets. Many dozen of these are in common use on the Internet today 🌈 Examples include ISO 8859-5 for Cyrillic and Shift-JIS for Japanese scripts.

The official list of registered charset names for use with IETF protocols is maintained by IANA and lives at <http://www.iana.org/assignments/character-sets>. The list has preferred names and aliases. Note that this list has historically contained many errors, like names that aren't actually charsets, or references that don't give enough detail to reliably map names to charsets 🙃

Probably the most well-known native CCS is ASCII [US-ASCII] 🧚 This CCS is the basis for keywords and parameter names in many IETF protocols, and it's the sole CCS in numerous IETF protocols that haven't been internationalized yet.

[UTR22] describes issues involved in mapping character data between charsets, and it also includes an XML format for mapping table data ✨

4. Character Issues ✨

okay bestie, this section is all about the character stuff that matters when you're bolting non-ASCII text onto a protocol 🌈 the wild part is that none of these topics are standardized by the IETF. they're handled outside it, so we borrow the definitions 💅

combining character

A member of an identified subset of the coded character set of ISO/IEC 10646 🎀 it's meant to be stuck onto the preceding non-combining graphic character, or onto a sequence of combining characters that starts with a non-combining character. <ISOIEC10646>

composite sequence

A sequence of graphic characters made of a non-combining character followed by one or more combining characters 🦋 the graphic symbol for the whole sequence is usually the combo of the graphic symbols of each character in it. A composite sequence is NOT a character, so it isn't in the ISO/IEC 10646 repertoire 🙈 <ISOIEC10646>

Some CCSs build certain characters out of other characters. For example, "a with acute" could be made of "a" plus "combining acute", or of three characters: "a", a non-destructive backspace, and an acute 🌸 The rules for putting characters together are called "composition rules", and the rules for splitting a character back into other characters are called "decomposition rules". The result of composition is a "precomposed character", and the result of decomposition is a "decomposed character" ✨

normalization

Normalization is transforming data into a normal form, for example to unify spelling 💖 <UNICODE>

Quick heads up, "unify spelling" here does NOT mean making different words with the same meaning match up (like "color" and "colour") 🙅‍♀️ it means unifying different character sequences that are meant to form the same composite characters, like <a><n><combining tilde><o> and <a><n with tilde><o>.

The whole point of normalization is letting two strings get compared for equivalence 🔥 <a><n><combining tilde><o> and <a><n with tilde><o> would look exactly the same on a text display device. If a protocol designer wants those two strings to count as equivalent during comparison, the protocol must define where normalization happens 👑

"Normalization" and "canonicalization" get used interchangeably a lot. Basically they both mean converting a string of one or more characters into another string using standardized rules 🧚 some CCSs allow several equivalent ways to write the same string, and normalization picks one of them as the base for reference when comparing strings. For text, these rules usually involve decomposing combined characters or composing characters with combining characters 💜 [UTR15] goes into the process and the many forms of normalization in lots of detail. Normalization matters a ton when you're checking whether two strings are the same 😭

case

Case is the feature of certain alphabets where the letters come in two distinct forms 🍓 those variants can look super different in shape and size, and they're called the uppercase letter (also known as capital or majuscule) and the lowercase letter (also known as small or minuscule) 🌈 Case mapping is the link between the uppercase and lowercase forms of a letter. <UNICODE>

Usually (but not always) each letter has a one-to-one mapping between its two cases 🙈 But there are lots of characters that exist in only one case, or that have no matching partner in the other case, or that follow a special mapping rule. Examples are the Turkish dotless "i" and some Greek characters with modifiers 💀 Case mapping can even depend on locale, which is very much a vibe. Turning text into just one case is called "case folding" 🦋

sorting and collation

Collating is the process of ordering units of textual information 💅 Collation is usually specific to a particular language. It's sometimes called alphabetizing, though alphabetization is really just one special case of sorting and collation. <UNICODE> Collation is all about figuring out the relative order of any particular pair of strings 🔥 Collation algorithms focus on providing suitable weighted keys for string values, so that a binary comparison of the key values can decide how the strings are ordered ✨

Sorting is actually putting data records into a specified order, based on criteria for comparing the records 💖 Sorting can apply to any kind of data, textual or not, as long as an ordering criterion can be defined 🎀 Sorting algorithms focus on performance (in terms of time, memory, or other resources) while they put the data records into that order 🫶

A sorting algorithm for string data can be internationalized by feeding it the right collation-weighted keys for the strings being ordered 🌸

Lots of processes need to put strings in a consistent sequence (sorted) 🧚 For only a few CCS/CES combinations is there an obvious sort order that doesn't need any reference to the linguistic meaning of the characters. For those, codepoint order is enough for sorting, meaning codepoint order is also the order a person would pick when sorting the characters 👑 For many CCS/CES combinations, though, codepoint order would make no sense to a person, so it isn't useful for sorting if the results are going to be shown to a human 😭

Codepoint order is usually NOT how any human educated by a local school system expects strings to be ordered 💀 If you order to a human's expectations, you've got a language-specific sort 🦋 Sorting by codepoint order will look inconsistent if the strings aren't normalized before sorting, because different representations of the same character will sort differently 🙈 That problem may be smaller with a language-specific sort 🌈

code table

A code table is a table showing the characters allocated to the octets in a code. <ISOIEC10646>

Code tables are also commonly called "code charts" 🍓

4.1 Types of characters 💜

The definitions below for types of characters don't cleanly put each character into just one type, and they won't let you accurately predict which types apply to a particular character 🙃 They're meant for application designers to help them think through the many (sometimes confusing) properties of text 💖

alphabetic

An informative Unicode property 🌸 Characters that are the primary units of alphabets and/or syllabaries, whether combining or noncombining. This includes composite characters that are canonical equivalents to a combining character sequence of an alphabetic base character plus one or more combining characters: letter digraphs; contextual variants of alphabetic characters; ligatures of alphabetic characters; contextual variants of ligatures; modifier letters; letterlike symbols that are compatibility equivalents of single alphabetic letters; and miscellaneous letter elements. <UNICODE>

ideographic

Any symbol that primarily denotes an idea (or meaning) rather than a sound (or pronunciation) 🧚 For example, a symbol showing a telephone, or the Han characters used in Chinese, Japanese, and Korean. <UNICODE>

punctuation

Characters that separate units of text, like sentences and phrases, which makes the meaning of the text clearer ✨ Punctuation isn't just for prose either, since it shows up in mathematical and scientific formulae too 🔥 <UNICODE>

symbol

One of a set of characters other than those used for letters, digits, or punctuation, which represent various concepts that are generally not connected to written language use per se 🎀 Examples include symbols for mathematical operators, symbols for OCR, symbols for box-drawing or graphics, and symbols for dingbats. <NONE>

Examples of symbols include characters for arrows, faces, and geometric shapes 🦋 [UNICODE] has a property that defines characters as symbols.

nonspacing character

A combining character whose positioning in presentation depends on its base character 🫶 It generally doesn't take up space along the visual baseline by itself. <UNICODE>

A combining acute accent (U+0301) is an example of a nonspacing character 💅

diacritic

A mark applied or attached to a symbol to create a new symbol that represents a modified or new value 💖 Diacritics can also be marks applied to a symbol whether or not they change that symbol's value. In that second case, the diacritic usually stands for an independent value, like an accent, a tone, or some other linguistic information 🌈 Also called diacritical mark or diacritical. <UNICODE>

control character

The 65 characters in the ranges U+0000..U+001F and U+007F..U+009F 🙈 They're also known as control codes. <UNICODE>

formatting character

Characters that are inherently invisible but that have an effect on the surrounding characters 🧚 <UNICODE>

Examples of formatting characters include characters for specifying the direction of text and characters that specify how to join multiple characters 🌸

compatibility character

A graphic character included as a coded character of ISO/IEC 10646 primarily for compatibility with existing coded character sets 🍓 <ISOIEC10646>

For example, U+FF01 (FULLWIDTH EXCLAMATION MARK) was included for compatibility with Asian character sets that include full-width and half-width ASCII characters 🎀

5. User interface for text 🎀

The IETF doesn't standardize user interfaces, okay? 🙈 But lots of protocols quietly assume how a user will type or see text, and internationalization messes with those assumptions because different apps run on wildly different input and output devices. So it's worth thinking about how people actually interact with text that has one or more non-ASCII characters in it. 💅

input methods ⌨️

An input method is a way for a person to type text into an application.

Keyboards are by far the most common device, but a lot of characters can't be entered on a typical keyboard in a single stroke. Most operating systems ship with system software that lets users enter characters outside what the keyboard can directly produce. 🧚

For example, there are dozens of input methods for Han characters in Chinese, Japanese, and Korean. Some start from phonetic input typed on the keyboard, while others use the number of strokes in the character. Scripts with lots of diacritics need input methods too, like European characters that carry two or three diacritics on a single letter. 🍓

rendering rules 🖼️

A rendering rule is an algorithm a system uses to decide how to display a string of text.

Some scripts can be shown directly with fonts. Each character from the input stream gets copied from a glyph set and put on the screen or printed page, easy peasy. Other scripts need rules that depend on the context of the characters to render text properly. ✨

Some examples of these rendering rules:

  • Scripts like Arabic (and many others), where a letter's form changes depending on its neighbours, whether it stands alone, starts a word, sits in the middle, or ends one. The rendering rules have to pick between two or more glyphs. 🌸
  • Indic scripts, where consonants can change form when they sit next to certain other consonants, or get displayed in a different order from the order they're stored and pronounced in. The rendering rules have to pick between two or more glyphs. 🪷
  • Arabic and Hebrew, where the displayed order of characters gets flipped by the bidirectional properties of the letters, plus right-to-left and left-to-right ordering marks. The rendering rules have to choose the order the characters appear in. 🔄

graphic symbol 🔤

A graphic symbol is the visual representation of a graphic character or of a composite sequence. <ISOIEC10646>

font 🎨

A font is a collection of glyphs used to visually depict character data. A font usually comes with a set of parameters, like size, posture, weight, and serifness. When those parameters get set to particular values, they produce a collection of glyphs you can actually image. <UNICODE>

bidirectional display ↔️

Bidirectional display is the process, or the result, of mixing left-to-right text and right-to-left text in a single line. 💖

Most of the world's written languages go left-to-right. Still, plenty of widely used languages, like ones based on the Hebrew or Arabic scripts, go right-to-left. Right-to-left text really messes with protocol writers, because they keep thinking about the order of characters in a string in memory, and that order can be totally different from what shows up on screen. (Some languages are written both horizontally and vertically, too.) 😭

Bidirectional text can also get confusing because ISO/IEC 10646 has formatting characters that change the display order of text. These explicit formatting characters change the display no matter what the implicit left-to-right or right-to-left properties of the characters are. 🙃

It's really common to see strings with text in both directions, like strings with both text and numbers, or strings that mix several scripts. 🌈

[UNICODE] has a long and incredibly detailed algorithm for displaying bidirectional text. Respect. 🫶

undisplayable character 🫥

An undisplayable character is a character that has no displayable form.

Take the zero-width space (U+200B). It can't be displayed, because it takes up no horizontal space at all. Formatting characters that set the direction of text are undisplayable too. Here's the twist, though: every character in [UNICODE] has a glyph associated with it. Glyphs for undisplayable characters get drawn inside a dashed square, as a signal that the real character can't be displayed. 💀

6. Text in current IETF protocols 🧚‍♀️

Lots of IETF protocols started out fully internationalized, and others got internationalized along the way as they were revised. Through that process, IETF members have noticed patterns in how many protocols use text, and this section walks through some specific protocol interactions with text. 💜

protocol elements 🧩

Protocol elements are uniquely named parts of a protocol.

Almost every protocol has named elements, like "source port" in TCP. In some protocols, the names of the elements (or the text tokens for those names) get sent within the protocol itself. For instance, in SMTP and plenty of other IETF protocols, the names of the verbs are part of the command stream. That makes those names part of the protocol standard. End users don't normally get to see the names of protocol elements. 🙈

name spaces 🏷️

A name space is the set of valid names for a particular item, or the syntactic rules for generating those valid names.

Lots of items in Internet protocols use names to identify specific instances or values. Names might be generated by prescribed rules, registered centrally (for example, with IANA), or handled by a distributed registration and control mechanism, like the names in the DNS. 🌐

on-the-wire encoding 📡

The encoding and decoding used before and after transmission over the network is often called the "on-the-wire" format (or sometimes just "wire" format). 🛰️

Characters get identified by codepoints. Before they go out in a protocol, they have to be encoded as bits and octets first. Going the other way, when characters arrive in a transmission, they've been encoded, so a protocol that needs to process the individual characters has to decode them before processing. 🔓

parsed text 🔍

Parsed text is a text string that gets analyzed for its subparts. 🪄

In some protocols, free text in text fields might get parsed. For example, many mail user agents parse the words in the text of the Subject: field, trying to thread messages based on what comes after the "Re:" prefix. 📬

charset identification 🔖

Charset identification is the specification of the charset used for a string of text. ✨

Protocols that allow more than one charset in the same place should insist that the text be identified with the appropriate charset. Without that identification, a program looking at the text can't definitively tell what charset it's in. Charset identification is also called "charset tagging." 🏷️

language identification 🗣️

Language identification is the specification of the human language used for a string of text. 🌍

Some protocols, like MIME and HTTP, let text meant for machine processing be identified with the language used in the text. That identification matters for machine processing, like systems that render text by speaking it out loud. Language identification is also called "language tagging." 💬

MIME 💌

MIME (Multipurpose Internet Mail Extensions) is a message format that allows textual message bodies and headers in character sets other than US-ASCII, in formats that require ASCII. The big one it works with is [RFC2822], the standard for Internet mail headers. MIME is described in RFCs 2045 through 2049, plus more recent RFCs. 📮

transfer encoding syntax 🎁

A transfer encoding syntax (TES), sometimes called a transfer encoding scheme, is a reversible transform of already-encoded data that's represented in one or more character encoding schemes. 🔁

TESs are useful for encoding types of character data into another format, usually so new types of data can be sent over legacy protocols. The main TESs used in the IETF are Base64 and quoted-printable. 🧃

Base64 🔢

Base64 is a transfer encoding syntax that lets binary data be represented by the ASCII characters A through Z, a through z, 0 through 9, +, /, and =. It's defined in [RFC2045]. 🧮

quoted printable 🦄

Quoted printable is a transfer encoding syntax that lets strings with non-ASCII characters mixed in with mostly printable ASCII characters be somewhat human readable. It's described in [RFC2047]. 📜

The quoted printable syntax is generally considered a failure at being readable. People jokingly call it "quoted unreadable." 💀

XML 🪶

XML (an approximate abbreviation for Extensible Markup Language) is a popular method for structuring text. XML text is explicitly tagged with charsets. You can find the XML specification at http://www.w3.org/XML/. 💅

ASN.1 text formats 🧾

The ASN.1 data description language has many formats for text data. These formats allow different repertoires and different encodings. Some of the formats that show up in IETF standards based on ASN.1 are:

  • IA5String (all ASCII characters)
  • PrintableString (most ASCII characters, but missing many punctuation characters)
  • BMPString (characters from ISO/IEC 10646 plane 0 in UTF-16BE format)
  • UTF8String (just as the name implies)
  • TeletexString (also called T61String; its repertoire changes over time) 🌀

ASCII-compatible encoding (ACE) 🌏

Starting in 1996, many ASCII-compatible encoding schemes (which are actually transfer encoding syntaxes) were proposed as possible solutions for internationalizing host names. Their goal is to encode any string of ISO/IEC 10646 characters as legal DNS host names (as described in STD 13). At the time of this writing, no ACE has become an IETF standard. 🫶

7. Other Common Terms In Internationalization ✨

okay so this section is a bestie-approved hodge-podge 🧚 of other terms that keep popping up in internationalization chats at the IETF. It's giving "more terms will probably get added as this doc matures" 🌸

locale

Locale is the user-specific location and cultural information managed by a computer. <NONE>

Languages are sooo different from country to country (and even region to region inside a country), so the user's locale can be a seriously big deal 💅 Usually the locale info for a user includes the language(s) they use.

Locale issues go beyond character use too 💖 They can include stuff like the display format for currency, dates, and times. Some locales (especially the popular "C" and "POSIX" locales) don't include any language information at all 🙈

Okay and it has to be said that locale has a bunch of thorny, unsolved issues 💀 For example: should text be viewed using the locale info of the person who wrote it, or the person looking at it? What if the person viewing it is travelling to different places? Should only some of the locale info affect how text gets created and edited? 😭

Latin characters

"Latin characters" is a not-precise term 🦋 for characters historically related to ancient Greek script that are used all over the world now. <NONE>

The base Latin characters make up the ASCII repertoire, and they've been augmented by lots of single and multiple diacritics and quite a few other characters 🎀 ISO/IEC 10646 encodes the Latin characters in the ranges U+0020..U+024F, U+1E00..U+1EFF, and other ranges.

romanization

Romanization is the transliteration of a non-Latin script into Latin characters. <NONE>

Because Latin characters are everywhere, people have tried to write lots of languages that aren't based on a Latin repertoire in Latin ✨ For example, there are two popular romanizations of Chinese: Wade-Giles and Pinyin, and the latter is by far more common today 🇨🇳 Lots of romanization systems are inexact, so they don't give perfect round trip mappings between the native script and the Latin characters 😬

CJK characters and Han characters

The ideographic characters used in Chinese, Japanese, Korean, and traditional Vietnamese writing systems are often called 'CJK characters' after the initial letters of the language names in English. They're also called "Han characters", after the term in Chinese that's often used for them. <NONE>

Note that CJK and Han characters do not include the phonetic characters used in the Japanese and Korean languages 🌸

In ISO/IEC 10646, the Han characters were "unified" 🔥 That means each set of Han characters from Japanese, Chinese, and/or Korean that had the same origin got a single code point. The positive result? Way fewer code points were needed to represent Han 💜 The negative result? Characters that people who write the three languages think are different end up sharing the same code point 😭 There is a great deal of disagreement on the nature, the origin, and the severity of the problems caused by Han unification. Iykyk 🙈

translation

Translation is the process of conveying the meaning of some passage of text in one language, so that it can be expressed equivalently in another language. <NONE>

Lots of language translation systems are inexact, and they can't be applied repeatedly to go from one language to another to another 💀 (Not it, translating in a loop.)

transliteration

Transliteration is the process of representing the characters of an alphabetical or syllabic system of writing by the characters of a conversion alphabet. <NONE>

Many script transliterations are exact, and many have perfect round-trip mappings ✨ The notable exception is romanization, described above 💅 Transliteration means converting text written in one script into another script, usually letter-by-letter 🦋

transcription

Transcription is the process of systematically writing the sounds of some passage of spoken language, usually with a technical phonetic alphabet (usually Latin-based) or some other systematic transcriptional orthography. Transcription also sometimes refers to converting written text into a transcribed (usually Latin-based) form, based on the sound of the text as if it had been spoken. <NONE>

Unlike transliterations, which are generally designed to be round-trip convertible, transcriptions of written material are almost never round-trip convertible back to their original form 🥲

regular expressions

Regular expressions give you a mechanism to select specific strings from a set of character strings 🔍 They're a language used to search for text within strings, and to possibly modify the text found with other text. <NONE>

Pattern matching for text means representing one or more code points in an abstract notation, like searching for all capital Latin letters or all punctuation 💖 The most common mechanism in IETF protocols for naming those patterns is regular expressions. There's no single regular expression language, but there are numerous very similar dialects 🌈

The Unicode Consortium has a good discussion about how to adapt regular expression engines to use Unicode. [UTR18]

private use

ISO/IEC 10646 code points from U+E000 to U+F8FF, U+F0000 to U+FFFFD, and U+100000 to U+10FFFD are available for private use. This refers to code points of the standard whose interpretation is not specified by the standard, and whose use may be determined by private agreement among cooperating users. <UNICODE>

The use of these "private use" characters is defined by the parties who transmit and receive them, so it's not appropriate for standardization 🙅‍♀️ (The IETF has a long history of private use names for things like "x-" names in MIME types, charsets, and languages. The experience with these has been quite negative 😭, with many implementors assuming that private use names are actually public and long-lived.)

8. Security Considerations 🔐

Security is not discussed in this document. 🫶

9. References

9.1 Normative References

[ISOIEC10646] ISO/IEC 10646-1:2000.  International Standard --
              Information technology -- Universal Multiple-Octet
              Coded Character Set (UCS) -- Part 1: Architecture and
              Basic Multilingual Plane, 2000.

[UNICODE]     The Unicode Standard, Version 3.2.0 is defined by The
              Unicode Standard, Version 3.0 (Reading, MA, Addison-
              Wesley, 2000.  ISBN 0-201-61633-5), as amended by the
              Unicode Standard Annex #27: Unicode 3.1
              (http://www.unicode.org/reports/tr27/) and by the
              Unicode Standard Annex #28: Unicode 3.2
              (http://www.unicode.org/reports/tr28/), The Unicode
              Consortium, 2002.

9.2 Informative References

[CHARMOD]     Character Model for the World Wide Web 1.0, W3C,
              <http://www.w3.org/TR/charmod/>.

[FRAMEWORK]   ISO/IEC TR 11017:1997(E).  Information technology -
              Framework for internationalization, prepared by ISO/IEC
              JTC 1/SC 22/WG 20, 1997.

[ISO 639]     ISO 639:2000 (E/F) - Code for the representation of
              names of languages, 2000.

[ISO 3166]    ISO 3166:1988 (E/F) - Codes for the representation of
              names of countries, 2000.

[RFC2045]     Freed, N. and N. Borenstein, "MIME Part One: Format of
              Internet Message Bodies", November 1996.

[RFC2047]     Moore, K., "MIME Part Three: Message Header Extensions
              for Non-ASCII Text", RFC 2047, November 1996.

[RFC2277]     Alvestrand, H., "IETF Policy on Character Sets and
              Languages", BCP 18, RFC 2277, January 1998.

[RFC2279]     Yergeau, F., "UTF-8, a transformation format of ISO
              10646", RFC 2279, January 1998.

[RFC2781]     Hoffman, P. and F. Yergeau, "UTF-16, an encoding of ISO
              10646", RFC 2781, February 2000.

[RFC2822]     Resnick, P., "Internet Message Format", RFC 2822, April
              2001.

[RFC3066]     Alvestrand, H., "Tags for the Identification of
              Languages", BCP 47, RFC 3066, January 2001.

[US-ASCII]    Coded Character Set -- 7-bit American Standard Code for
              Information Interchange, ANSI X3.4-1986, 1986.

[UTN6]        "BOCU-1: MIME-Compatible Unicode Compression", M.
              Scherer & M.  Davis, Unicode Technical Note #6.

[UTR6]        "A Standard Compression Scheme for Unicode", M. Wolf,
              et. al., Unicode Technical Report #6.

[UTR15]       "Unicode Normalization Forms", M. Davis & M. Duerst,
              Unicode Technical Report #15.

[UTR18]       "Unicode Regular Expression Guidelines", M. Davis,
              Unicode Technical Report #18.

[UTR19]       "UTF-32", M. Davis, Unicode Technical Report #19.

[UTR22]       "Character Mapping Markup Language", M. Davis, Unicode
              Technical Report #22.

10. Additional Interesting Reading 📚

Some extra reading if you want to go deeper 🧠

ALA-LC Romanization Tables, Randall Barry (ed.), U.S. Library of
Congress, 1997, ISBN 0844409405

Blackwell Encyclopedia of Writing Systems, Florian Coulmas, Blackwell
Publishers, 1999, ISBN 063121481X

The World's Writing Systems, Peter Daniels and William Bright, Oxford
University Press, 1996, ISBN 0195079930

Writing Systems of the World, Akira Nakanishi, Charles E. Tuttle
Company, 1980, ISBN 0804816549

11. Index 🗂️

alphabetic -- 4.1
ASCII-compatible encoding (ACE) -- 6
ASN.1 text formats -- 6
Base64 -- 6
Basic Multilingual Plane (BMP) -- 3.2
bidirectional display -- 5
BOCU-1 -- 3.2
case -- 4
character -- 2
character encoding form -- 2
character encoding scheme -- 2
charset -- 2
charset identification -- 6
CJK characters and Han characters -- 7
code chart -- 4
code table -- 4
coded character -- 2
coded character set -- 2
combining character -- 4
compatibility character -- 4.1
composite sequence -- 4
control character -- 4.1
diacritic -- 4.1
displaying and rendering text -- 2
font -- 5
formatting character -- 4.1
glyph -- 2
glyph code -- 2
graphic symbol -- 5
i18n, l10n -- 2
ideographic -- 4.1
input methods -- 5
internationalization -- 2
ISO -- 3.1
language -- 2
language identification -- 6
Latin characters -- 7
local and regional standards organizations -- 3.1
locale -- 7
localization -- 2
MIME -- 6
multilingual -- 2
name spaces -- 6
nonspacing character -- 4.1
normalization -- 4
on-the-wire encoding -- 6
parsed text -- 6
private use -- 7
protocol elements -- 6
punctuation -- 4.1
quoted printable -- 6
regular expressions -- 7
rendering rules -- 5
romanization -- 7
script -- 2
SCSU -- 3.2
sorting and collation -- 4
symbol -- 4.1
transcoding -- 2
transcription -- 7
transfer encoding syntax -- 6
translation -- 7
transliteration -- 7
UCS-2 and UCS-4 -- 3.2
undisplayable character -- 5
Unicode Consortium -- 3.1
UTF-32 -- 3.2
UTF-16, UTF-16BE, and UTF-16LE -- 3.2
UTF-8 -- 3.2
World Wide Web Consortium -- 3.1
XML -- 6

A. Acknowledgements ✨

okay so the definitions in this doc come from a bunch of different places, like a wide variety of IETF docs 🫶

James Seng helped with the initial outline 🌸 Harald Alvestrand and Martin Duerst left extremely useful comments on early versions (they were so helpful, literally iconic 💅). Others who contributed to the development include:

Dan Kohn Jacob Palme Johan van Wingen Peter Constable Yuri Demchenko Susan Harris Zita Wenzel John Klensin Henning Schulzrinne Leslie Daigle Markus Scherer Ken Whistler

B. Author's Address 🦋

Paul Hoffman Internet Mail Consortium and VPN Consortium 127 Segre Place Santa Cruz, CA 95060 USA

EMail: paul.hoffman@imc.org and paul.hoffman@vpnc.org

Copyright (C) The Internet Society (2003). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🌈

Funding for the RFC Editor function is currently provided by the Internet Society. 💖

📦 where this page comes from