RFC 5242
A Generalized Unified Character Code: Western European and CJK Sections
RFC 5242: A Generalized Unified Character Code: Western European and CJK Sections 🎀
Okay bestie, the header says this one's from the Network Working Group, it's RFC 5242, and the authors are J. Klensin and H. Alvestrand (from Google) 💖 It's an Informational doc dated 1 April 2008 🦋 The header doesn't list any Obsoletes or Updates lines, so it isn't replacing or updating anything else.
Status of This Memo
This memo gives info to the Internet community 📚 It does not specify an Internet standard of any kind, and distribution of it is unlimited 🌸
IESG Note 🙈
Okay, this is not an IETF document, so the IESG wants you to know that first 😭 Go read RFC 4690, "Review and Recommendations for Internationalized Domain Names (IDNs)", and its references too.
This doc isn't a candidate for any level of Internet Standard. The IETF disclaims any knowledge of how fit it is for any purpose, and it specifically notes that it hasn't had IETF review for stuff like security, congestion control, or inappropriate interaction with deployed protocols. The RFC Editor picked publishing it at their own discretion, so readers should exercise caution when judging its value for implementation and deployment 🧐
Abstract ✨
Lots of issues have been spotted with using general-purpose character sets for internationalized domain names and similar stuff 💅 This memo describes a fully unified coded character set for scripts based on Latin, Greek, Cyrillic, and Chinese (CJK) characters. It is not a complete spec of that character set, so don't treat it like one 🫶
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Types of Characters . . . . . . . . . . . . . . . . . . . . . 4
2.1. Base Character . . . . . . . . . . . . . . . . . . . . . . 4
2.2. Nonspacing Marks . . . . . . . . . . . . . . . . . . . . . 4
2.3. Case Indicators . . . . . . . . . . . . . . . . . . . . . 4
2.4. Joining Indicators . . . . . . . . . . . . . . . . . . . . 5
2.5. Character-Matrix Positioning Indicators . . . . . . . . . 5
2.6. Position Shaping Controls . . . . . . . . . . . . . . . . 6
2.7. Repetition Indicators . . . . . . . . . . . . . . . . . . 6
2.8. Control Characters . . . . . . . . . . . . . . . . . . . . 7
3. Code Assigment Groupings . . . . . . . . . . . . . . . . . . . 7
4. Canonical Form . . . . . . . . . . . . . . . . . . . . . . . . 7
5. Examples of Graphic Element Codes . . . . . . . . . . . . . . 8
6. Composite Characters and Unicode Equivalences . . . . . . . . 10
7. Ideographic Characters . . . . . . . . . . . . . . . . . . . . 11
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11
9. Security Considerations . . . . . . . . . . . . . . . . . . . 12
10. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 12
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 13
11.1. Normative References . . . . . . . . . . . . . . . . . . . 13
11.2. Informative References . . . . . . . . . . . . . . . . . . 131. Introduction 🦋
Lots of issues have been spotted with using general-purpose character sets for internationalized domain names and similar purposes 💭 This memo specifies a fully unified coded character set for scripts based on Latin, Greek, Cyrillic, and Chinese characters.
There are four important principles in this work, and they're the vibe for everything else:
- If it looks alike, it is alike. The number of base characters and marks should be minimized. Glyphs are more important than character abstractions.
- If it is the same thing, it is the same thing. Two symbols that have the same semantic meaning in all contexts should be encoded in a way that allows their identity to be discovered by removing modifiers, rather than having to resort to external equivalence tables.
- For simplicity, when a character form can be evaluated on the basis of either serif or sanserif fonts, the sanserif font is always preferred.
- The use of combining characters and modifiers is preferred to adding more base characters.
Based on these principles, it becomes obvious that:
- Ligatures, digraphs, and final forms are constructed with special modifiers so that relationships to basic forms are obvious 🔥
- Symbols consisting of multiple marks are always constructed from combining characters and positional modifiers. So the "i" character is built from the vertical line symbol followed by a combining dot above 💜 Similarly, "f" is composed of a centered vertical line, a right hook in the top position, and an appropriately-positioned composing hyphen. The way that it all clicks together is so satisfying 😭
This document draws strongly from the design and terminology of Unicode [Unicode], but it represents a radically different approach 🌈
1.1. Terminology
All special-use terms in this document, including descriptions of behaviors and related relationships, are used with their common-sense meanings 🧠
1.2. Discussion
Questions to, and contributions for, this coding system should be addressed to the mailing list unified-ccs@xn--iwem3b1f.xn--90ase1a.bogus.domain.name.
2. Types of Characters 🧩
This document defines several types of characters. Heads up: these definitions are not the same as the Unicode definitions for similar or identical terms 🙃
2.1. Base Character
A base character is any character used as an atomic shape, rather than being assembled from such a character in combination with combining (overstriking) marks, symbols, or specially-designed base characters. When used alone, base characters always take up space. For example: a, c, l,...
2.2. Nonspacing Marks
Nonspacing marks are marks, symbols, and character components used to form characters when combined with base characters. They do not occupy separate character positions when displayed. For example, the special combining symbols LeftUpperHook and RightLowerHook, described in Section 5, are nonspacing marks ✨
2.3. Case Indicators
In scripts with case, only the lower-case characters are base characters. Upper-case forms are represented by using the UC modifier, so the traditional "A" character is represented by a<UC>. This means case-independent comparisons are made simply by ignoring the <UC> modifiers, instead of by complicated mapping operations. Iconic 👑
The initial set of case modifiers consists exclusively of:
UC Upper-case, code value 1 (hexadecimal)
The code values two through four are reserved for the impending encoding of scripts with more than two cases. Five is reserved for expansion, in case a script with more than four cases is identified 🌸
2.4. Joining Indicators
Zero-width joiners are used to build characters, not only to separate or join words 🧚 Compared to Unicode, a richer set of joiners is used to tell apart the inter-word and ligature-forming (including half-character forming) cases. Unicode ZWJ and ZWNJ are supplemented by ZWCJ, OJ, and ONJ.
ZWCJ is used to modify a spacing basic character into a nonspacing role. For example, there is no "w" character, only u<ZWCJ>u. Upper-case "W" is coded as u<ZWCJ>u<UC>, because the CWCJ binds more tightly than the UC modifier 💅
The initial set of joining indicators consists exclusively of:
ZWCJ Character joiner (also known as "ligature joiner"), code value 6 (hexadecimal).
OJ Overlay joiner (permits use of a subsequent character that would normally be spacing as nonspacing), code value 7 (hexadecimal).
ONJ Overlay non-joiner (turns a nonspacing mark into a standalone character), code value 8 (hexadecimal). This joiner should not be necessary, and is normally prohibited by the "shortest string" rule. But there may be unanticipated cases 🙈
ZWJ Zero-width joiner for words or word-like constructions, code value 9 (hexadecimal).
ZWNJ Zero-width non-joiner for words or word-like constructions, code value A (hexadecimal).
2.5. Character-Matrix Positioning Indicators
Lots of characters are defined by constructed glyphs using nonspacing marks 🦋 For example, "b" and "d" are coded as o<VerticalLine><PositionLeft> and o<VerticalLine><PositionRight>, respectively. The Catalan ligature that has caused some difficulties in Internationalizing Domain Names in Applications (IDNA) [RFC3490] is coded as l<ZWCJ><.><PositionVMiddle><ZWCJ>l.
The initial table of positioning indicators is:
+-------------------+-----------+
| Name | Hex value |
+-------------------+-----------+
| PositionLeft | 20 |
| PositionCenter | 21 |
| PositionRight | 22 |
| PositionTop | 30 |
| PositionVMiddle | 31 |
| PositionBottom | 32 |
| PositionDescender | 33 |
+-------------------+-----------+2.6. Position Shaping Controls
These controls designate character form changes for initial or final-form characters 🎀 Where the distinction is important, medial-form characters are the default when no qualification occurs. As with case comparisons, comparisons are performed by ignoring these control functions.
+-------------+-----------+
| Name | Hex value |
+-------------+-----------+
| InitialForm | 71 |
| FinalForm | 72 |
+-------------+-----------+2.7. Repetition Indicators 🍓
For compactness of coding, two repetition indicators are introduced for double (Repeat2) and triple (Repeat3) characters that may be treated as ligatures or special cases. Two consecutive uses of a character compare equal to the character followed by <Repeat2>. The interpretation of u<ZWCJ>u<Repeat3> is left as an exercise for the reader 📝
The initial table of repetition indicators is:
+---------+-----------+
| Name | Hex value |
+---------+-----------+
| Repeat2 | 50 |
| Repeat3 | 51 |
| Repeat1 | 52 |
+---------+-----------+For larger repeats, these repeats can be combined. The sequence <Repeat2><Repeat3> represents six repeats, while <Repeat3><Repeat2> represents five repeats. Following the "shortest string" principle (see Section 4), Repeat1 must not ever appear except in combination with Repeat2 and/or Repeat3. The generation of other numbers is left as an exercise for the reader 🤭
2.8. Control Characters
Because it is intended primarily for domain names, this specification has no provision for control or spacing characters 🌷
3. Code Assigment Groupings ✨
okay bestie so 💅 following the same reasoning Unicode [Unicode] uses, every character takes up exactly 23 bits 🔢 (usually stored in three octets, with the leading bit always zero 🎀). and why 23?? because both 3 and 23 are prime numbers, unlike 42 💀 iconic behavior tbh.
The code point value zero is permanently reserved 🚫 and won't get used unless the code space absolutely has to expand 🌱.
Code values from 1 to 255 (decimal) are reserved for the special character formation codes in Section 2.3 through Section 2.7 🧚.
Code values from 256 to 511 (decimal) are reserved for character formation marks for non-ideographic characters 🦋. Most of them are nonspacing (combining) characters, but not all of them 👀.
Code values from 512 to 1023 are reserved on general principles 🌈, and also just in case we need to invent new rules and make them retroactive 😭.
Code values of 1024 and above are for characters, glyphs, and other character elements 💖.
4. Canonical Form 👑
okay this part is giving strict teacher energy 🙈. when glyphs get built with these mechanisms, there is a single canonical form for representing any given glyph ✨. no exceptions!! any sequence of characters and qualifiers that doesn't match that form is invalid 💀.
if there are two possible ways to represent the same character, the shorter one (counted in octets) is the only permitted form 📏. and if both ways are the same length, the only permitted form is the one whose octet values add up to the smaller total when you sum the numeric values of all the octets in each 🧮. main character energy for the smallest number 🏆.
the ordering rules go like this:
- A base character or composite character (see below) must come first. 👑
- The base character may be followed by ZWCJ or OJ, but not both, followed by a base or nonspacing character or mark. 💅
- If ZWCJ appears, the next character must be a base character or nonspacing mark. 🫶
- If OJ appears, the next character must be a base character, since the whole job of OJ is to turn a spacing base character into a nonspacing (overlay) character 🦋.
- That character can be followed by positional qualifiers that apply to it. Vertical positional qualifiers come before horizontal positional qualifiers 🌸.
- That sequence of characters may be followed by a case qualifier. 🎀
- That entire sequence of characters forms a composite character. When the composite character is non-trivial, the rules can be applied to it recursively 🔁. If grouping is needed to tell one composite character apart from the next, ZWNCJ may be used at the beginning of a composite character to mark a group boundary 🧚.
5. Examples of Graphic Element Codes 🌸
the initial lists of positioning and combining controls are above, so this section is just the vibe check 💖 with codes for some base characters. names in upper case are the Unicode names for the characters ✨. they're followed, just for info, by the Unicode code point designations. the code point list is informative, not normative 🤓, and it might not be complete (especially since more matching code points could get added to Unicode over time 🌈). also, heads up: several Unicode characters that Unicode considers different get assigned the same code sequence in this system 🙈. that's giving "unify it" energy.
+------------------------+-------+----------------------------------+
| Name | Hex | Comment |
| | value | |
+------------------------+-------+----------------------------------+
| FULL STOP (U+002E) | 110 | Used as both base character (in |
| | | bottom center position) and as |
| | | movable dot with OJ and |
| | | positional qualifiers. |
| HYPHEN-MINUS (U+002D) | 108 | Used as a spacing base character |
| | | (in horizontally and vertically |
| | | centered position) and as a |
| | | movable half-width horizontal |
| | | line with OJ and positional |
| | | qualifiers. In the context of |
| | | this specification, should be |
| | | known as Half Horizontal Line. |
| LOW LINE (U+005F) | 109 | Used as a spacing base character |
| | | (in bottom position) and as a |
| | | movable full-width horizontal |
| | | line with OJ and positional |
| | | qualifiers. In the context of |
| | | this specification, should be |
| | | known as Horizontal Line. |
| VERTICAL LINE (U+007C) | 102 | As with the horizontal lines, |
| | | normally a spacing base |
| | | character (in the middle |
| | | position between left and |
| | | right), but can be used as a |
| | | right to left movable |
| | | full-height vertical line with |
| | | OJ and/or positional qualifiers. |
| HalfHeightVerticalLine | 105 | Similar to VERTICAL LINE, but |
| | | only half height. |
| SOLIDUS (U+002F) | 103 | Used only for character |
| | | formation; forward slash |
| REVERSE SOLIDUS | 104 | Used only for character |
| (U+005C) | | formation; reverse slash |
| RightUpperHook | 131 | Used only for character |
| | | formation; nonspacing mark. |
| LeftUpperHook | 132 | Used only for character |
| | | formation; nonspacing mark. |
| LeftLowerHook | 133 | Used only for character |
| | | formation; nonspacing mark. |
| RightLowerHook | 134 | Used only for character |
| | | formation; nonspacing mark. |
| HalfHeightHoop | 140 | Used only for character |
| | | formation; nonspacing mark. |
| HalfHeightInvertedHoop | 141 | Used only for character |
| | | formation; nonspacing mark. |
| DIGIT ZERO (U+0030) | 400 | |
| DIGIT ONE (U+0031) | 401 | |
| DIGIT TWO (U+0032) | 402 | |
| DIGIT NINE (U+0039) | 409 | |
| LATIN SMALL LETTER A | 40A | |
| (U+0061) | | |
| LATIN SMALL LETTER O | 418 | Unify with Greek Omicron |
| (U+006F, U+03BF) | | |
| LATIN SMALL LETTER C | 40C | Unifying C with Cyrillic ES |
| (U+0063, U+0441) | | |
| GREEK SMALL LETTER | 491 | |
| SIGMA (U+03C3) | | |
+------------------------+-------+----------------------------------+6. Composite Characters and Unicode Equivalences 🦋
this section is all about characters that are derived from or based on other characters, which are called "composite characters" 💅✨. here's the lineup:
+------------------+--------------+---------------------------------+
| Name | Hex value | Comment |
+------------------+--------------+---------------------------------+
| LATIN SMALL | 418 007 102 | |
| LETTER B | 020 | |
| (U+0062) | | |
| LATIN SMALL | 418 007 102 | |
| LETTER D | 022 | |
| (U+0064) | | |
| LATIN SMALL | 40C 007 108 | |
| LETTER E | 031 | |
| (U+0065) | | |
| LATIN SMALL | 40A 006 40C | |
| LETTER AE | 007 108 031 | |
| (U+00E6) | | |
| LATIN SMALL | 102 131 030 | Note that 007 is not needed |
| LETTER F | 007 108 | before 131 because hooks are |
| (U+0066) | | exclusively nonspacing |
| | | (combining). |
| LATIN SMALL | 102 020 141 | |
| LETTER H | 021 032 | |
| (U+0068) | | |
| LATIN SMALL | 105 007 110 | |
| LETTER I | 021 030 | |
| (U+0069) | | |
| LATIN SMALL | 105 020 141 | |
| LETTER N | 021 032 | |
| (U+006E) | | |
| LATIN SMALL | 418 007 102 | Unified P, Greek Rho, Cyrillic |
| LETTER P | 033 020 033 | ER |
| (U+0070, U+03C1, | | |
| U+0440) | | |
| LATIN CAPITAL | 40A 001 | |
| LETTER A | | |
| (U+0041) | | |
| LATIN CAPITAL | 418 007 102 | |
| LETTER B | 020 001 | |
| (U+0042) | | |
| LATIN CAPITAL | 40C 001 | |
| LETTER C | | |
| (U+0043) | | |
| LATIN CAPITAL | 418 007 102 | |
| LETTER D | 022 001 | |
| (U+0044) | | |
| GREEK SMALL | 491 072 | |
| LETTER FINAL | | |
| SIGMA (U+03C2) | | |
+------------------+--------------+---------------------------------+7. Ideographic Characters ✨
Okay so Han-derived ("CJK") characters are literally built like LEGO 🧱 out of radicals and strokes, so they fit this system even better than European letters do, with way less ambiguity. The mechanisms used here and shown in the tables (see Section 8) are basically the same vibe as the "Radicals" and "Strokes" ideas in Section 5.1 and the "Ideographic Description Characters" in Section 5.2 of The Unicode Standard [Unicode]. Same rules as the European characters above, so only radicals, strokes, and description controls count as base characters. No separate compound precomposed ideographic characters get registered. Those are just not a thing here 🙅♀️💖
8. IANA Considerations 🎀
IANA is asked to keep the real registry of characters and code tables 📋. Each registry entry has a character name (ideally the same as the Unicode character name when there is one), the code sequence that represents the character, and optional descriptive info. The characters and codes from Section 2, Section 5, and Section 6 above should be used to set up the table at the start 🌸. The coding system can be extended by users, so registrations should be accepted for new characters as long as they don't look like old ones 👀. A designated expert should be picked to advise IANA on "looks like" 🦋. This expert needs a background in calligraphy or abstract art, plus a LOT of experience judging claims about how many angels fit on the head of a pin 😭🧚
9. Security Considerations 🔐
Representing characters this way is a huge win for security 👑. It cuts out a lot of phishing opportunities, because Principle 1 means two characters that look alike but are different can't exist 🙅♀️.
Also, by separating the encoding of characters used in domain names from the encoding used for everything else, you get a guarantee that names which look reasonable were encoded by competent entities. That gives a solid amount of safety by obscurity 🕵️♀️✨
⚠️ Heads up, bestie: this won't totally kill visual confusion. Because of how upper-case forms are encoded, and because similarity is sometimes in the eye of the beholder, lookalikes can still sneak through 😬. For example, the lower-case letters are quite different, so LATIN CAPITAL LETTER A and GREEK CAPITAL LETTER ALPHA will never compare equal, even though they look identical 💀
10. Acknowledgments 🫶
The authors want to thank J.F.C. Morphin, who called out how inadequate it is to try to solve internationalization using only the existing engineering principles 🙌. His comments, plus the problems people ran into trying to internationalize domain names on top of Unicode, really shaped the mindset behind big chunks of this proposal 🧠💜. The theory behind this coding system is partly based on Unicode and its collection of names and sample glyphs, but the coding system itself takes a very different approach 🌈
11. References 📚
11.1. Normative References
[Unicode] The Unicode Consortium, "The Unicode Standard, Version
5.0", 2007.
Boston, MA, USA: Addison-Wesley. ISBN 0-321-48091-011.2. Informative References
[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello,
"Internationalizing Domain Names in Applications (IDNA)",
RFC 3490, March 2003.Authors' Addresses 💌
John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA 02140
USA
Phone: +1 617 491 5735
EMail: john+ietf@jck.com
Harald Tveit Alvestrand
Google
Beddingen 10
Trondheim, 7014
Norway
EMail: harald@alvestrand.noFull Copyright Statement 🧾
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions contained in BCP 78 and at http://www.rfc-editor.org/copyright.html, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 💅
The IETF takes no position on whether any Intellectual Property Rights or other rights that might be claimed to cover the implementation or use of the technology in this document are valid or in scope, or on whether licenses under those rights might be available. It also doesn't claim to have independently checked for any such rights. Info about how rights work for RFC documents is in BCP 78 and BCP 79 📖
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the results of an attempt to get a general license or permission for implementers or users to use these proprietary rights can be found in the IETF online IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to tell it about any copyrights, patents or patent applications, or other proprietary rights that may cover technology needed to implement this standard. Send that info to ietf-ipr@ietf.org 🦋
📦 where this page comes from
- ietf: rfc5242 2008-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms