RFC 5051

RFC 5051 💖 Part 1 of 2

i;unicode-casemap - Simple Unicode Collation Algorithm ✨

Network Working Group 🌸 Standards Track 🎀 October 2007

Author: Mark R. Crispin (University of Washington) 💅 Obsoletes / Updates: this doc's title block doesn't list anything it obsoletes or updates 🧚

---

Status of This Memo 📋

okay so this is an Internet standards track protocol 🦋 it's for the whole Internet community, and it's literally asking for discussion and suggestions for improvements 💬 for the current standardization state and status, go check the "Internet Official Protocol Standards" (STD 1) 👑 and yes, distribution of this memo is unlimited 🫶

Abstract ✨

this RFC describes "i;unicode-casemap", a simple case-insensitive collation for Unicode strings 🌈 it gives you equality, substring, and ordering operations, and it's giving clean, simple vibes 💜

1. Introduction 🎀

so the "i;ascii-casemap" collation from [COMPARATOR] is super easy to implement and gives case-independent comparisons for the 26 Latin alphabetics 🔤 it's the default and/or baseline comparator in some application protocols, e.g., [IMAP-SORT] 💅

BUT ngl, "i;ascii-casemap" does not give satisfactory results with non-ASCII characters 😭 with a modest extension, it's possible to get a more sophisticated collation with way better multilingual applicability 🌍 this extension gives case-independent comparisons for a much greater number of characters, and it also collates characters with diacriticals alongside the non-diacritical character forms ✨

this collation, "i;unicode-casemap", is meant to be an alternative to "i;ascii-casemap", and it's the preferred one 👑 it does not replace the "i;basic" collation described in [BASIC] 🦋

2. Unicode Casemap Collation Description 🧚

the "i;unicode-casemap" collation is simple and case-insensitive in how it treats characters 💖 it provides equality, substring, and ordering operations 🔥 the validity test operation returns "valid" for any input, so no drama there 💅

this collation allows strings in arbitrary (and mixed) character sets 🌸 as long as the character set for each string is identified and it's possible to convert the string to Unicode 🦄 strings with an unidentified character set and/or ones that can't be converted to Unicode are not rejected, they're just treated as binary 🙈

each input string gets prepared by converting it to a "titlecased canonicalized UTF-8" string 🎀 you do this in the following steps, using UnicodeData.txt ([UNICODE-DATA]) 📖

• (1) Get a Unicode codepoint from the input string. 🍓
  • (1)(a) If the input string is in a known charset that can be converted to Unicode, a sequence in the string's charset gets read and checked for validity according to the rules of that charset ✨ if the sequence is valid, it's converted to a Unicode codepoint 💖 Note that for input strings in UTF-8, the UTF-8 sequence must be valid according to the rules of [UTF-8]; e.g., overlong UTF-8 sequences are invalid 🙅‍♀️
  • (1)(b) If the input string is in an unknown charset, or an invalid sequence occurs in step (1)(a), conversion ceases 💀 no further preparation happens, any partial preparation results get thrown out, and the original string is used unchanged with the i;octet comparator 😭
• (2) Run these steps on the codepoint from step (1)(a), using UnicodeData.txt ([UNICODE-DATA]). 🔥
  • (2)(a) If the codepoint has a titlecase property in UnicodeData.txt (this is normally the same as the uppercase property), it gets converted to the codepoints in the titlecase property 👑
  • (2)(b) If the resulting codepoint from (2)(a) has a decomposition property of any type in UnicodeData.txt, it gets converted to the codepoints in the decomposition property 🌈 this step gets applied recursively to each resulting codepoint until no more decomposition is possible (effectively Normalization Form KD) 🦋
• (3) The resulting codepoint(s) from step (2) get appended, in UTF-8 format, to the "titlecased canonicalized UTF-8" string 💜
• (4) Repeat from step (1) until there's no more data in the input string 🔁

here's an example, because it's iconic 🤩 codepoint U+01C4 (LATIN CAPITAL LETTER DZ WITH CARON) has a titlecase property of U+01C5 (LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON) ✨ codepoint U+01C5 has a decomposition property of U+0044 (LATIN CAPITAL LETTER D) U+017E (LATIN SMALL LETTER Z WITH CARON) 🎀 and U+017E has a decomposition property of U+007A (LATIN SMALL LETTER Z) U+030c (COMBINING CARON) 🧚 none of U+0044, U+007A, or U+030C have any decomposition properties, so U+01C4 gets converted to U+0044 U+007A U+030C by this step 🫶

after you've done the preparation process on each string, the equality, ordering, and substring operations work the same as they do for i;octet 💅

it's permitted to use an alternative implementation of the preparation process as long as it produces the same results 🌸 for example, it might be more convenient to convert all input strings to a sequence of UTF-16 or UTF-32 values before doing any of the step (2) actions 💖 similarly, if all input strings are (or are convertible to) Unicode, it may be possible to use UTF-32 instead of UTF-8 in step (3) 🦄

Note: UTF-16 is unsuitable as an alternative to UTF-8 in step (3), because UTF-16 surrogates will cause i;octet to collate codepoints U+E0000 through U+FFFF after non-BMP codepoints 😱

this collation is not locale sensitive 🙅‍♀️ so be careful when you use OS-supplied functions to implement it 🫠 functions like strcasecmp and toupper are sometimes locale sensitive and can inconsistently casemap letters 💀

the i;unicode-casemap collation is well suited to use with many Internet protocols and computer languages 💻 using it with natural language is often inappropriate though 🙈 even though the collation apparently supports languages like Swahili and English, in the real world it tends to mis-sort a bunch of types of string:

• people and place names containing scripts that aren't collated according to "alphabetical order" 🗺️
• words with characters that have diacriticals 🇩🇪 however, i;unicode-casemap generally does a better job than i;ascii-casemap for most (but not all) languages 🌈 for example, German umlaut letters sort correctly, but some Scandinavian letters won't 😬
• names such as "Lloyd" (which in Welsh sorts after "Lyon", unlike in English) 🏴󠁧󠁢󠁷󠁬󠁳󠁿
• strings containing other non-letter symbols, e.g., euro and pound sterling symbols, quotation marks other than '"', dashes/hyphens, etc. 💸

3. Unicode Casemap Collation Registration 📝

here's the registration record, exactly as it's written 🎀

<?xml version='1.0'?>
<!DOCTYPE collation SYSTEM 'collationreg.dtd'>
<collation rfc="5051" scope="global" intendedUse="common">
<identifier>i;unicode-casemap</identifier>
<title>Unicode Casemap</title>
<operations>equality order substring</operations>
<specification>RFC 5051</specification>
<owner>IETF</owner>
<submitter>mrc@cac.washington.edu</submitter>
</collation>

Part 1 ends here 🌈✨

Full Copyright Statement 💖

Okay bestie, the legal stuff, but make it cute 🎀 Copyright (C) The IETF Trust (2007) 🦋

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. 💅

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 🙈

---

Intellectual Property 🌸

Okay so the IETF is not taking a side on whether any IPR (intellectual property rights) claims are valid, or how much they cover the tech in this doc, or whether any license for them even exists. 🫶 It also isn't saying it looked for any of those rights itself. Lowkey a very fair boundary. ✨ If you want the rules on RFC rights, they live in BCP 78 and BCP 79. 📚

Copies of IPR disclosures sent to the IETF Secretariat, any promises of licenses, or the results of trying to get a general license or permission to use proprietary rights for implementing this spec can all be found in the IETF's online IPR repository at http://www.ietf.org/ipr. 🔥

And the IETF invites anyone to tell it about any copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to implement this standard. 👑 Send that info to the IETF at ietf-ipr@ietf.org. 💜