RFC 4790

Internet Application Protocol Collation Registry

📅 published March 2007, from ietf

RFC 4790 ✨ Internet Application Protocol Collation Registry

Okay bestie, this is the RFC that makes IMAP, ACAP and friends stop reinventing string comparison 💅🎀

  • Number: RFC 4790 🌸
  • Title: Internet Application Protocol Collation Registry
  • Authors: C. Newman (Sun Microsystems), M. Duerst (Aoyama Gakuin University), A. Gulbrandsen (Oryx) 🫶
  • Working group: Network Working Group
  • Category: Standards Track 👑
  • Date: March 2007 📅
  • Obsoletes / Updates: The title block doesn't list anything this RFC obsoletes or updates. 🧚

Status of This Memo ✨

This document is an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 🦋 For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 💜

Copyright (C) The IETF Trust (2007).

Abstract 🍓

Lots of Internet application protocols do string-based lookup, searching, or sorting 😭 But searching and sorting international strings is a HUGE, not-fully-explored problem, and it's outside the IETF's area of expertise 💀 So instead of trying to solve all of that at once, this spec builds an abstraction framework. Application protocols can precisely name a comparison function, and the set of comparison functions can grow later on ✨

Table of Contents 🧚

1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
  1.1.  Conventions Used in This Document  . . . . . . . . . . . .  4
2.  Collation Definition and Purpose . . . . . . . . . . . . . . .  4
  2.1.  Definition . . . . . . . . . . . . . . . . . . . . . . . .  4
  2.2.  Purpose  . . . . . . . . . . . . . . . . . . . . . . . . .  4
  2.3.  Some Other Terms Used in this Document . . . . . . . . . .  5
  2.4.  Sort Keys  . . . . . . . . . . . . . . . . . . . . . . . .  5
3.  Collation Identifier Syntax  . . . . . . . . . . . . . . . . .  6
  3.1.  Basic Syntax . . . . . . . . . . . . . . . . . . . . . . .  6
  3.2.  Wildcards  . . . . . . . . . . . . . . . . . . . . . . . .  6
  3.3.  Ordering Direction . . . . . . . . . . . . . . . . . . . .  7
  3.4.  URIs . . . . . . . . . . . . . . . . . . . . . . . . . . .  7
  3.5.  Naming Guidelines  . . . . . . . . . . . . . . . . . . . .  7
4.  Collation Specification Requirements . . . . . . . . . . . . .  8
  4.1.  Collation/Server Interface . . . . . . . . . . . . . . . .  8
  4.2.  Operations Supported . . . . . . . . . . . . . . . . . . .  8
    4.2.1.  Validity . . . . . . . . . . . . . . . . . . . . . . .  9
    4.2.2.  Equality . . . . . . . . . . . . . . . . . . . . . . .  9
    4.2.3.  Substring  . . . . . . . . . . . . . . . . . . . . . . .  9
    4.2.4.  Ordering . . . . . . . . . . . . . . . . . . . . . . . 10
  4.3.  Sort Keys  . . . . . . . . . . . . . . . . . . . . . . . . 10
  4.4.  Use of Lookup Tables . . . . . . . . . . . . . . . . . . . 11
5.  Application Protocol Requirements  . . . . . . . . . . . . . . 11
  5.1.  Character Encoding . . . . . . . . . . . . . . . . . . . . 11
  5.2.  Operations . . . . . . . . . . . . . . . . . . . . . . . . 11
  5.3.  Wildcards  . . . . . . . . . . . . . . . . . . . . . . . . 12
  5.4.  String Comparison  . . . . . . . . . . . . . . . . . . . . 12
  5.5.  Disconnected Clients . . . . . . . . . . . . . . . . . . . 12
  5.6.  Error Codes  . . . . . . . . . . . . . . . . . . . . . . . 13
  5.7.  Octet Collation  . . . . . . . . . . . . . . . . . . . . . 13
6.  Use by Existing Protocols  . . . . . . . . . . . . . . . . . . 13
7.  Collation Registration . . . . . . . . . . . . . . . . . . . . 14
  7.1.  Collation Registration Procedure . . . . . . . . . . . . . 14
  7.2.  Collation Registration Format  . . . . . . . . . . . . . . 15
    7.2.1.  Registration Template  . . . . . . . . . . . . . . . . 15
    7.2.2.  The Collation Element  . . . . . . . . . . . . . . . . 15
    7.2.3.  The Identifier Element . . . . . . . . . . . . . . . . 16
    7.2.4.  The Title Element  . . . . . . . . . . . . . . . . . . 16
    7.2.5.  The Operations Element . . . . . . . . . . . . . . . . 16
    7.2.6.  The Specification Element  . . . . . . . . . . . . . . 16
    7.2.7.  The Submitter Element  . . . . . . . . . . . . . . . . 16
    7.2.8.  The Owner Element  . . . . . . . . . . . . . . . . . . 16
    7.2.9.  The Version Element  . . . . . . . . . . . . . . . . . 17
    7.2.10. The Variable Element . . . . . . . . . . . . . . . . . 17
  7.3.  Structure of Collation Registry  . . . . . . . . . . . . . 17
  7.4.  Example Initial Registry Summary . . . . . . . . . . . . . 18
8.  Guidelines for Expert Reviewer . . . . . . . . . . . . . . . . 18
9.  Initial Collations . . . . . . . . . . . . . . . . . . . . . . 19
  9.1.  ASCII Numeric Collation  . . . . . . . . . . . . . . . . . 20
    9.1.1.  ASCII Numeric Collation Description  . . . . . . . . . 20
    9.1.2.  ASCII Numeric Collation Registration . . . . . . . . . 20
  9.2.  ASCII Casemap Collation  . . . . . . . . . . . . . . . . . 21
    9.2.1.  ASCII Casemap Collation Description  . . . . . . . . . 21
    9.2.2.  ASCII Casemap Collation Registration . . . . . . . . . 22
  9.3.  Octet Collation  . . . . . . . . . . . . . . . . . . . . . 22
    9.3.1.  Octet Collation Description  . . . . . . . . . . . . . 22
    9.3.2.  Octet Collation Registration . . . . . . . . . . . . . 23
10. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 23
11. Security Considerations  . . . . . . . . . . . . . . . . . . . 23
12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 23
13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 24
  13.1. Normative References . . . . . . . . . . . . . . . . . . . 24
  13.2. Informative References . . . . . . . . . . . . . . . . . . 24

1. Introduction 🌸

ACAP [11] (the Application Configuration Access Protocol) introduced the idea of a comparator, which this document calls a collation 🦋 but it never made an IANA registry for them 😤 Now that we've got stringprep [6] and the Unicode Collation Algorithm [7], it's finally time to create that registry and fill it with some starter values that work for an international community 🌍 This spec replaces and generalizes ACAP's comparator definition, and creates a collation registry ✨

1.1. Conventions Used in This Document 🎀

The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this document are to be interpreted as defined in "Key words for use in RFCs to Indicate Requirement Levels" [1] 💅

The attribute syntax specifications use the Augmented Backus-Naur Form (ABNF) [2] notation, including the core rules defined in Appendix A. The ABNF production "Language-tag" is imported from Language Tags [5], and "reg-name" is imported from URI: Generic Syntax [4] 🫶

2. Collation Definition and Purpose 🧚

2.1. Definition 💜

A collation is a named function. It takes two arbitrary length strings as input, and it can be used to perform one or more of three basic comparison operations: equality test, substring match, and ordering test 🔥

2.2. Purpose 🦋

Collations are an abstraction for comparison functions, so the same comparison functions can be used across multiple protocols 🌈 Someone with the right expertise can specify the details of a comparison operation on their own, separate from the application protocols that use that collation. It's kind of like how a charset [13] keeps octet-to-character mapping details out of a protocol spec like MIME [9], or how SASL [10] keeps the details of an authentication mechanism out of a protocol spec like ACAP [11] 👑

Here's a little diagram to show why this abstraction slaps:

  +-------------------+                         +-----------------+
  | IMAP i18n SEARCH  |--+                      | Basic           |
  +-------------------+  |                   +--| Collation Spec  |
                         |                   |  +-----------------+
  +-------------------+  |  +-------------+  |  +-----------------+
  | ACAP i18n SEARCH  |--+--| Collation   |--+--| A stringprep    |
  +-------------------+  |  | Registry    |  |  | Collation Spec  |
                         |  +-------------+  |  +-----------------+
  +-------------------+  |                   |  +-----------------+
  | ...other protocol |--+                   |  | locale-specific |
  +-------------------+                      +--| Collation Spec  |
                                                +-----------------+

So IMAP, ACAP, and future application protocols with international search can just say how they plug into the collation registry, instead of every single protocol spec having to list every collation it supports 🙈

2.3. Some Other Terms Used in this Document 🍓

"Client", "server", and "protocol" are used in kind of unusual senses here, so heads up 👀

Client means a user, or a program acting directly on behalf of a user. That could be a mail reader acting as an IMAP client, an interactive shell where the user types protocol commands/requests directly, or a script or program the user wrote 💖

Server means a program that performs services the client asks for. That could be a traditional server like an HTTP server, or a Sieve [14] interpreter running a Sieve script that a user wrote. A server needs to use the operations provided by collations in order to fulfill the client's requests 🧚

Protocol describes how the client tells the server what it wants done, and (if applicable) how the server tells the client about the results. By this definition IMAP is a protocol, and so is the Sieve language 🌸

2.4. Sort Keys ✨

One part of a collation is a transformation, which turns a string into a sort key, and that sort key is what gets used while sorting 🔑

The transformation can go anywhere from an identity mapping (e.g., the i;octet collation Section 9.3) all the way to a mapping that makes the string unreadable to a human 🤯

This is an implementation detail of collations or servers. A protocol SHOULD NOT expose it to clients, because some collations leave the sort key's format up to the implementation, and current conformant implementations are known to use different formats 🙅‍♀️

3. Collation Identifier Syntax 🎀

3.1. Basic Syntax 💅

The collation identifier itself is a single US-ASCII string. The identifier MUST NOT be longer than 254 characters, and it follows this grammar 💖

     collation-char  = ALPHA / DIGIT / "-" / ";" / "=" / "."

     collation-id    = collation-prefix ";" collation-core-name
                       *collation-arg

     collation-scope = Language-tag / "vnd-" reg-name

     collation-core-name = ALPHA *( ALPHA / DIGIT / "-" )

     collation-arg   = ";" ALPHA *( ALPHA / DIGIT ) "="
                       1*( ALPHA / DIGIT / "." )

Note: the ABNF production "Language-tag" is imported from Language Tags [5], and "reg-name" is imported from URI: Generic Syntax [4] 🦋

There's a special identifier called "default" 🌈 For protocols that have a default collation, "default" refers to that collation. For other protocols, the identifier "default" MUST match no collations, and servers SHOULD treat it the same way they treat nonexistent collations 👑

3.2. Wildcards 🎀

okay so bestie, the string a client uses to pick a collation MAY have one or more wildcard ("*") characters, and each one matches zero or more collation-chars 🦋. Wildcard characters MUST NOT be right next to each other, so no double stars 🙅‍♀️. If your wildcard matches a bunch of collations, the server SHOULD try to grab a widely useful one instead of a narrowly useful one ✨

     collation-wild  =  ("*" / (ALPHA ["*"])) *(collation-char ["*"])
                         ; MUST NOT exceed 254 characters total

3.3. Ordering Direction 🧚

when a collation identifier is being used for ordering, it MAY get a "+" or "-" prefix to say which direction you want 🔥 "+" does nothing to the ordering, but "-" flips the result of the ordering operation. Generally, collation-order is what a client sends when it asks for a collation, and collation-selected is what the server sends back to say which one it picked 💜

     collation-selected =  ["+" / "-"] collation-id

     collation-order =  ["+" / "-"] collation-wild

3.4. URIs 🌸

some protocols use URIs [4] to point at collations instead of plain tokens, and there's a special section of the IANA URL space set aside just for that 🌈 The "collation-uri" form refers to one specific named collation, and that registration might not actually be there. The "collation-auri" form is more of an abstract name for an ordering, a collation pattern, or a vendor private collator 👑

     collation-uri   =  "http://www.iana.org/assignments/collation/"
                        collation-id ".xml"

     collation-auri  =  ( "http://www.iana.org/assignments/collation/"
                        collation-order ".xml" ) / other-uri

     other-uri       =  <absoluteURI>
                     ;  excluding the IANA collation namespace.

3.5. Naming Guidelines 💅

this spec doesn't lock down exactly how collation identifiers have to be built, but naming consistency matters a lot, so here are some starter guidelines 🫶

Collation identifiers meant for an international audience usually start with "i;". Ones meant for a particular language or locale usually start with a language tag [5] followed by a ";". After that first ";" comes the name of the general collation algorithm, then a bunch of algorithm modifications separated by ";". Parameterized modifications use "=" to split the parameter from its value. The version numbers of any lookup tables the algorithm uses SHOULD be included as parameterized modifications 📚

Identifiers that look like *;vnd-hostname;* are reserved for vendor-specific collations made by whoever owns the hostname after the "vnd-" prefix. Like, vnd-example.com is for the vendor example.com 🍓 Registering these collations (or the whole namespace), with the intended use set to "Vendor", is encouraged when there's a public spec or an open-source implementation, but it's not required ✨

4. Collation Specification Requirements 💖

4.1. Collation/Server Interface 🦋

the collation itself decides what it works on. Most collations are expected to operate on character strings. The i;octet (Section 9.3) collation operates on octet strings, and the i;ascii-numeric (Section 9.1) operation operates on numbers 🔢

this spec describes the collation interface in terms of octet strings. However, implementations may choose to use character strings instead. Those implementations may not be able to implement e.g., i;octet. Since i;octet isn't currently mandatory to implement for any protocol, this should not be a problem 🙈

4.2. Operations Supported 💅

a collation specification MUST say which of the three basic operations it supports (equality, substring, ordering) and how to perform each supported operation on any two input character strings, including empty strings 🎀 Collations must be deterministic, i.e., given a collation with a specific identifier and any two fixed input strings, the result MUST be the same for the same operation 🔁

in general, collation operations should act the way their names suggest. A collation might be new, but the operations aren't, so a new collation's operations should be similar to the older ones. For example, a date/time collation should not offer a "substring" operation that would morph IMAP substring SEARCH into, say, a date-range search 📅

a non-obvious thing about the rules for each collation operation is that, for any single collation, either none or all of the operations can return "undefined". So it's not possible to have an equality operation that never returns "undefined" and a substring operation that sometimes does 🤯

4.2.1. Validity 🫧

the validity test takes one string as an argument. It returns valid if its input string is a valid input to the collation's other operations, and invalid if not. (In other words, a string is valid if it's equal to itself according to the collation's equality operation.) 💕

every collation provides the validity test. It MUST NOT be listed separately in the collation registration 🙅‍♀️

4.2.2. Equality 💜

the equality test always returns "match" or "no-match" when it gets valid input, and it MAY return "undefined" if one or both input strings aren't valid 🌷

the equality test MUST be reflexive and symmetric. For valid input, it MUST be transitive 🔄

if a collation gives you a substring or an ordering test, it MUST also provide an equality test. The substring and/or ordering tests MUST be consistent with the equality test 🫶

the return values of the equality test are called "match", "no-match" and "undefined" in this doc 🎀

4.2.3. Substring 🌈

the substring matching operation figures out whether the first string is a substring of the second string, i.e., whether one or more substrings of the second string are equal to the first, as defined by the collation's equality operation 🔍

a collation that supports substring matching will automatically support two special cases: prefix and suffix matching, if the application protocol supports those special cases. It returns "match" or "no-match" when it gets valid input, and "undefined" when it gets invalid input 🧁

application protocols MAY return position information for substring matches. If they do, the position info SHOULD include both the starting offset and the ending offset for each match. This matters because fancier collations can match strings of unequal length (for example, a pre-composed accented character can match a decomposed accented character) 🇫🇷 In general, overlapping matches SHOULD be reported (like how "ana" shows up twice inside "banana"), although sometimes a collation might decide not to. For example, in a collation that treats all whitespace sequences as identical, the substring operation could be defined so that " 1 " (SP "1" SP) is reported just once inside " 1 " (SP SP "1" SP SP), instead of four times (SP SP "1" SP, SP "1" SP, SP "1" SP SP and SP SP "1" SP SP). Those four matches are, in a sense, the same match, so it's giving tidy 🧹

a string is a substring of itself, and the empty string is a substring of all strings 🌟

heads up that the substring operation of some collations can match strings of unequal length. Again, a pre-composed accented character can match a decomposed accented character. The Unicode Collation Algorithm [7] goes into more detail on this 📖

the return values of the substring operation are called "match", "no-match", and "undefined" in this doc 🎀

4.2.4. Ordering 🦢

the ordering operation decides how two strings are ordered. It MUST be reflexive. For valid input, it MUST be transitive and trichotomous 📏

ordering returns "less" if the first string comes before the second string in the collation, "greater" if the second string comes before the first, and "equal" if the two strings are equal as defined by the collation's equality operation. If one or both strings are invalid, the ordering result is "undefined" 💅

when the collation is used with a "+" prefix, it behaves the same as with no prefix. When it's used with a "-" prefix, the result of the collation's ordering operation MUST be reversed 🔁

the return values of the ordering operation are called "less", "equal", "greater", and "undefined" in this doc ✨

4.3. Sort Keys 🔑

a collation specification SHOULD describe the internal transformation algorithm used to generate sort keys. That algorithm can be applied to individual strings, and the result can be stored to potentially speed up future comparisons 🏎️ A collation MAY say that its sort key is generated by the identity function. The sort key may have no meaning to a human, and it may not be valid input to the collation 🫠

4.4. Use of Lookup Tables 📚

some collations use customizable lookup tables, e.g., because the tables depend on locale, and they might get changed after the software ships 🌍 Collations that use more than one customizable lookup table in a documented format MUST assign numbers to the tables they use. That way an application protocol command can access the tables a server collation uses, so clients and servers are using the same tables 🤝

5. Application Protocol Requirements 👑

this section covers what an application protocol needs to think about if it offers searching, substring matching, and/or sorting, and if it lets people use characters outside the US-ASCII charset 🌏

5.1. Character Encoding 🫧

the protocol spec has to be super clear about which characters (not just octets) the collations are applied to. You can do this by describing the protocol itself in terms of characters (like in a query language), by picking a single character encoding for the protocol (like UTF-8 [3]), or by carefully describing the relevant issues of character encoding labeling and conversion. In that last case, you have to think about how to handle unknown charsets, any charsets that are mandatory-to-implement, any byte-order issues that might apply, and any transfer encodings that need to be supported 🧠

5.2. Operations ✨

okay so the protocol has to spell out which operations from this spec (equality matching, substring matching, and ordering) can actually be invoked in it, and how 🧠. there can be more than one way to call an operation, bestie 💅. the protocol MUST give the client a way to pick the collation used for equality matching, substring matching, and ordering 🎀.

if a protocol needs a total ordering but the chosen collation doesn't give one, because the ordering operation returns "undefined" at least once, the recommended fallback is to sort all the invalid strings after the valid ones and use i;octet to order the invalid ones 🦋.

ngl this is so clever. even though the collation's substring function gives back a list of matches, the protocol doesn't have to pass all of that along to the client 🙈. it can just send the first matching substring, or even only the info that a substring search matched. that's how collations can work with protocols where "x is a substring of y" just comes back true or false ✨.

now the important bit 😭. if the protocol gives positional info for substring results, that info SHOULD fully pin down the matching substring(s) in the result, no matter how long the search string was. returning the start and end offset works, and so does the start offset plus a length. just returning the start offset is NOT acceptable 💀. the reason is that fancy collations can treat strings of different lengths as equal (think pre-composed vs decomposed accented characters) 🌸.

5.3. Wildcards 🌈

the protocol MUST say whether it allows wildcards in collation identifiers 🍓. if it does allow them, then:

  • the protocol MUST specify how comparisons behave when there's no explicit collation negotiation, or when a collation of "default" is requested. it MAY say that the default collation in those cases depends on server configuration 🔧.
  • the protocol SHOULD provide a way to list the available collations that match a given wildcard pattern or patterns 📋.

5.4. String Comparison 💜

if a protocol compares strings in any non-trivial way, using a collation might be a good move 👑. lots of protocols use case-independent strings, and in plenty of cases a simple ASCII mapping to upper and lower case is totally fine ✨. but sometimes a specifiable collation is better. like, a server can treat "i" and "I" as equivalent in Italy but different in Turkey, where there's a dotted upper-case" I" and a dotless lower-case "i" 🇹🇷.

protocol designers should think about whether to use a specifiable collation for each case. keywords often have different needs than user variables, and search arguments might need something else again 🫶.

5.5. Disconnected Clients 🌙

if the protocol supports disconnected clients, and the collation uses configurable tables (say, to support locale-specific extensions), the client might not be able to reproduce the server's collation operations while it's offline 📴.

a way to download those tables has been talked about, but it isn't in this spec, because the problem isn't well understood yet 🤷‍♀️.

5.6. Error Codes 🚨

the protocol spec should consider assigning protocol error codes for these situations:

  • the client asks for a collation by identifier or pattern, but no implemented collation matches that pattern 🙅‍♀️.
  • the client tries to use a collation for an operation that the collation doesn't support. for example, trying to use the "i;ascii-numeric" collation for substring matching 😬.
  • the client uses an equality or substring matching collation and gets an error. it might be useful to tell apart the two input strings, especially when one comes from the client and the other is stored on the server 🧐. it might also make sense to call out the specific case of an invalid UTF-8 string 💅.

5.7. Octet Collation 🔐

the i;octet (Section 9.3) collation is only usable with protocols based on octet-strings. clients and servers MUST NOT use i;octet with other protocols 🚫. and if the protocol lets collations work with data structures other than strings, the protocol MUST describe the default behaviour for a collation with those structures 🎀.

6. Use by Existing Protocols 📚

This section is informative.

okay so both ACAP [11] and Sieve [14] are standards track specs that were already using collations before this spec and the registry existed 🕰️. they don't meet all the application protocol requirements from Section 5, though 😭.

these protocols let you use the i;octet (Section 9.3) collation working directly on UTF-8 data, which is how they handle it 🍓.

in Sieve, every match is either true or false. so Sieve servers must treat "undefined" and "no-match" results of the equality and substring operations as false, and only "match" as true ✨.

in ACAP and Sieve, there are no invalid strings. in this document's terms, invalid strings sort after valid strings 🦋.

IMAP [15] also collates, but that's only explicit when the COMPARATOR [17] extension is used 🤔. the built-in IMAP substring operation and the ordering from the SORT [16] extension might not meet the requirements in this document 🙈.

other protocols might be in a similar spot 🌸.

in IMAP, the default collation is i;ascii-casemap, because its operations are understood to match IMAP's built-in operations 💜.

7. Collation Registration 🎀

7.1. Collation Registration Procedure 🗂️

the IETF will create a mailing list, collation@ietf.org, for public discussion of collation proposals before they get registered 💬. using the list is strongly encouraged 🫶. the IESG will appoint a designated expert who will keep an eye on the collation@ietf.org list and review registrations 👀.

the registration process starts when a completed registration template gets sent to iana@iana.org and collation@ietf.org. the designated expert is expected to tell IANA and the submitter within two weeks whether the registration is approved, approved with minor changes, or rejected with cause ⏳. if it's rejected with cause, it can be re-submitted once the concerns listed in the cause are addressed 🔁. decisions from the designated expert can be appealed to the IESG Applications Area Director, and then to the IESG. they follow the normal appeals procedure for IESG decisions ⚖️.

collation registrations in a standards track, BCP, or IESG-approved experimental RFC are owned by the IETF. changes to those follow the normal procedures for updating such documents 🏛️. collation registrations in other RFCs are owned by the RFC author(s). other collation registrations are owned by the individual(s) listed in the contact field of the registration, and IANA will preserve that info 📇.

if the registration is a change to an existing collation, it MUST be approved by the owner 🙋‍♀️. if the owner can't be contacted for one month, and the designated expert thinks the change is necessary, the IESG MAY re-assign ownership to an appropriate party 🔄.

7.2. Collation Registration Format 📝

registering a collation means sending a well-formed XML document to collation@ietf.org and iana@iana.org 💌.

7.2.1. Registration Template 🧾

here's the template for the registration 💖:

<?xml version='1.0'?>
<!DOCTYPE collation SYSTEM 'collationreg.dtd'>
<collation rfc="YYYY" scope="global" intendedUse="common">
  <identifier>collation identifier</identifier>
  <title>technical title for collation</title>
  <operations>equality order substring</operations>
  <specification>specification reference</specification>
  <owner>email address of owner or IETF</owner>
  <submitter>email address of submitter</submitter>
  <version>1</version>
</collation>

7.2.2. The Collation Element 🌟

the root of the registration document MUST be a <collation> element 👑. the collation element holds the other elements in the registration, which are described in the sub-subsections that follow, in the order given here 📑.

the <collation> element MAY include an "rfc=" attribute if the specification is in an RFC. the "rfc=" attribute only gives the number of the RFC, with no prefix like "RFC" and no suffix like ".txt" 🔢.

the <collation> element MUST include a "scope=" attribute, and it MUST have one of the values "global", "local", or "other" 🌍.

the <collation> element MUST include an "intendedUse=" attribute, which must have one of the values "common", "limited", "vendor", or "deprecated". collation specs intended for "common" use are expected to reference standards from standards bodies with serious experience with the details of international character sets 🌏.

heads up, future revisions of this spec might add more function types, plus more XML attributes, values, and elements 🔮. any system that automatically parses these XML documents MUST take that into account to stay future-compatible 🫶.

7.2.3. The Identifier Element 🪪

the <identifier> element gives the precise identifier of the collation, e.g., i;ascii-casemap. the <identifier> element is mandatory 💯.

7.2.4. The Title Element 🏷️

the <title> element gives the title of the collation. the <title> element is mandatory ✅.

7.2.5. The Operations Element ⚙️

the <operations> element lists which of the three operations ("equality", "order" or "substring") the collation provides, separated by single spaces. the <operations> element is mandatory 🙌.

7.2.6. The Specification Element 📖

the <specification> element describes where to find the specification. it is mandatory, and it MAY have a URI attribute 🔗. there can be more than one <specification> element, and together they form the specification 🧩.

if it's discovered that parts of a collation specification conflict, a new revision of the collation is necessary, and the collation@ietf.org mailing list should be notified 📣.

7.2.7. The Submitter Element ✉️

the <submitter> element provides an RFC 2822 [12] email address for the person who submitted the registration. it is optional if the <owner> element contains an email address 💌.

there may be more than one <submitter> element 👯‍♀️.

7.2.8. The Owner Element 👑

the <owner> element contains either the four letters "IETF" or an email address of the owner of the registration. the <owner> element is mandatory 🙋‍♀️. there may be more than one <owner> element. if there is, all owners are equal, and each owner can speak for all of them 🤝.

7.2.9. The Version Element 🔖

the <version> element MUST be included when the registration is likely to be revised, or has been revised in a way that changes the results for one or more input strings 🔄. the <version> element is otherwise optional 💫.

7.2.10. The Variable Element 🎛️

the <variable> element specifies an optional variable that controls the collation's behaviour, like whether it's case sensitive 🔠. the <variable> element is optional. when <variable> is used, it must contain <name> and <default> elements, and it may contain one or more <value> elements 🧃.

7.2.10.1. The Name Element 🏷️

the <name> element specifies the name value of a variable. the <name> element is mandatory 💖.

7.2.10.2. The Default Element 🎀

okay so the <default> element is where you spell out the default value of a variable 💖 and it's literally mandatory, so you can't skip it 🙈

7.2.10.3. The Value Element ✨

the <value> element lets you list a legal value of a variable 🦋 it's optional, but if you do include one or more <value> elements, then ONLY those values are legal 💅 and if you include none, the variable's legal values don't form an enumerated set, and the rules MUST be specified in an RFC accompanying the registration 🫶 that part is non-negotiable bestie 😭

7.3. Structure of Collation Registry 🌸

once the registration is approved, IANA stores each XML registration document at a URL of the form http://www.iana.org/assignments/collation/collation-id.xml 🧚 where collation-id is the content of the identifier element in the registration 💜 the submitter AND the designated expert are both responsible for checking the XML is well-formed ✨ the registration document SHOULD avoid using new elements 🙈 wait I need to be precise here: the original says "should avoid", so it's a soft suggestion, not a SHOULD in capitals 💅 if new elements are necessary, it is important to stay consistent with other registrations 🍓

IANA also maintains a text summary of the registry under the name http://www.iana.org/assignments/collation/collation-index.html 🌈 this summary has four sections 👑

  • the first section is for collations intended for common use 🔥 this one is for collation registrations published in IESG-approved RFCs, or for locally scoped collations from the primary standards body for that locale 💖
  • the designated expert is encouraged to reject collation registrations with an intended use of "common" if the expert believes it should be "limited" 🦋 because it's desirable to keep the number of "common" registrations small and high quality 💅
  • the second section is reserved for limited-use collations 🌸
  • the third section is reserved for registered vendor-specific collations 🎀
  • the final section is reserved for deprecated collations 🥀

7.4. Example Initial Registry Summary ⭐

here's an example of how IANA might structure the initial registry summary.html file 🧚

  Collation                              Functions Scope Reference
  ---------                              --------- ----- ---------
Common Use Collations:
  i;ascii-casemap                        e, o, s   Local [RFC 4790]

Limited Use Collations:
  i;octet                                e, o, s   Other [RFC 4790]
  i;ascii-numeric                        e, o      Other [RFC 4790]

Vendor Collations:

Deprecated Collations:


References
----------
[RFC 4790]  Newman, C., Duerst, M., Gulbrandsen, A., "Internet
            Application Protocol Collation Registry", RFC 4790,
            Sun Microsystems, March 2007.
  1. Guidelines for Expert Reviewer 🕵️‍♀️

the expert reviewer appointed by the IESG has fairly broad latitude for this registry 💅 a bunch of collations are expected (especially customizations of the UCA for localized use) 🌈 but an explosion of collations (especially common-use ones) is NOT desirable for widespread interoperability 🙅‍♀️ it's important for the expert reviewer to give cause when rejecting a registration, and, when possible, to describe corrective action so the registration can proceed 🫶 here are some example reasons to reject a registration with cause:

  • The registration is not a well-formed XML document. 💀
  • The registration has an intended use of "common", but there is no evidence the collation will be widely deployed, so it should be listed as "limited". 🙈
  • The registration has an intended use of "common", but it is redundant with the functionality of a previously registered "common" collation. 😭
  • The registration has an intended use of "common", but the specification is not detailed enough to allow interoperable implementations by others. 💅
  • The collation identifier fails to precisely identify the version numbers of relevant tables to use. 🍓
  • The registration fails to meet one of the "MUST" requirements in Section 4. 🙈
  • The collation identifier fails to meet the syntax in Section 3. ✨
  • The collation specification referenced in the registration is vague or has optional features without a clear behavior specified. 🌸
  • The referenced specification does not adequately address security considerations specific to that collation. 🥺
  • The registration's operations are needlessly different from those of traditional operations. 🎀
  • The registration's XML is needlessly different from that of already registered collations. 💜
  1. Initial Collations 🍭

this section registers the three collations that were originally defined in [11], and are implemented in most [14] engines 💖 some of the behavior is perhaps not ideal, like i;ascii-casemap accepting non-ASCII input 😬 but compatibility with widely deployed code was judged more important than fixing them 🫶 some aspects are necessary to keep that compatibility 🌷

9.1. ASCII Numeric Collation 🔢

9.1.1. ASCII Numeric Collation Description ✨

the "i;ascii-numeric" collation is a simple one for arbitrarily-sized, unsigned decimal integer numbers stored as octet strings 🧚 US-ASCII digits (0x30 to 0x39) represent the digits of the numbers 💜 before converting from string to integer, the input string is truncated at the first non-digit character 🦋 all input is valid, and strings that don't start with a digit represent positive infinity ♾️ wild, right? 🤯

it supports equality and ordering, but NOT the substring operation 🙅‍♀️

the equality operation returns "match" if the two strings represent the same number (leading zeroes and trailing non-digits are disregarded), and "no-match" if they represent different numbers 💅

the ordering operation returns "less" if the first string represents a smaller number than the second, "equal" if they represent the same number, and "greater" if the first string represents a larger number than the second 🌈

some examples: "0" is less than "1", and "1" is less than "4294967298". "4294967298", "04294967298", and "4294967298b" are all equal. "04294967298" is less than "". "", "x", and "y" are equal. 🎀

9.1.2. ASCII Numeric Collation Registration 📝

<?xml version='1.0'?>
<!DOCTYPE collation SYSTEM 'collationreg.dtd'>
<collation rfc="4790" scope="other" intendedUse="limited">
  <identifier>i;ascii-numeric</identifier>
  <title>ASCII Numeric</title>
  <operations>equality order</operations>
  <specification>RFC 4790</specification>
  <owner>IETF</owner>
  <submitter>chris.newman@sun.com</submitter>
</collation>

9.2. ASCII Casemap Collation 🔠

9.2.1. ASCII Casemap Collation Description 💫

the "i;ascii-casemap" collation is a simple one that operates on octet strings and treats US-ASCII letters case-insensitively 🫶 it provides equality, substring, and ordering operations 🦋 all input is valid 💅 note that letters outside ASCII are NOT treated case-insensitively 🙈

its equality, ordering, and substring operations are as for i;octet, except that at first, the lower-case letters (octet values 97-122) in each input string are changed to upper case (octet values 65-90) 🧚

⚠️ care should be taken when using OS-supplied functions to implement this collation, because it is not locale sensitive 😭 functions like strcasecmp and toupper are sometimes locale sensitive, and may inappropriately map lower-case letters other than a-z to upper case 💀

the i;ascii-casemap collation is well-suited for use with many Internet protocols and computer languages 💖 using it with natural language is often inappropriate 🙃 even though it apparently supports languages like Swahili and English, in real-world use it tends to mis-sort a number of types of string:

  • people and place names containing non-ASCII, 🌍
  • words such as "naive" (if spelled with an accent, the accented character could push the word to the wrong spot in a sorted list), 📚
  • names such as "Lloyd" (which, in Welsh, sorts after "Lyon", unlike in English), 🏴
  • strings containing euro and pound sterling symbols, quotation marks other than '"', dashes/hyphens, etc. 💸

9.2.2. ASCII Casemap Collation Registration 📝

<?xml version='1.0'?>
<!DOCTYPE collation SYSTEM 'collationreg.dtd'>
<collation rfc="4790" scope="local" intendedUse="common">
  <identifier>i;ascii-casemap</identifier>
  <title>ASCII Casemap</title>
  <operations>equality order substring</operations>
  <specification>RFC 4790</specification>
  <owner>IETF</owner>
  <submitter>chris.newman@sun.com</submitter>
</collation>

9.3. Octet Collation 🦋

9.3.1. Octet Collation Description 🌷

the "i;octet" collation is a simple and fast one meant for binary octet strings rather than character data 💜 protocols that want to offer it have to do so by explicitly allowing it 🙌 if it's NOT explicitly allowed, it MUST NOT be used 🚫 it never returns an "undefined" result 🙅‍♀️ it provides equality, substring, and ordering operations 💅

the ordering algorithm goes like this 👇

  1. If both strings are the empty string, return the result "equal".
  2. If the first string is empty and the second is not, return the result "less".
  3. If the second string is empty and the first is not, return the result "greater".
  4. If both strings begin with the same octet value, remove the first octet from both strings and repeat this algorithm from step 1.
  5. If the unsigned value (0 to 255) of the first octet of the first string is less than the unsigned value of the first octet of the second string, then return "less".
  6. If this step is reached, return "greater".

this algorithm is roughly equivalent to the C library function memcmp, with appropriate length checks added 🧠

the matching operation returns "match" if the sorting algorithm would return "equal" 🎯 otherwise, it returns "no-match" 🙅‍♀️

the substring operation returns "match" if the first string is the empty string, or if there exists a substring of the second string of length equal to the length of the first string, which would result in a "match" result from the equality function 🔍 otherwise, it returns "no-match" 💔

9.3.2. Octet Collation Registration 📝

this collation is defined with intendedUse="limited" because it can only be used by protocols that explicitly allow it 🫶

<?xml version='1.0'?>
<!DOCTYPE collation SYSTEM 'collationreg.dtd'>
<collation rfc="4790" scope="global" intendedUse="limited">
  <identifier>i;octet</identifier>
  <title>Octet</title>
  <operations>equality order substring</operations>
  <specification>RFC 4790</specification>
  <owner>IETF</owner>
  <submitter>chris.newman@sun.com</submitter>
</collation>
  1. IANA Considerations 🌈

Section 7 defines how to register collations with IANA 📋 Section 9 defines a list of predefined collations that have been registered with IANA ✨

  1. Security Considerations 🚨

collations will normally be used with UTF-8 strings 💻 so the security considerations for UTF-8 [3], stringprep [6], and Unicode TR-36 [8] also apply, and are normative to this specification 🔐 this part is seriously important, bestie, don't skip it 😤💅

12. Acknowledgements 🎀

The authors want to give a huge shoutout to everyone who helped make this doc happen 🫶 That includes Brian Carpenter, John Cowan, Dave Cridland, Mark Davis, Spencer Dawkins, Lisa Dusseault, Lars Eggert, Frank Ellermann, Philip Guenther, Tony Hansen, Ted Hardie, Sam Hartman, Kjetil Torgrim Homme, Michael Kay, John Klensin, Alexey Melnikov, Jim Melton, and Abhijit Menon-Sen. You all are literally the real MVPs 👑✨

13. References 📚

13.1. Normative References

These are the must-read sources the spec leans on, so bookmark them 🔖

  • [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [2] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005.
  • [3] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.
  • [4] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", RFC 3986, January 2005.
  • [5] Phillips, A. and M. Davis, "Tags for Identifying Languages", BCP 47, RFC 4646, September 2006.
  • [6] Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, December 2002.
  • [7] Davis, M. and K. Whistler, "Unicode Collation Algorithm version 14", May 2005, http://www.unicode.org/reports/tr10/tr10-14.html.
  • [8] Davis, M. and M. Suignard, "Unicode Security Considerations", February 2006, http://www.unicode.org/reports/tr36/.

13.2. Informative References

These ones are more like extra reading for the curious 🧚

  • [9] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.
  • [10] Melnikov, A., "Simple Authentication and Security Layer (SASL)", RFC 4422, June 2006.
  • [11] Newman, C. and J. Myers, "ACAP -- Application Configuration Access Protocol", RFC 2244, November 1997.
  • [12] Resnick, P., "Internet Message Format", RFC 2822, April 2001.
  • [13] Freed, N. and J. Postel, "IANA Charset Registration Procedures", BCP 19, RFC 2978, October 2000.
  • [14] Showalter, T., "Sieve: A Mail Filtering Language", RFC 3028, January 2001.
  • [15] Crispin, M., "Internet Message Access Protocol - Version 4rev1", RFC 3501, March 2003.
  • [16] Crispin, M. and K. Murchison, "Internet Message Access Protocol - Sort and Thread Extensions", Work in Progress, May 2004.
  • [17] Newman, C. and A. Gulbrandsen, "Internet Message Access Protocol Internationalization", Work in Progress, January 2006.

Authors' Addresses 💌

Meet the people behind the doc 💖

Chris Newman
Sun Microsystems
1050 Lakes Drive
West Covina, CA  91790
USA

EMail: chris.newman@sun.com
Martin Duerst
Aoyama Gakuin University
5-10-1 Fuchinobe
Sagamihara, Kanagawa  229-8558
Japan

Phone: +81 42 759 6329
Fax:   +49 42 759 6495
EMail: duerst@it.aoyama.ac.jp
URI:   http://www.sw.it.aoyama.ac.jp/D%C3%BCrst/
Note: Please write "Duerst" with u-umlaut wherever possible, for example as "D&#252;rst" in XML and HTML. 🙈
Arnt Gulbrandsen
Oryx Mail Systems GmbH
Schweppermannstr. 8
81671 Munich
Germany

Fax:   +49 89 4502 9758
EMail: arnt@oryx.com
URI:   http://www.oryx.com/arnt/

Copyright (C) The IETF Trust (2007).

This doc is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🔒

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement 🫶

Funding for the RFC Editor function is currently provided by the Internet Society. 🌈

📦 where this page comes from