RFC 7564

PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols

📅 published May 2015, from ietf

PRECIS Framework 🦋 (RFC 7564, Part 1 of 9)

okay bestie, so this is the big one about the PRECIS Framework 💅 It's about how apps should prep, enforce, and compare internationalized strings (aka any text with Unicode characters in it 🌸). Let's get into the header stuff first 🎀

Quick facts: this is RFC 7564, published in May 2015 ✨. It's a Standards Track doc, which means it's an official Internet standard, not just a vibe 👑. The authors are P. Saint-Andre from &yet and M. Blanchet from Viagenie 🫶. It's ISSN 2070-1721 (the number for the series, not the doc itself 🤓). It also obsoletes RFC 3454, which means this one replaces the old Stringprep framework 🍓

Abstract ✨

okay so the main idea is this: apps that use Unicode characters in their protocol strings need to handle those strings properly 💖 That means two things. First, they have to enforce the internationalization rules for strings that go into different slots, like addresses and identifiers 🧚 Second, they need to do comparisons that actually work, like when you're checking someone's login or what they're allowed to access 🔐

This doc defines a framework called PRECIS (short for "Preparation, Enforcement, and Comparison of Internationalized Strings") 🦋 The best part is that it depends on the properties of Unicode characters instead of hardcoded lists, so it stays agile across different Unicode versions. Literally so iconic 😭

Compared to the old framework, Stringprep (RFC 3454), PRECIS is a way more sustainable approach to handling these strings. And yes, as I said, this document obsoletes RFC 3454 💅

Status of This Memo ✨

This is an Internet Standards Track document 🌈

It's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 👑 If you want more info on how Internet Standards work, check out Section 2 of RFC 5741 📚

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc7564 💜

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. ✨

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date of publication 📜 Please read these carefully, because they describe your rights and restrictions for this document, and that's a big deal 🙈

Also, any Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come without warranty, as described in the Simplified BSD License 💀

⚠️ bestie, the rule above says Code Components "must include" the Simplified BSD License text. Don't skip it when you copy code out of here 🫶

Table of Contents 🧚

   1. Introduction ....................................................4
   2. Terminology .....................................................7
   3. Preparation, Enforcement, and Comparison ........................7
   4. String Classes ..................................................8
      4.1. Overview ...................................................8
      4.2. IdentifierClass ............................................9
           4.2.1. Valid ...............................................9
           4.2.2. Contextual Rule Required ...........................10
           4.2.3. Disallowed .........................................10
           4.2.4. Unassigned .........................................11
           4.2.5. Examples ...........................................11
      4.3. FreeformClass .............................................11
           4.3.1. Valid ..............................................11
           4.3.2. Contextual Rule Required ...........................12
           4.3.3. Disallowed .........................................12
           4.3.4. Unassigned .........................................12
           4.3.5. Examples ...........................................12
   5. Profiles .......................................................13
      5.1. Profiles Must Not Be Multiplied beyond Necessity ..........13
      5.2. Rules .....................................................14
           5.2.1. Width Mapping Rule .................................14
           5.2.2. Additional Mapping Rule ............................14
           5.2.3. Case Mapping Rule ..................................14
           5.2.4. Normalization Rule .................................15
           5.2.5. Directionality Rule ................................15
      5.3. A Note about Spaces .......................................16
   6. Applications ...................................................17
      6.1. How to Use PRECIS in Applications .........................17
      6.2. Further Excluded Characters ...............................18
      6.3. Building Application-Layer Constructs .....................18
   7. Order of Operations ............................................19
   8. Code Point Properties ..........................................20
   9. Category Definitions Used to Calculate Derived Property ........22
      9.1. LetterDigits (A) ..........................................23
      9.2. Unstable (B) ..............................................23
      9.3. IgnorableProperties (C) ...................................23
      9.4. IgnorableBlocks (D) .......................................23
      9.5. LDH (E) ...................................................23
      9.6. Exceptions (F) ............................................23
      9.7. BackwardCompatible (G) ....................................23
      9.8. JoinControl (H) ...........................................24
      9.9. OldHangulJamo (I) .........................................24
      9.10. Unassigned (J) ...........................................24
      9.11. ASCII7 (K) ...............................................24
      9.12. Controls (L) .............................................24
      9.13. PrecisIgnorableProperties (M) ............................24
      9.14. Spaces (N) ...............................................25
      9.15. Symbols (O) ..............................................25
      9.16. Punctuation (P) ..........................................25
      9.17. HasCompat (Q) ............................................25
      9.18. OtherLetterDigits (R) ....................................25
   10. Guidelines for Designated Experts .............................26
   11. IANA Considerations ...........................................27
      11.1. PRECIS Derived Property Value Registry ...................27
      11.2. PRECIS Base Classes Registry .............................27
      11.3. PRECIS Profiles Registry .................................28
   12. Security Considerations .......................................29
      12.1. General Issues ...........................................29
      12.2. Use of the IdentifierClass ...............................30
      12.3. Use of the FreeformClass .................................30
      12.4. Local Character Set Issues ...............................31
      12.5. Visually Similar Characters ..............................31
      12.6. Security of Passwords ....................................33
   13. Interoperability Considerations ...............................34
      13.1. Encoding .................................................34
      13.2. Character Sets ...........................................34
      13.3. Unicode Versions .........................................34
      13.4. Potential Changes to Handling of Certain Unicode
            Code Points ..............................................34
   14. References ....................................................35
      14.1. Normative References .....................................35
      14.2. Informative References ...................................36
   Acknowledgements ..................................................40
   Authors' Addresses ................................................40

🎀 that's the front matter, bestie! The actual Introduction and the rest of the doc are coming up in the next parts 💖

1. Introduction ✨

Okay bestie, so application protocols that use Unicode characters [Unicode] in their protocol strings have to handle those strings properly 💅. That means enforcing the internationalization rules for strings that go into different protocol slots (like addresses and identifiers) and doing valid comparisons (like for authentication or authorization) 🔐. This document defines a framework that lets application protocols do the preparation, enforcement, and comparison of internationalized strings ("PRECIS"). It depends on the properties of Unicode characters, so it stays agile across Unicode versions 🦋.

As the PRECIS problem statement [RFC6885] explains, lots of IETF protocols used the Stringprep framework [RFC3454] to prep, enforce, and compare protocol strings with Unicode characters, especially ones outside the ASCII range [RFC20] 🌸. Stringprep was developed during the work on the original internationalized domain names (IDNs) technology, which is called "IDNA2003" here [RFC3490]. Nameprep [RFC3491] was the Stringprep profile for IDNs. Stringprep was designed as a general framework so other application protocols could make their own profiles, and a bunch of them did 💖.

After [RFC3454] came out in 2002, several big issues showed up with Stringprep in the IDN case. The IAB's recommendations regarding IDNs [RFC4690] document them (most significantly, Stringprep was tied to Unicode version 3.2) 🙈. So the newer IDNA specs, called "IDNA2008" ([RFC5890], [RFC5891], [RFC5892], [RFC5893], [RFC5894]), no longer use Stringprep and Nameprep. This move away from Stringprep for IDNs made the other "customers" of Stringprep start looking for new approaches to preparing, enforcing, and comparing internationalized strings, as described in [RFC6885] 👑.

This document defines a framework for a post-Stringprep approach to preparing, enforcing, and comparing internationalized strings in application protocols. It's built on these principles:

  1. Define a small set of string classes that specify the Unicode characters (aka specific "code points") that fit common application protocol constructs.
  2. Define each PRECIS string class in terms of Unicode code points and their properties, so an algorithm can work out whether each code point or character category is (a) valid, (b) allowed in certain contexts, (c) disallowed, or (d) unassigned 🧚.
  3. Use an "inclusion model", so a string class only has code points that are explicitly allowed. Anything not explicitly allowed is forbidden 🚫.
  4. Let application protocols define profiles of the PRECIS string classes when needed (covering things like width mapping, case mapping, Unicode normalization, and directionality). But strongly discourage piling up profiles past what's necessary, so we don't break the "Principle of Least Astonishment" 🙃.

It's expected that this framework will bring these benefits:

  • Application protocols will be agile with regard to Unicode versions ✨.
  • Implementers will be able to share code point tables and software code across application protocols, most likely through software libraries 📚.
  • End users will get more accurate expectations about which characters are acceptable in various contexts. With a more uniform set of string classes, copy/paste between software implementing different application protocols should also be more predictable and coherent 📋.

Whereas the string classes define the "baseline" code points for a range of applications, profiling lets application protocols use the string classes in ways that suit common constructs. Examples are usernames [PRECIS-Users-Pwds], opaque strings like passwords [PRECIS-Users-Pwds], and nicknames [PRECIS-Nickname] 🎀. Profiles are responsible for defining how right-to-left characters get handled, plus the various mapping operations also discussed for IDNs in [RFC5895]. Those include case preservation or lowercasing, Unicode normalization, mapping certain characters to other characters or to nothing, and mapping fullwidth and halfwidth characters 🌈.

When an application applies a profile of a PRECIS string class, it transforms an input string (which might or might not be conforming) into an output string that definitively conforms to the profile 💯. This document focuses on two objectives:

a. Enforcing all the rules of a profile on a single output string (e.g., to decide whether a string can go into a protocol slot, be communicated to another entity within a protocol, be stored in a retrieval system, and so on).

b. Comparing two output strings to see if they're equivalent, usually with octet-for-octet matching to test for "bit-string identity". This is how you make an access decision for authentication or authorization, as further described in [RFC6943] 🔑.

Since profiles can be defined, there's a real chance of a profile explosion, which could cancel out the benefits of common code and go against user expectations. See Section 5 for the deep dive on this important topic 👀.

It's also super important for protocol designers and application developers to understand that turning an input string into an output string is rarely reversible 🙅‍♀️. As a simple example, case mapping would turn the input string "StPeter" into "stpeter", and the info about the capitalization of the first and third characters would be lost forever 😭. Other kinds of mapping and normalization have similar issues.

Heads up: once the mapping is done, the original capitalization or form is gone. Don't assume you can get it back later.

Even though this framework is similar to IDNA2008, and it includes by reference some of the character categories defined in [RFC5892], it defines extra character categories so it meets the needs of common application protocols other than DNS 💅.

The character categories and calculation rules defined under Sections 8 and 9 are normative and apply to all Unicode code points. The code point table that comes out of applying those categories and rules to the latest version of Unicode can be found in an IANA registry 📖.

2. Terminology

Lots of the important terms in this document are defined in [RFC5890], [RFC6365], [RFC6885], and [Unicode]. The terms "left-to-right" (LTR) and "right-to-left" (RTL) are defined in Unicode Standard Annex #9 [UAX9] ➡️⬅️.

As of the date of writing, the version of Unicode published by the Unicode Consortium is 7.0 [Unicode7.0]. But PRECIS isn't tied to one specific Unicode version, and the latest version of Unicode is always available [Unicode] 🌸.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🫶.

3. Preparation, Enforcement, and Comparison

This document distinguishes between three different actions an entity can take with regard to a string 🧐:

  • Enforcement means applying all of the rules for a particular string class (or a profile of it) to an individual string, to figure out whether the string can be used in a given protocol slot 🚪.
  • Comparison means applying all of the rules for a particular string class (or a profile of it) to two separate strings, to figure out whether they're equivalent ⚖️.
  • Preparation only means making sure the characters in an individual string are allowed by the underlying PRECIS string class 🧹.

In most cases, authoritative entities like servers handle enforcement, while subsidiary entities like clients only handle preparation 👑. The reason for this split is that clients might not have the device memory and processing power to enforce all the rules for internationalized strings (like width mapping and Unicode normalization). They can, though, easily limit the set of characters they offer to an end user 📱. Servers are assumed to have more capacity to enforce the rules, and they act as the authority on allowable strings in protocol slots like addresses and endpoint identifiers 🏛️. Plus, a client can't necessarily be trusted to generate those strings properly, especially in security-sensitive contexts like authentication and authorization 🥺.

4. String Classes ✨

4.1. Overview 💖

okay so starting in 2010, a bunch of "customers" of Stringprep began saying they needed a post-Stringprep way to prep and compare internationalized strings that aren't IDNs 🌸 the community looked at the existing Stringprep profiles and weighed two options: a smaller set of Unicode characters that keeps user confusion down by skipping look-alike characters (so it's "safe"), or a much bigger set that lets people get super creative (so it's "expressive") ✨ in the end they decided most existing uses could be covered by just two string classes:

  • IdentifierClass: a sequence of letters, numbers, and some symbols used to identify or address a network entity like a user account, a venue (e.g., a chatroom), an information source (e.g., a data feed), or a collection of data (e.g., a file). the vibe is keeping user confusion low across a wide range of application protocols, so safety wins over expressiveness for this class 🔒
  • FreeformClass: a sequence of letters, numbers, symbols, spaces, and other characters used for free-form strings, including passwords and display stuff like human-friendly nicknames for devices or chatroom participants. the vibe here is allowing nearly any Unicode character, so expressiveness wins over safety 🎀
Note well that protocol designers, application developers, service providers, and end users might not understand or be able to enter all of the characters that can be included in the FreeformClass. see Section 12.3 for details ⚠️

future specs might define more PRECIS string classes, like one that sits somewhere between the IdentifierClass and the FreeformClass 🧚 it's not clear yet how useful that would be 🤷 but since application developers can define profiles of PRECIS string classes, a protocol that needs something in between could define a restricted profile of the FreeformClass if needed 💅

the next subsections go deeper on the IdentifierClass and FreeformClass, using the dimensions from Section 5 of [RFC6885]. each string class is defined by these behavioral rules:

  • Valid: defines which code points are treated as valid for the string.
  • Contextual Rule Required: defines which code points are treated as allowed only if the requirements of a contextual rule are met (i.e., either CONTEXTJ or CONTEXTO).
  • Disallowed: defines which code points need to be excluded from the string.
  • Unassigned: defines application behavior in the presence of code points that are unknown (i.e., not yet designated) for the version of Unicode used by the application.

this document defines the valid, contextual rule required, disallowed, and unassigned rules for the IdentifierClass and FreeformClass 🫶 as described under Section 5, profiles of these string classes are responsible for defining the width mapping, additional mappings, case mapping, normalization, and directionality rules.

4.2. IdentifierClass 🦋

most application technologies need strings that can be used to refer to, include, or communicate protocol strings, like usernames, filenames, data feed identifiers, and chatroom names. we group those into a class called "IdentifierClass" with the following features ✨

4.2.1. Valid

  • Code points traditionally used as letters and numbers in writing systems, i.e., the LetterDigits ("A") category first defined in [RFC5892] and listed here under Section 9.1.
  • Code points in the range U+0021 through U+007E, i.e., the (printable) ASCII7 ("K") category defined under Section 9.11. these are "grandfathered" into PRECIS, so they're valid even if they would otherwise be disallowed by the property-based rules in the next section 🌈
Note: Although the PRECIS IdentifierClass reuses the LetterDigits category from IDNA2008, the range of characters allowed in the IdentifierClass is wider than the range of characters allowed in IDNA2008. The main reason is that IDNA2008 applies the Unstable category before the LetterDigits category, thus disallowing uppercase characters, whereas the IdentifierClass does not apply the Unstable category 💀

4.2.2. Contextual Rule Required

  • A number of characters from the Exceptions ("F") category defined under Section 9.6 (see Section 9.6 for a full list).
  • Joining characters, i.e., the JoinControl ("H") category defined under Section 9.8.

4.2.3. Disallowed

  • Old Hangul Jamo characters, i.e., the OldHangulJamo ("I") category defined under Section 9.9.
  • Control characters, i.e., the Controls ("L") category defined under Section 9.12.
  • Ignorable characters, i.e., the PrecisIgnorableProperties ("M") category defined under Section 9.13.
  • Space characters, i.e., the Spaces ("N") category defined under Section 9.14.
  • Symbol characters, i.e., the Symbols ("O") category defined under Section 9.15.
  • Punctuation characters, i.e., the Punctuation ("P") category defined under Section 9.16.
  • Any character that has a compatibility equivalent, i.e., the HasCompat ("Q") category defined under Section 9.17. these code points are disallowed even if they would otherwise be valid according to the property-based rules in the previous section 🙅‍♀️
  • Letters and digits other than the "traditional" letters and digits allowed in IDNs, i.e., the OtherLetterDigits ("R") category defined under Section 9.18.

4.2.4. Unassigned

any code points that aren't designated yet in the Unicode character set are considered unassigned for purposes of the IdentifierClass, and those code points are to be treated as disallowed 🙃 see Section 9.10.

4.2.5. Examples

as described in the Introduction to this document, the string classes don't handle every issue around string preparation and comparison (like case mapping). those get handled at the level of profiles instead 🧁 examples of IdentifierClass profiles can be found in [PRECIS-Users-Pwds] (the UsernameCaseMapped and UsernameCasePreserved profiles).

4.3. FreeformClass 🎀

some application technologies need strings that can be used in a free-form way, e.g., as a password in an authentication exchange (see [PRECIS-Users-Pwds]) or a nickname in a chatroom (see [PRECIS-Nickname]). we group those into a class called "FreeformClass" with the following features 💜

Security Warning: As mentioned, the FreeformClass prioritizes expressiveness over safety; Section 12.3 describes some of the security hazards involved with using or profiling the FreeformClass. 🚨
Security Warning: Consult Section 12.6 for relevant security considerations when strings conforming to the FreeformClass, or a profile thereof, are used as passwords. 🔐

4.3.1. Valid

  • Traditional letters and numbers, i.e., the LetterDigits ("A") category first defined in [RFC5892] and listed here under Section 9.1.
  • Letters and digits other than the "traditional" letters and digits allowed in IDNs, i.e., the OtherLetterDigits ("R") category defined under Section 9.18.
  • Code points in the range U+0021 through U+007E, i.e., the (printable) ASCII7 ("K") category defined under Section 9.11.
  • Any character that has a compatibility equivalent, i.e., the HasCompat ("Q") category defined under Section 9.17.
  • Space characters, i.e., the Spaces ("N") category defined under Section 9.14.
  • Symbol characters, i.e., the Symbols ("O") category defined under Section 9.15.
  • Punctuation characters, i.e., the Punctuation ("P") category defined under Section 9.16.

4.3.2. Contextual Rule Required

  • A number of characters from the Exceptions ("F") category defined under Section 9.6 (see Section 9.6 for a full list).
  • Joining characters, i.e., the JoinControl ("H") category defined under Section 9.8.

4.3.3. Disallowed

  • Old Hangul Jamo characters, i.e., the OldHangulJamo ("I") category defined under Section 9.9.
  • Control characters, i.e., the Controls ("L") category defined under Section 9.12.
  • Ignorable characters, i.e., the PrecisIgnorableProperties ("M") category defined under Section 9.13.

4.3.4. Unassigned

any code points that aren't designated yet in the Unicode character set are considered unassigned for purposes of the FreeformClass, and those code points are to be treated as disallowed 🧊

4.3.5. Examples

as described in the Introduction to this document, the string classes don't handle every issue around string preparation and comparison (like case mapping). those get handled at the level of profiles instead 🌸 examples of FreeformClass profiles can be found in [PRECIS-Users-Pwds] (the OpaqueString profile) and [PRECIS-Nickname] (the Nickname profile).

5. Profiles 🌈

this framework document defines the valid, contextual-rule-required, disallowed, and unassigned rules for the IdentifierClass and the FreeformClass 🍓 a profile of a PRECIS string class MUST define the width mapping, additional mappings (if any), case mapping, normalization, and directionality rules. a profile MAY also restrict the allowable characters above and beyond the definition of the relevant PRECIS string class (but MUST NOT add as valid any code points that are disallowed by the relevant PRECIS string class). these matters are discussed in the following subsections 👑

profiles of the PRECIS string classes are registered with the IANA as described under Section 11.3 📝 profile names follow this convention: they take the form "Profilename of BaseClass", where the "Profilename" string is a differentiator and "BaseClass" is the name of the PRECIS string class being profiled. for example, the profile of the FreeformClass used for opaque strings such as passwords is the OpaqueString profile [PRECIS-Users-Pwds] 🦋

5.1. Profiles Must Not Be Multiplied beyond Necessity 🚫

okay so the risk of profile proliferation is seriously a big deal bestie 😭 if there are too many profiles, the same string acts different in every app, and that breaks the "Principle of Least Astonishment" from user interface design. nobody wants that vibe 💀

and honestly? we already have way too many profiles 🙈 ideally we'd only have two or three. the problem is that loads of application protocols already have their own quirks with strings. domain names, email addresses, instant messaging addresses, chatroom nicknames, filenames, authentication identifiers, passwords, and other strings are already out there in the wild and still need to work in existing protocols like DNS, SMTP, the Extensible Messaging and Presence Protocol (XMPP), Internet Relay Chat (IRC), NFS, the Internet Small Computer System Interface (iSCSI), the Extensible Authentication Protocol (EAP), and the Simple Authentication and Security Layer (SASL), among others 🌈

still, profiles must not be multiplied beyond necessity 💅

to help stop the profile glow-up from getting out of hand, this document recommends sensible defaults for the options profile creators get (like width mapping and Unicode normalization) ✨ and the designated expert guidelines under Section 10 are there to push a high level of due diligence whenever someone proposes a new profile 🎀

5.2. Rules 📜

5.2.1. Width Mapping Rule 📏

the width mapping rule of a profile says whether width mapping happens to the characters in a string, and how that mapping works 🦋 usually it means mapping fullwidth and halfwidth characters (code points with a Decomposition Type of Wide or Narrow) to their decomposition mappings. for example, FULLWIDTH DIGIT ZERO (U+FF10) gets mapped to DIGIT ZERO (U+0030) 🍓

the normalization form a profile uses changes whether width mapping is even needed 🧚 because width mapping is part of compatibility decomposition, a profile using normalization form KD (NFKD) or normalization form KC (NFKC) doesn't have to specify width mapping. but if Unicode normalization form C (NFC) is used (which is the recommended one), then the profile does need to say whether to apply width mapping. in that case, width mapping is in general RECOMMENDED, because leaving fullwidth and halfwidth characters unmapped to their compatibility variants would break the "Principle of Least Astonishment" 😭🔥 if you want the deets on width in East Asian scripts within Unicode, check out Unicode Standard Annex #11 [UAX11] 💜

5.2.2. Additional Mapping Rule 🌸

the additional mapping rule of a profile says whether extra mappings happen to the characters in a string. examples include:

  • Mapping of delimiter characters (such as '@', ':', '/', '+', and '-')
  • Mapping of special characters (e.g., non-ASCII space characters to ASCII space or control characters to nothing).

the PRECIS mappings document [PRECIS-Mappings] goes deeper into these mappings if you're curious 🫶

5.2.3. Case Mapping Rule 🔤

the case mapping rule of a profile says whether case mapping happens (instead of case preservation) to the characters in a string, and how that mapping gets applied. like, mapping uppercase and titlecase characters to their lowercase equivalents 🌷

if you want case mapping instead of case preservation, it is RECOMMENDED to use Unicode Default Case Folding as defined in the Unicode Standard [Unicode] (at the time of writing, the algorithm is specified in Chapter 3 of [Unicode7.0]) ✨

Note: Unicode Default Case Folding isn't built to handle localization issues like the so-called "dotless i" in several Turkic languages 🙈 the PRECIS mappings document [PRECIS-Mappings] covers these issues in way more detail, and it defines a "local case mapping" method that handles some locale-dependent and context-dependent mappings. iykyk 💖

to get max entropy and minimize the chance of false positives, it is NOT RECOMMENDED for application protocols to map uppercase and titlecase code points to their lowercase equivalents when strings conforming to the FreeformClass (or a profile of it) are used in passwords 🔐 instead, it is RECOMMENDED to preserve the case of every code point in those strings and then do a case-sensitive comparison. see also the related discussion in Section 12.6 and in [PRECIS-Users-Pwds] 💅

5.2.4. Normalization Rule 🧩

the normalization rule of a profile says which Unicode normalization form (D, KD, C, or KC) gets applied. for background info, check out Unicode Standard Annex #15 [UAX15] 📚

following [RFC5198], normalization form C (NFC) is RECOMMENDED 💜

5.2.5. Directionality Rule ↔️

the directionality rule of a profile says how to handle strings that contain what people often call "right-to-left" (RTL) characters (see Unicode Standard Annex #9 [UAX9]) 🌈 RTL characters come from scripts that are normally written right to left, and Unicode considers them to have right-to-left directionality themselves. some strings with RTL characters also contain "left-to-right" (LTR) characters, like numerals, plus characters without directional properties. that's why these strings are called "bidirectional strings" 🦋

presenting bidirectional strings in different layout systems can get messy 😭 (like a UI set up mainly for an RTL script versus one set up mainly for an LTR script). the results can be predictable to people who know the display rules, but counter-intuitive to casual users. the same bidirectional string (in PRECIS terms) might not show up the same way for users of those different layout systems, even though it's consistent inside any single layout system. some applications might find these differences problematic, so the designers might want to restrict bidirectional strings with a directionality rule. other applications won't mind, especially for more "free-form" strings, so no directionality rule is needed there 🎀

the PRECIS framework doesn't directly deal with bidirectional strings across every string class and profile, and it doesn't define any new directionality rules. that's because there's no widely accepted, widely implemented solution for safely displaying arbitrary bidirectional strings beyond the Unicode bidirectional algorithm [UAX9] yet ✨ the "Bidi Rule" in the IDNA2008 specification for right-to-left scripts [RFC5893] does define rules for domain name labels and similar identifiers, but those rules are pretty restrictive and might not fit every bidirectional string 💀

if you're writing a PRECIS profile and think you need your own directionality rule, this document strongly suggests you think again 🙈 authors of a profile almost always end up being wrong about needing a new one, even when they've done tons of careful research into displaying bidirectional strings. instead, consider the "Bidi Rule" [RFC5893] (for profiles based on the IdentifierClass) or follow the Unicode bidirectional algorithm [UAX9] (for profiles based on the FreeformClass, or in situations where the IdentifierClass isn't a good fit) 👑

5.3. A Note about Spaces 🌟

for the IdentifierClass, the PRECIS Working Group agreed that spaces are problematic for a bunch of reasons, including the following:

  • Many Unicode characters are confusable with ASCII space.
  • Even if non-ASCII space characters get mapped to ASCII space (U+0020), space characters often don't render in user interfaces. so a human user might think a string with spaces is the same as that string without spaces 😭
  • In some locales, some devices are known to generate a character other than ASCII space (such as ZERO WIDTH JOINER, U+200D) when a user hits the space bar on a keyboard 🤯

disallowing space characters in the IdentifierClass might end up discouraging their use in identifiers created in newer application protocols. given how hard it is to handle spaces properly (especially non-ASCII spaces) in identifiers and other protocol strings, the PRECIS Working Group considered this to be a feature, not a bug 💅

that said, the FreeformClass does allow spaces. that means application protocols can define profiles of the FreeformClass that are more flexible than any profile of the IdentifierClass 🌸 and as Section 6.3 explains, application protocols can also define application-layer constructs that contain spaces 🦋

6. Applications 🎀

6.1. How to Use PRECIS in Applications 💖

okay so PRECIS was designed with applications in mind, but internationalization doesn't magically become easy just because you're using PRECIS 😭 application developers still need to think about how they'll use the PRECIS string classes, or profiles of them, in their apps. this section gives some guidelines for application developers (and for expert reviewers of application protocol specifications) 🧚

  • Don't define your own profile unless absolutely necessary (see Section 5.1). existing profiles were designed for wide reuse, so it's super likely one of them will meet your needs. this is especially true given the ability to specify further excluded characters (Section 6.2) and to build application-layer constructs (see Section 6.3) 💅
  • Do specify:
    • Exactly which entities are responsible for preparation, enforcement, and comparison of internationalized strings (e.g., servers or clients).
    • Exactly when those entities need to complete their tasks (e.g., a server might need to enforce the rules of a profile before allowing a client to gain network access).
    • Exactly which protocol slots need to be checked against which profiles (e.g., checking the address of a message's intended recipient against the UsernameCaseMapped profile [PRECIS-Users-Pwds] of the IdentifierClass, or checking the password of a user against the OpaqueString profile [PRECIS-Users-Pwds] of the FreeformClass).

for definitions of these matters for several applications, see [PRECIS-Users-Pwds] and [XMPP-Addr-Format] 🫶

6.2. Further Excluded Characters ✨

Okay so, an application protocol that uses a profile MAY specify particular code points that aren't allowed in its relevant slots, on top of whatever the string class or profile already excludes. Think of it as an extra bouncer at the door 💅

Specifically, an application protocol MAY do either of these:

  1. Exclude specific code points that the relevant string class allows.
  2. Exclude characters that match certain Unicode properties (e.g., math symbols) even though the relevant PRECIS string class includes them.

Because of these exclusions, code points that are valid for the PRECIS string class or profile end up being disallowed for that particular protocol slot. Plot twist, but it makes sense 🙈

Usually these exclusions exist for backward compatibility with legacy formats inside an application protocol. They're defined at the application protocol level rather than the profile level so we don't end up with a pile of profiles nobody needs. That's the whole "prevent multiplication of profiles beyond necessity" thing (see Section 5.1) 🫶

6.3. Building Application-Layer Constructs 🧚

Sometimes an application-layer construct doesn't map neatly onto one of the base string classes or a profile of one. A good example is the "simple user name" construct in the Simple Authentication and Security Layer (SASL) [RFC4422]. Depending on the deployment, a simple user name could be a user's full name (like a personal name, then a space, then a family name). That can't be defined as an instance of the IdentifierClass or a profile of it, because space characters aren't allowed in the IdentifierClass. 😭

Instead, it can be defined as a space-separated sequence of IdentifierClass instances. Here's the ABNF [RFC5234] from [PRECIS-Users-Pwds]:

   username   = userpart *(1*SP userpart)
   userpart   = 1*(idbyte)
                ;
                ; an "idbyte" is a byte used to represent a
                ; UTF-8 encoded Unicode code point that can be
                ; contained in a string that conforms to the
                ; PRECIS "IdentifierClass"
                ;

You can use similar techniques to define lots of other application-layer constructs, like "user@domain" or "/path/to/file" 🦋

7. Order of Operations 💖

To make sure comparisons work properly, the rules for a particular string class or profile MUST be applied in this order:

  1. Width Mapping Rule
  2. Additional Mapping Rule
  3. Case Mapping Rule
  4. Normalization Rule
  5. Directionality Rule
  6. Behavioral rules for determining whether a code point is valid, allowed under a contextual rule, disallowed, or unassigned

Like we already covered, the width mapping, additional mapping, case mapping, normalization, and directionality rules are specified for each profile. The behavioral rules are specified for each string class. Some of the logic behind this order is explained under Section 5.2.1 (see also the PRECIS mappings document [PRECIS-Mappings]) 🎀

8. Code Point Properties 🌈

To actually implement the string classes from earlier, this document does two things:

  1. Reviews and classifies the collections of code points in the Unicode character set by looking at various code point properties.
  2. Defines an algorithm for working out a derived property value, which can change depending on which string class the relevant application protocol uses.

This document isn't meant to spell out exactly how derived property values get applied in protocol strings. That's the job of the protocol specification that uses or profiles a PRECIS string class from this document. Here's how to read the property values 👑

  • PROTOCOL VALID: Code points that are allowed in any PRECIS string class (right now, IdentifierClass and FreeformClass). The abbreviated term "PVALID" is used for this value for the rest of this document.
  • SPECIFIC CLASS PROTOCOL VALID: Code points that are allowed in specific string classes. For the rest of this document, the abbreviated term *_PVAL is used, where * = (ID | FREE), so it's either FREE_PVAL or ID_PVAL. Honestly, the derived property ID_PVAL isn't used in this specification, since every ID_PVAL code point is PVALID already 💀
  • CONTEXTUAL RULE REQUIRED: Some characters are invisible in certain contexts or problematic in others, so they can't be used in labels unless specific other characters or properties are present. Just like in IDNA2008, there are two subdivisions of CONTEXTUAL RULE REQUIRED: one for Join_controls (called "CONTEXTJ") and one for other characters (called "CONTEXTO"). A character with the derived property value CONTEXTJ or CONTEXTO MUST NOT be used unless an appropriate rule has been established and the character's context is consistent with that rule. The most notable CONTEXTUAL RULE REQUIRED characters are the Join Control characters U+200D ZERO WIDTH JOINER and U+200C ZERO WIDTH NON-JOINER, which have a derived property value of CONTEXTJ. See Appendix A of [RFC5892] for more info 🔥
  • DISALLOWED: Code points that aren't permitted in any PRECIS string class. Big no 🚫
  • SPECIFIC CLASS DISALLOWED: Code points that shouldn't be included in one of the string classes, but that might be fine in others. For the rest of this document, the abbreviated term *_DIS is used, where * = (ID | FREE), so it's either FREE_DIS or ID_DIS. Also, the derived property FREE_DIS isn't used in this specification, since every FREE_DIS code point is DISALLOWED anyway.
  • UNASSIGNED: Code points that aren't designated (meaning they're unassigned) in the Unicode Standard.

Now for the algorithm that calculates the derived property value. Implementations MUST NOT change the order of operations inside it, because that would cause inconsistent results between implementations. Don't touch the order, bestie 🙅‍♀️

   If .cp. .in. Exceptions Then Exceptions(cp);
   Else If .cp. .in. BackwardCompatible Then BackwardCompatible(cp);
   Else If .cp. .in. Unassigned Then UNASSIGNED;
   Else If .cp. .in. ASCII7 Then PVALID;
   Else If .cp. .in. JoinControl Then CONTEXTJ;
   Else If .cp. .in. OldHangulJamo Then DISALLOWED;
   Else If .cp. .in. PrecisIgnorableProperties Then DISALLOWED;
   Else If .cp. .in. Controls Then DISALLOWED;
   Else If .cp. .in. HasCompat Then ID_DIS or FREE_PVAL;
   Else If .cp. .in. LetterDigits Then PVALID;
   Else If .cp. .in. OtherLetterDigits Then ID_DIS or FREE_PVAL;
   Else If .cp. .in. Spaces Then ID_DIS or FREE_PVAL;
   Else If .cp. .in. Symbols Then ID_DIS or FREE_PVAL;
   Else If .cp. .in. Punctuation Then ID_DIS or FREE_PVAL;
   Else DISALLOWED;

The value you get can depend on the string class. For example, if an identifier in an application protocol profiles the PRECIS IdentifierClass, then a space character like U+0020 gets assigned to ID_DIS. If the identifier profiles the PRECIS FreeformClass instead, that same character gets assigned to FREE_PVAL. To keep things short, "FREE_PVAL" is used in this document instead of the longer "ID_DIS or FREE_PVAL". Again, the derived properties ID_PVAL and FREE_DIS aren't used in this specification, since every ID_PVAL code point is PVALID and every FREE_DIS code point is DISALLOWED ✨

Using a rule's name (like "Exceptions") means the set of code points that rule defines. Using the same name as a function call (like "Exceptions(cp)") means the value that code point has in the Exceptions table 🧠

The mechanisms here let you work out the property value for future versions of Unicode, including characters added after Unicode 5.2 or 7.0 depending on the category. Some categories mentioned here are just pointers to IDNA2008, so they were defined back when Unicode 5.2 was current. Changes to Unicode properties that don't affect the outcome of this process also don't affect this framework. For example, a character can have its Unicode General_Category value change from So to Sm, or from Lo to Ll (at the time of writing, see Chapter 4 of [Unicode7.0]), and the algorithm results stay the same. Even if changes did shift results, the BackwardCompatible list (Section 9.7) can be adjusted to keep the results stable 💜

9. Category Definitions Used to Calculate Derived Property 🍓

The derived property gets its value through a two-step procedure:

  1. Characters are placed into one or more character categories, either (1) based on core properties defined by the Unicode Standard or (2) by treating the code point as an exception and handling it based on its code point value. These categories aren't mutually exclusive.
  2. Set operations are used with these categories to determine the values for a property specific to a given string class. These operations are specified under Section 8.

Note: Unicode property names and property value names might have short abbreviations, such as "gc" for the General_Category property and "Ll" for the Lowercase_Letter property value of the gc property.

When listing the character categories below, the operation that returns the value of a particular Unicode character property for a code point uses the property's formal name (from the Unicode PropertyAliases.txt file [PropertyAliases]), followed by "(cp)" for "code point". For example, the value of the General_Category property for a code point is written General_Category(cp).

The first ten categories (A-J) were previously defined for IDNA2008 and are referenced from [RFC5892] to help people understand how PRECIS handles various characters. Some of these categories get reused in PRECIS and some don't, but the lettering is kept so categories don't overlap, and so that implementing both IDNA2008 and PRECIS in one software application is easier. The next eight categories (K-R) are specific to PRECIS 🌸

9.1. LetterDigits (A)

This category is defined in Section 2.1 of [RFC5892] and is included by reference for use in PRECIS. 🔤

9.2. Unstable (B)

This category is defined in Section 2.2 of [RFC5892]. However, it is not used in PRECIS. 🙅‍♀️

9.3. IgnorableProperties (C)

This category is defined in Section 2.3 of [RFC5892]. However, it is not used in PRECIS.

Note: See the PrecisIgnorableProperties ("M") category below for a more inclusive category used in PRECIS identifiers. 👀

9.4. IgnorableBlocks (D)

This category is defined in Section 2.4 of [RFC5892]. However, it is not used in PRECIS. 🫠

9.5. LDH (E)

This category is defined in Section 2.5 of [RFC5892]. However, it is not used in PRECIS.

Note: See the ASCII7 ("K") category below for a more inclusive category used in PRECIS identifiers. 🧃

9.6. Exceptions (F)

This category is defined in Section 2.6 of [RFC5892] and is included by reference for use in PRECIS. 🦄

9.7. BackwardCompatible (G) 🦋

okay so this category is defined in Section 2.7 of [RFC5892] and PRECIS just borrows it by reference 💅

Note: managing this category is handled through the processes in [RFC5892]. As of when this was written (and also when RFC 5892 came out), the category was literally the empty set 🫥 BUT it can totally change later, as RFC 5892 explains. no cap, it's a living list 🌈

9.8. JoinControl (H) 🌸

this one's also defined in Section 2.8 of [RFC5892] and PRECIS includes it by reference. bestie, we're just borrowing it again ✨

9.9. OldHangulJamo (I) 🧚

same energy here: defined in Section 2.9 of [RFC5892], and PRECIS pulls it in by reference 🫶

9.10. Unassigned (J) 🍓

defined in Section 2.10 of [RFC5892], and PRECIS includes it by reference. iconic consistency 👑

9.11. ASCII7 (K) 💜

this one is PRECIS-specific 🎀 It covers all the printable, non-space characters in the 7-bit ASCII range. The vibe is that the algorithm in Section 8 lets these characters skip any other rules that might get applied during PRECIS processing. Why? Because they're in SO much use that blocking them would be counter-productive 🙈 literally nobody wants to ban the letter A.

K: cp is in {0021..007E}

9.12. Controls (L) 🌈

PRECIS-specific again! This one is just all the control characters 💀

L: Control(cp) = True

9.13. PrecisIgnorableProperties (M) 🫧

PRECIS-specific ✨ This is the group for code points that PRECIS strongly discourages using in its string classes. Lowkey a red flag list 🚩

M: Default_Ignorable_Code_Point(cp) = True or
   Noncharacter_Code_Point(cp) = True

For the definition of Default_Ignorable_Code_Point, check the DerivedCoreProperties.txt file [DerivedCoreProperties] 📖

9.14. Spaces (N) 🍑

PRECIS-specific, grouping the code points that are space characters. the spacing crew 🫶

N: General_Category(cp) is in {Zs}

9.15. Symbols (O) 💎

PRECIS-specific too! This group collects the code points that are symbols ✨

O: General_Category(cp) is in {Sm, Sc, Sk, So}

9.16. Punctuation (P) 🎀

PRECIS-specific. This group holds the punctuation characters, periodt 💅

P: General_Category(cp) is in {Pc, Pd, Ps, Pe, Pi, Pf, Po}

9.17. HasCompat (Q) 🦋

PRECIS-specific. This group is for code points that have compatibility equivalents, as the Unicode Standard explains (at the time of writing, see Chapters 2 and 3 of [Unicode7.0]) 🌸

Q: toNFKC(cp) != cp

The toNFKC() operation gives you back the code point in normalization form KC. For more details, check Section 5 of Unicode Standard Annex #15 [UAX15] 🤓

9.18. OtherLetterDigits (R) 🧚

PRECIS-specific, and it groups the letters and digits that are NOT the "traditional" ones already in the LetterDigits (A) class (see Section 9.1). so basically the extras 💖

R: General_Category(cp) is in {Lt, Nl, No, Me}

10. Guidelines for Designated Experts 🫶

okay the big lesson here: internationalization is tricky, and protocol designers and app devs usually don't know the subtleties and tradeoffs until they get burned. so they really need guidance to make sensible choices 😭

Therefore:

  • Protocol designers are strongly encouraged to question the assumption that they need to define new profiles, because existing profiles are built for wide reuse (see Section 5 for further discussion) 💅
  • Those who still want to define new profiles are strongly encouraged to clearly explain a strong justification for it, and to publish a stable specification that gives all of the information described under Section 11.3 🌈
  • The designated experts for profile registration requests ought to ask all of the questions provided under Section 11.3, and encourage applicants to provide a stable specification documenting the profile (even though the registration policy for PRECIS profiles is Expert Review and a stable specification is not strictly required) 👑
  • Developers of applications that use PRECIS are strongly encouraged to apply the guidelines provided under Section 6, and to seek out the advice of the designated experts or other knowledgeable individuals while doing so 🧚
  • Everyone is strongly encouraged to help prevent the multiplication of profiles beyond necessity, as described under Section 5.1, and to use PRECIS in ways that minimize user confusion and insecure application behavior 🫧

Internationalization can be difficult and contentious 😤 so designated experts, profile registrants, and application developers are strongly encouraged to work together in good faith and mutual understanding to reach rough consensus on profile registration requests and on the use of PRECIS in particular applications. They're also encouraged to bring in extra expertise if that helps add perspective or resolves issues ✨

11. IANA Considerations 🎀

11.1. PRECIS Derived Property Value Registry 🌸

IANA created and now maintains the "PRECIS Derived Property Value" registry, which records the derived properties for Unicode versions released after (and including) version 7.0 🦋 The derived property value is calculated in cooperation with a designated expert [RFC5226], following the rules in Sections 8 and 9.

The IESG needs to be notified if backward-incompatible changes to the table of derived properties are discovered, or if other problems come up while creating the table of derived property values or during expert review 🚨 Changes to the rules in Sections 8 and 9 require IETF Review.

11.2. PRECIS Base Classes Registry 💜

IANA created the "PRECIS Base Classes" registry 🧚 Following [RFC5226], the registration policy is "RFC Required".

The registration template looks like this:

Base Class:  [the name of the PRECIS string class]

Description:  [a brief description of the PRECIS string class and its
   intended use, e.g., "A sequence of letters, numbers, and symbols
   that is used to identify or address a network entity."]

Specification:  [the RFC number]

The initial registrations are:

Base Class: FreeformClass.
Description: A sequence of letters, numbers, symbols, spaces, and
      other code points that is used for free-form strings.
Specification: Section 4.3 of RFC 7564.

Base Class: IdentifierClass.
Description: A sequence of letters, numbers, and symbols that is
      used to identify or address a network entity.
Specification: Section 4.2 of RFC 7564.

11.3. PRECIS Profiles Registry 🌈

IANA created the "PRECIS Profiles" registry to identify profiles that use the PRECIS string classes 🫶 Following [RFC5226], the registration policy is "Expert Review". This policy was picked to ease the registration burden while making sure "customers" of PRECIS get proper guidance on the sometimes complex and subtle internationalization issues tied to PRECIS string class profiles 🤓

The registration template looks like this:

Name:  [the name of the profile]

Base Class:  [which PRECIS string class is being profiled]

Applicability:  [the specific protocol elements to which this profile
   applies, e.g., "Localparts in XMPP addresses."]

Replaces:  [the Stringprep profile that this PRECIS profile replaces,
   if any]

Width Mapping Rule:  [the behavioral rule for handling of width,
   e.g., "Map fullwidth and halfwidth characters to their
   compatibility variants."]

Additional Mapping Rule:  [any additional mappings that are required
   or recommended, e.g., "Map non-ASCII space characters to ASCII
   space."]

Case Mapping Rule:  [the behavioral rule for handling of case, e.g.,
   "Unicode Default Case Folding"]

Normalization Rule:  [which Unicode normalization form is applied,
   e.g., "NFC"]

Directionality Rule:  [the behavioral rule for handling of right-to-
   left code points, e.g., "The 'Bidi Rule' defined in RFC 5893
   applies."]

Enforcement:  [which entities enforce the rules, and when that
   enforcement occurs during protocol operations]

Specification:  [a pointer to relevant documentation, such as an RFC
   or Internet-Draft]

To request a review, the registrant shall send a completed template to the precis@ietf.org list or its designated successor 📬

Factors to focus on while defining profiles and reviewing profile registrations include the following:

  • Would an existing PRECIS string class or profile solve the problem? If not, why not? (See Section 5.1 for related considerations.) 🤔
  • Is the problem being addressed by this profile well defined? 💭
  • Does the specification define what kinds of applications are involved, and the protocol elements to which this profile applies? 📋
  • Is the profile clearly defined? ✨
  • Is the profile based on an appropriate dividing line between user interface (culture, context, intent, locale, device limitations, etc.) and the use of conformant strings in protocol elements? 🎀
  • Are the width mapping, case mapping, additional mappings, normalization, and directionality rules appropriate for the intended use? 🌸
  • Does the profile explain which entities enforce the rules, and when such enforcement occurs during protocol operations? 🦋
  • Does the profile reduce the degree to which human users could be surprised or confused by application behavior (the "Principle of Least Astonishment")? 🫧
  • Does the profile introduce any new security concerns, such as those described under Section 12 of this document (e.g., false positives for authentication or authorization)? 🚨

12. Security Considerations 💀

12.1. General Issues ✨

okay so if strings that look "the same" to users get treated as distinct in different systems, or strings that look distinct to users get treated as "the same" in different systems, users can get seriously confused 😭. That confusion can have security implications, like the false positives and false negatives discussed in [RFC6943]. One starting goal of PRECIS was to limit how often users get confused (consistent with the "Principle of Least Astonishment") 💖. Unfortunately, that's been really hard to pull off because there are already so many application protocols out there 💀. Still, profiles should not be multiplied beyond necessity (see Section 5.1) 🫶. App protocol designers should think long and hard before making a brand new profile instead of using one that already exists, and if they do make one, they should clearly explain why 💅.

The security of apps that use this framework can depend partly on the proper preparation, enforcement, and comparison of internationalized strings 🔐. For example, these strings can be used to make authentication and authorization decisions, and an app's security could be compromised if someone providing a string gets connected to the wrong account or online resource because of different interpretations of that string (again, see [RFC6943]) 😬.

Specs for application protocols that use this framework are strongly encouraged to describe how internationalized strings are used in the protocol, including the security implications of any false positives and false negatives that might result from various enforcement and comparison operations 🌸. For some helpful guidelines, refer to [RFC6943], [RFC5890], [UTR36], and [UTS39] 📚.

12.2. Use of the IdentifierClass 🦋

Strings that conform to the IdentifierClass and any profile of it are meant to be relatively safe for use in a broad range of applications 🛡️. That's mainly because they only include letters, digits, and "grandfathered" non-space characters from the ASCII range ✨. So they exclude spaces, characters with compatibility equivalents, and almost all symbols and punctuation marks. However, because these strings can still include so-called confusable characters (see Section 12.5), protocol designers and implementers are encouraged to pay close attention to the security considerations described elsewhere in this document 👑.

12.3. Use of the FreeformClass 🌈

Strings that conform to the FreeformClass and many profiles of it can include virtually any Unicode character 🎨. That makes it super expressive, but also kinda problematic when it comes to possible user confusion 😵. Protocol designers are hereby warned that the FreeformClass contains code points they might not understand 🙈, so they're encouraged to profile the IdentifierClass wherever feasible. However, if an application protocol needs more code points than the IdentifierClass allows, protocol designers are encouraged to define a profile of the FreeformClass that restricts the allowable code points as tightly as possible 💖.

(Fun fact: the PRECIS Working Group considered allowing "superclasses" as well as profiles of PRECIS string classes, but decided against allowing superclasses to reduce the likelihood of security and interoperability problems 🧚.)

12.4. Local Character Set Issues 🍓

When systems use local character sets other than ASCII and Unicode, this specification leaves the problem of converting between the local character set and Unicode up to the application or local system 🤷‍♀️. If different applications (or different versions of one application) implement different rules for conversions among coded character sets, they could interpret the same name differently and contact different application servers or other network entities 😱. This problem is not solved by security protocols, such as Transport Layer Security (TLS) [RFC5246] and the Simple Authentication and Security Layer (SASL) [RFC4422], that do not take local character sets into account 🫠.

12.5. Visually Similar Characters 👀

Some characters look visually similar and can cause confusion among humans, which is why they're often called "confusable characters" or "confusables" ✨.

The problem of confusable characters isn't necessarily caused by the use of Unicode code points outside the ASCII range 🤓. For example, in some presentations and to some individuals, the string ju1iet (spelled with DIGIT ONE, U+0031, as the third character) might appear to be the same as juliet (spelled with LATIN SMALL LETTER L, U+006C), especially on casual visual inspection. This phenomenon is sometimes called "typejacking" 😳.

However, the problem gets way more serious when you introduce the full range of Unicode code points into protocol strings 🌪️. For example, the characters U+13DA U+13A2 U+13B5 U+13AC U+13A2 U+13AC U+13D2 from the Cherokee block look similar to the ASCII characters "STPETER" as they might appear when presented using a "creative" font family 🎭.

In some examples of confusable characters, it is unlikely that the average human could tell the difference between the real string and the fake string 🥺. (Indeed, there is no programmatic way to distinguish with full certainty which is the fake string and which is the real string. In some contexts, the string formed of Cherokee characters might be the real string and the string formed of ASCII characters might be the fake string 🤯.) Because PRECIS-compliant strings can contain almost any properly encoded Unicode code point, it can be relatively easy to fake or mimic some strings in systems that use the PRECIS framework 🎣. The fact that some strings are easily confused introduces security vulnerabilities of the kind that have also plagued the World Wide Web, specifically the phenomenon known as phishing 🚨.

Despite the fact that some specific suggestions about identification and handling of confusable characters appear in the Unicode Security Considerations [UTR36] and the Unicode Security Mechanisms [UTS39] 📚, it is also true (as noted in [RFC5890]) that "there are no comprehensive technical solutions to the problems of confusable characters." Because it is impossible to map visually similar characters without a great deal of context (such as knowing the font families used), the PRECIS framework does nothing to map similar-looking characters together, nor does it prohibit some characters because they look like others 🙅‍♀️.

Nevertheless, specifications for application protocols that use this framework are strongly encouraged to describe how confusable characters can be abused to compromise the security of systems that use the protocol in question, along with any protocol-specific suggestions for overcoming those threats 💪. In particular, software implementations and service deployments that use PRECIS-based technologies are strongly encouraged to define and implement consistent policies regarding the registration, storage, and presentation of visually similar characters 🫶. The following recommendations are appropriate:

  1. An application service SHOULD define a policy that specifies the scripts or blocks of characters that the service will allow to be registered (e.g., in an account name) or stored (e.g., in a filename). Such a policy SHOULD be informed by the languages and scripts that are used to write registered account names; in particular, to reduce confusion, the service SHOULD forbid registration or storage of strings that contain characters from more than one script and SHOULD restrict registrations to characters drawn from a very small number of scripts (e.g., scripts that are well understood by the administrators of the service, to improve manageability) 🧁.
  2. User-oriented application software SHOULD define a policy that specifies how internationalized strings will be presented to a human user. Because every human user of such software has a preferred language or a small set of preferred languages, the software SHOULD gather that information either explicitly from the user or implicitly via the operating system of the user's device. Furthermore, because most languages are typically represented by a single script or a small set of scripts, and because most scripts are typically contained in one or more blocks of characters, the software SHOULD warn the user when presenting a string that mixes characters from more than one script or block, or that uses characters outside the normal range of the user's preferred language(s). (Such a recommendation is not intended to discourage communication across different communities of language users; instead, it recognizes the existence of such communities and encourages due caution when presenting unfamiliar scripts or characters to human users.) 🌍

The challenges inherent in supporting the full range of Unicode code points have in the past led some to hope for a way to programmatically negotiate more restrictive ranges based on locale, script, or other relevant factors; to tag the locale associated with a particular string; etc. As a general-purpose internationalization technology, the PRECIS framework does not include such mechanisms 🌷.

12.6. Security of Passwords 🔐✨

Okay bestie, passwords need to be SO secure, and there are two big goals here 😤. First, maximize the entropy (more randomness means harder to guess). Second, minimize false positives (so the wrong password never accidentally gets in 💀). The way to hit both is to allow a wide range of code points and to handle passwords so code points aren't compared aggressively. So it is NOT RECOMMENDED for application protocols to profile the FreeformClass for passwords in a way that wipes out entire categories, like banning symbols or punctuation 🙅‍♀️. And it is NOT RECOMMENDED for application protocols to map uppercase and titlecase code points to their lowercase versions in these strings. Instead, it is RECOMMENDED to keep the case of every code point and compare them in a case-sensitive way 💅🎀.

That said, implementers need to know there's a tradeoff between entropy and usability 🫠. For example, letting someone set a password with "uncommon" code points might make it really hard for them to log in when they're on an unfamiliar or limited input device. Lowkey a real problem 😭.

Some application protocols use passwords directly, while others reuse technologies that already process passwords (the Simple Authentication and Security Layer [RFC4422] is one example 🧚). Passwords also often travel through a chain of protocols to backend authentication or data storage systems, like RADIUS [RFC2865] and the Lightweight Directory Access Protocol (LDAP) [RFC4510]. So developers of application protocols are encouraged to reuse these existing profiles instead of inventing new ones. That way, end-user expectations about passwords stay consistent no matter which application protocol they're using 🌈🫶.

In protocols that pass a password into a cryptographic algorithm, like a hash function, the client will need to prepare the password properly before applying the algorithm, because the server never sees the password in plaintext form 🔥. More on password handling can be found in [PRECIS-Users-Pwds] 📚.

13. Interoperability Considerations 🌸

13.1. Encoding

Strings used in PRECIS-based application protocols are often encoded in UTF-8 [RFC3629] 💖. Still, the exact encoding is up to the application protocol that uses PRECIS, not the PRECIS framework itself 🦋.

13.2. Character Sets

Some existing systems can't handle the full Unicode character set, or even any characters outside the ASCII range 😬. If two or more applications need to interoperate when exchanging data (say, to authenticate a username or password), they will naturally need at least one coded character set in common (as defined by [RFC6365]) 🤝. Setting up that shared baseline is the job of the application protocol that uses PRECIS, not the PRECIS framework ✨.

13.3. Unicode Versions

The properties of Unicode code points can change as the Unicode Standard gets modified over time 🕰️. For example, three code points changed their GeneralCategory between Unicode 5.2 (the version current when IDNA2008 was originally published) and Unicode 6.0, as described in [RFC6452] 👀. Implementers might need to watch out, because how they treat those characters can depend on which Unicode version the system is running for IDNA2008 or PRECIS. Other differences might show up between the version current at the time of this writing (7.0) and future versions 🔮.

13.4. Potential Changes to Handling of Certain Unicode Code Points

As part of the review of Unicode 7.0 for IDNA, someone raised a question about a newly added code point 🤔. That question led to a re-analysis of the normalization rules that IDNA uses and that this document inherits (Section 5.2.4). Some of the general issues are described in [IAB-Statement] and explored in more depth in [IDNA-Unicode] 📖.

These issues are not settled yet 🙃. Still, implementers need to know this specification will likely be updated in the future to address them. The potential changes include the following:

  • The range of characters in the LetterDigits category (Sections 4.2.1 and 9.1) might be narrowed.
  • Some characters with special properties that are now allowed might be excluded.
  • More "Additional Mapping Rules" (Section 5.2.2) might be defined.
  • Alternative normalization methods might be added.

Good news though: implementations and deployments that follow the advice in this specification are unlikely to run into significant problems from these issues or changes 🙌. The key advice is to use the more restrictive IdentifierClass whenever possible or, if using the FreeformClass, to allow only a restricted set of characters. Especially avoid characters whose implications you don't actually understand 🫶💅.

14. References 📚

14.1. Normative References

[RFC20]    Cerf, V., "ASCII format for network interchange", STD 80,
           RFC 20, DOI 10.17487/RFC0020, October 1969,
           <http://www.rfc-editor.org/info/rfc20>.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <http://www.rfc-editor.org/info/rfc2119>.

[RFC5198]  Klensin, J. and M. Padlipsky, "Unicode Format for Network
           Interchange", RFC 5198, DOI 10.17487/RFC5198, March 2008,
           <http://www.rfc-editor.org/info/rfc5198>.

[RFC6365]  Hoffman, P. and J. Klensin, "Terminology Used in
           Internationalization in the IETF", BCP 166, RFC 6365,
           DOI 10.17487/RFC6365, September 2011,
           <http://www.rfc-editor.org/info/rfc6365>.

[Unicode]  The Unicode Consortium, "The Unicode Standard",
           <http://www.unicode.org/versions/latest/>.

[Unicode7.0]
           The Unicode Consortium, "The Unicode Standard, Version
           7.0.0", (Mountain View, CA: The Unicode Consortium, 2014
           ISBN 978-1-936213-09-2),
           <http://www.unicode.org/versions/Unicode7.0.0/>.

14.2. Informative References 📚

Okay, bestie, this is the "further reading" stack, the stuff the spec points to for background and context. None of it is a must-read to implement, but it's where the whole family tree lives. 🌳✨

[DerivedCoreProperties] The Unicode Consortium, "DerivedCoreProperties-7.0.0.txt", Unicode Character Database, February 2014, http://www.unicode.org/Public/UCD/latest/ucd/DerivedCoreProperties.txt.

[IAB-Statement] Internet Architecture Board, "IAB Statement on Identifiers and Unicode 7.0.0", February 2015, https://www.iab.org/documents/correspondence-reports-documents/2015-2/iab-statement-on-identifiers-and-unicode-7-0-0/.

[IDNA-Unicode] Klensin, J. and P. Faltstrom, "IDNA Update for Unicode 7.0.0", Work in Progress, draft-klensin-idna-5892upd-unicode70-04, March 2015. 🦋

[PRECIS-Mappings] Yoneya, Y. and T. Nemoto, "Mapping characters for PRECIS classes", Work in Progress, draft-ietf-precis-mappings-10, May 2015.

[PRECIS-Nickname] Saint-Andre, P., "Preparation, Enforcement, and Comparison of Internationalized Strings Representing Nicknames", Work in Progress, draft-ietf-precis-nickname-17, April 2015.

[PRECIS-Users-Pwds] Saint-Andre, P. and A. Melnikov, "Preparation, Enforcement, and Comparison of Internationalized Strings Representing Usernames and Passwords", Work in Progress, draft-ietf-precis-saslprepbis-17, May 2015. 🔐

[PropertyAliases] The Unicode Consortium, "PropertyAliases-7.0.0.txt", Unicode Character Database, November 2013, http://www.unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt.

[RFC2865] Rigney, C., Willens, S., Rubens, A., and W. Simpson, "Remote Authentication Dial In User Service (RADIUS)", RFC 2865, DOI 10.17487/RFC2865, June 2000, http://www.rfc-editor.org/info/rfc2865.

[RFC3454] Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, DOI 10.17487/RFC3454, December 2002, http://www.rfc-editor.org/info/rfc3454.

[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello, "Internationalizing Domain Names in Applications (IDNA)", RFC 3490, DOI 10.17487/RFC3490, March 2003, http://www.rfc-editor.org/info/rfc3490.

[RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)", RFC 3491, DOI 10.17487/RFC3491, March 2003, http://www.rfc-editor.org/info/rfc3491.

[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, DOI 10.17487/RFC3629, November 2003, http://www.rfc-editor.org/info/rfc3629.

[RFC4422] Melnikov, A., Ed., and K. Zeilenga, Ed., "Simple Authentication and Security Layer (SASL)", RFC 4422, DOI 10.17487/RFC4422, June 2006, http://www.rfc-editor.org/info/rfc4422.

[RFC4510] Zeilenga, K., Ed., "Lightweight Directory Access Protocol (LDAP): Technical Specification Road Map", RFC 4510, DOI 10.17487/RFC4510, June 2006, http://www.rfc-editor.org/info/rfc4510.

[RFC4690] Klensin, J., Faltstrom, P., Karp, C., and IAB, "Review and Recommendations for Internationalized Domain Names (IDNs)", RFC 4690, DOI 10.17487/RFC4690, September 2006, http://www.rfc-editor.org/info/rfc4690.

[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, DOI 10.17487/RFC5226, May 2008, http://www.rfc-editor.org/info/rfc5226.

[RFC5234] Crocker, D., Ed., and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/RFC5234, January 2008, http://www.rfc-editor.org/info/rfc5234.

[RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, DOI 10.17487/RFC5246, August 2008, http://www.rfc-editor.org/info/rfc5246.

[RFC5890] Klensin, J., "Internationalized Domain Names for Applications (IDNA): Definitions and Document Framework", RFC 5890, DOI 10.17487/RFC5890, August 2010, http://www.rfc-editor.org/info/rfc5890.

[RFC5891] Klensin, J., "Internationalized Domain Names in Applications (IDNA): Protocol", RFC 5891, DOI 10.17487/RFC5891, August 2010, http://www.rfc-editor.org/info/rfc5891.

[RFC5892] Faltstrom, P., Ed., "The Unicode Code Points and Internationalized Domain Names for Applications (IDNA)", RFC 5892, DOI 10.17487/RFC5892, August 2010, http://www.rfc-editor.org/info/rfc5892.

[RFC5893] Alvestrand, H., Ed., and C. Karp, "Right-to-Left Scripts for Internationalized Domain Names for Applications (IDNA)", RFC 5893, DOI 10.17487/RFC5893, August 2010, http://www.rfc-editor.org/info/rfc5893.

[RFC5894] Klensin, J., "Internationalized Domain Names for Applications (IDNA): Background, Explanation, and Rationale", RFC 5894, DOI 10.17487/RFC5894, August 2010, http://www.rfc-editor.org/info/rfc5894.

[RFC5895] Resnick, P. and P. Hoffman, "Mapping Characters for Internationalized Domain Names in Applications (IDNA) 2008", RFC 5895, DOI 10.17487/RFC5895, September 2010, http://www.rfc-editor.org/info/rfc5895.

[RFC6452] Faltstrom, P., Ed., and P. Hoffman, Ed., "The Unicode Code Points and Internationalized Domain Names for Applications (IDNA) - Unicode 6.0", RFC 6452, DOI 10.17487/RFC6452, November 2011, http://www.rfc-editor.org/info/rfc6452.

[RFC6885] Blanchet, M. and A. Sullivan, "Stringprep Revision and Problem Statement for the Preparation and Comparison of Internationalized Strings (PRECIS)", RFC 6885, DOI 10.17487/RFC6885, March 2013, http://www.rfc-editor.org/info/rfc6885.

[RFC6943] Thaler, D., Ed., "Issues in Identifier Comparison for Security Purposes", RFC 6943, DOI 10.17487/RFC6943, May 2013, http://www.rfc-editor.org/info/rfc6943.

[UAX11] Unicode Standard Annex #11, "East Asian Width", edited by Ken Lunde. An integral part of The Unicode Standard, http://unicode.org/reports/tr11/.

[UAX15] Unicode Standard Annex #15, "Unicode Normalization Forms", edited by Mark Davis and Ken Whistler. An integral part of The Unicode Standard, http://unicode.org/reports/tr15/.

[UAX9] Unicode Standard Annex #9, "Unicode Bidirectional Algorithm", edited by Mark Davis, Aharon Lanin, and Andrew Glass. An integral part of The Unicode Standard, http://unicode.org/reports/tr9/.

[UTR36] Unicode Technical Report #36, "Unicode Security Considerations", by Mark Davis and Michel Suignard, http://unicode.org/reports/tr36/.

[UTS39] Unicode Technical Standard #39, "Unicode Security Mechanisms", edited by Mark Davis and Michel Suignard, http://unicode.org/reports/tr39/.

[XMPP-Addr-Format] Saint-Andre, P., "Extensible Messaging and Presence Protocol (XMPP): Address Format", Work in Progress, draft-ietf-xmpp-6122bis-22, May 2015.

Acknowledgements 💖

The authors would like to acknowledge the comments and contributions of the following individuals during working group discussion: David Black, Edward Burns, Dan Chiba, Mark Davis, Alan DeKok, Martin Duerst, Patrik Faltstrom, Ted Hardie, Joe Hildebrand, Bjoern Hoehrmann, Paul Hoffman, Jeffrey Hutzelman, Simon Josefsson, John Klensin, Alexey Melnikov, Takahiro Nemoto, Yoav Nir, Mike Parker, Pete Resnick, Andrew Sullivan, Dave Thaler, Yoshiro Yoneya, and Florian Zeitz. 🫶

Special thanks are due to John Klensin and Patrik Faltstrom for their challenging feedback and detailed reviews. 👑

Charlie Kaufman, Tom Taylor, and Tim Wicinski reviewed the document on behalf of the Security Directorate, the General Area Review Team, and the Operations and Management Directorate, respectively.

During IESG review, Alissa Cooper, Stephen Farrell, and Barry Leiba provided comments that led to further improvements.

Some algorithms and textual descriptions have been borrowed from [RFC5892]. Some text regarding security has been borrowed from [RFC5890], [PRECIS-Users-Pwds], and [XMPP-Addr-Format].

Peter Saint-Andre wishes to acknowledge Cisco Systems, Inc., for employing him during his work on earlier draft versions of this document. 🌸

Authors' Addresses 🦋

Peter Saint-Andre
&yet

EMail: peter@andyet.com
URI:   https://andyet.com/


Marc Blanchet
Viagenie
246 Aberdeen
Quebec, QC  G1R 2E1
Canada

EMail: Marc.Blanchet@viagenie.ca
URI:   http://www.viagenie.ca/

And that's the whole spec, babe! 🧚💜 Go forth and implement responsibly, bestie. 💅

📦 where this page comes from