RFC 8753

Internationalized Domain Names for Applications (IDNA) Review for New Unicode Versions

📅 published April 2020, from ietf

RFC 8753: Internationalized Domain Names for Applications (IDNA) Review for New Unicode Versions 🎀

Okay bestie, this is RFC 8753 💅 It's an IETF doc from April 2020, published on the Standards Track (ISSN 2070-1721). The authors are J. Klensin and P. Fältström from Netnod. It updates RFC 5892, which is the one that says how Unicode code points map to IDNA derived properties. The title block doesn't list anything it obsoletes.

Abstract ✨

Okay so the IDNA standards need a review of every new version of Unicode 🌸 The point is to check for incompatibilities with older versions and other problems, and where it makes sense, to let the IETF decide how to trade off compatibility with older IDNA versions against compatibility with Unicode going forward. That requirement, and how it connects to the tables IANA maintains, has caused a LOT of confusion. This doc tweaks the review procedure based on what we've learned, updates IDNA (specifically RFC 5892) so it matches those changes, and clarifies how all the pieces fit together. It also makes some smaller adjustments so that document lines up with experience 🦋

Status of This Memo

This is an Internet Standards Track document 💜

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 7841.

Info about the current status of this document, any errata, and how to give feedback on it is at https://www.rfc-editor.org/info/rfc8753 🔗

Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they were in effect on the date this document was published. Please read these carefully, because they spell out your rights and restrictions for this document. Code Components pulled out of this document must include the Simplified BSD License text described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 🩷

Table of Contents

1.  Introduction
2.  Brief History of IDNA Versions, the Review Requirement, and RFC
        5982
3.  The Review Model
  3.1.  Review Model Part I: Algorithmic Comparison
  3.2.  Review Model Part II: New Code Point Analysis
4.  IDNA Assumptions and Current Practice
5.  Derived Tables Published by IANA
6.  Editorial Clarification to RFC 5892
7.  IANA Considerations
8.  Security Considerations
9.  References
  9.1.  Normative References
  9.2.  Informative References
Appendix A.  Summary of Changes to RFC 5892
Appendix B.  Background and Rationale for Expert Review Procedure
        for New Code Point Analysis
Acknowledgments
Authors' Addresses

1. Introduction 🧚

Okay so the IDNA standards require a review of each new version of Unicode 💅 The goal is to figure out whether there are incompatibilities with prior versions or other problems, and, when it's appropriate, to let the IETF weigh the trade-offs between compatibility with older IDNA versions and compatibility with Unicode [Unicode] going forward. That requirement, and how it relates to the tables IANA maintains, has caused significant confusion in the past (see Section 3 and Section 4 for more on the question of appropriate decisions and the history of these reviews 🙈). This document adjusts the review procedure based on nearly a decade of experience. It also updates IDNA, specifically the document that defines the relationship between Unicode code points and IDNA derived properties [RFC5892], so it reflects those changes and clears up how everything connects 🦋

This spec does NOT change the requirement that registries at every level of the DNS tree take responsibility for the labels they insert into the DNS. That responsibility means allowing only a subset of the code points and strings the IDNA protocol itself allows. This is covered in more detail in a companion document [RegRestr] 💖

Terminology note: here, "IDNA" means the current version described in RFC 5890 [RFC5890] and later documents, and it's sometimes called "IDNA2008". The differences between it and the earlier version only get spelled out when they matter for understanding the relationships, like in Section 2 ✨

2. Brief History of IDNA Versions, the Review Requirement, and RFC 5982 🌈

Okay this is the tea ☕ The original version of IDNA, now obsolete and commonly known as "IDNA2003" [RFC3490] [RFC3491], was defined as a profile of a collection of IETF-specific tables [RFC3454]. Those tables said whether each Unicode code point could be used with IDNA. Since the tables were normative, they were tied to one specific version of Unicode. As Unicode evolved, IDNA2003 would have needed a new profile for every new Unicode version, or else the tables would keep falling further and further behind 🍓

When IDNA2003 got replaced by the current version, known as IDNA2008 [RFC5890], the designers went with a different strategy. It was property-based instead of table-based, and that was chosen for a bunch of reasons, with the reliance on normative tables not being the main one [RFC4690]. In IDNA2008, the normative tables got replaced with a set of procedures and rules that operate on Unicode properties [Unicode-properties] plus a few internal definitions. Together, these determine the category and status, and so the IDNA-specific "derived property", for any given code point. In principle, those rules don't depend on the Unicode version. They can be applied to any version of Unicode, at least from about version 5.0 onward, to produce the appropriate set of derived properties 💜

Here's the catch though 😭 The working group that defined IDNA2008 knew that not all Unicode properties are completely stable. Also, the criteria the Unicode Consortium uses for new code points and property assignments might not line up exactly with what IDNA needs. So there were possible incompatible changes to derived property values. Specifically, there could be changes that make previously disallowed labels valid, make previously valid labels disallowed, or disrupt IDNA's defining rule structure. Because of that, IDNA2008 provided for an expert review of each new Unicode version. The review could include exceptions to the rules for particular new code points, code points whose properties had changed, and newly discovered problems with the IDNA2008 rules. When problems were found, the reviewer was expected to notify the IESG. The assumption was that the IETF would look at the situation and modify IDNA2008 as needed, most likely by adding exceptions to preserve backward compatibility (see Section 3.1) 🔥

For the community's convenience, IDNA2008 also said that IANA would maintain copies of the calculated tables from each review, showing the derived properties for each code point. Those tables were expected to be really helpful, especially for people who don't have an easy way to compute derived properties themselves 🫶 But experience has shown that people have confused them with the normative tables of IDNA2003. The IDNA2008 tables published by IANA have never been normative. So any statement that IDNA2008 is out of date for some Unicode version because the IANA tables haven't been updated is incorrect or meaningless 💀

3. The Review Model 👑

Okay so the text has sometimes been read differently, but IDNA2008 actually calls for two types of review when a new Unicode version comes out 🔍 The first is an algorithmic comparison: take the set of derived properties calculated from the new Unicode version and compare it to the set calculated from the previous one, to see whether any incompatible changes happened. The second is a review of newly assigned code points, to check whether any of them need special treatment (for example, assigning what IDNA2008 calls contextual rules) and whether any of them break the assumptions behind the IDNA2008 derived property calculations. Either kind of review might lead to per-code-point exceptions or other adjustments to the rules for deriving properties that are part of RFC 5892. The subsections below give a revised specification for the review procedure ✨

Unless the IESG or the designated expert team concludes there are special problems or unusual circumstances, these reviews will only be done for major Unicode versions (the ones numbered NN.0, like 12.0), and NOT for minor updates (like 12.1) 🌸

As the detailed descriptions in the following subsections show, a proper review needs a team of experts. That team needs both broad and specific skills for reviewing Unicode characters and their properties, in relation to both the written standards and operational needs 🧚 The IESG will need to appoint experts who can tap into the broader community to get the skills needed for particular situations. See the IANA Considerations (Section 7) for details 💅

3.1. Review Model Part I: Algorithmic Comparison 🦋

Okay so Section 5.1 of RFC 5892 describes how the initial IANA tables were created. It's worth noting that, although it can be read as strongly implying new reviews and new tables for Unicode versions after 5.2, it doesn't explicitly require those reviews or, for example, a timetable for finishing them. It also says incompatibilities are to be "flagged for the IESG", but it doesn't spell out what the IESG is supposed to do about them, or when 🙈 For reasons tied to the other type of review, which is discussed below, only one review was actually completed and documented [RFC6452], and a corresponding set of new tables was installed. That review was for Unicode 6.0. It found only three incompatibilities, and the consensus was to ignore them. That meant not creating exceptions in IDNA2008, and staying consistent with calculations based on the current (Unicode 6.0) properties, rather than preserving backward compatibility within IDNA 💖 The 2018 review (covering Unicode 11.0 and the versions between it and 6.0) [IDNA-Unicode12] also concluded that Unicode compatibility, rather than IDNA backward compatibility, should be maintained. That decision was partly driven by the long gap between reviews, and by the concern that table calculations done by others in the meantime could cause unexpected incompatibilities if derived property definitions were changed later 🌈 See Section 4 for more discussion of these preferences 🎀

3.2. Review Model Part II: New Code Point Analysis 🔍

okay so the second kind of review is the one RFC 5892 doesn't really explain 👀. Its whole job is to catch newly added or recently discovered code points that are problematic, the ones that break IDNA's design assumptions 🫶. It also looks for flaws in those assumptions and for spots where IDNA doesn't match Unicode's promises about assignment, properties, and stability of new code points 💅. A classic example: after Unicode 7.0, people found new code points that could look basically identical to older code point sequences in the same script [IAB-Unicode7-2015] [IDNA-Unicode7]. Iconic chaos 😭.

This review is not gonna work unless a team does it 👑. Finding one all-knowing expert is highkey impossible, and even a single brilliant expert usually can't produce a solid analysis on their own 💀. So the designated expert (DE) team has to accept that there will always be gaps in what they know, figure out exactly what they don't know, and go find the right expertise for each review 🧚. It's also super important that the DE team stays in close contact with the Area Directors (ADs), and that the ADs stay aware of the team's changing needs 🌸. The ADs should keep checking and adjusting the team's membership over time, with periodic reexamination at least annually 🗓️.

Also, be prepared: if this review finds a problem, that problem is probably complex and/or full of trade-offs 🙈. Fixing it is likely to be disruptive (maybe not for huge communities of users, but still), or it can leave security risks, like opportunities for attacks and inadvertent confusion when expected matches don't happen 😬. Or it can lead to too much reliance on registries understanding and taking responsibility for what they register [RFC5894] [RegRestr]. That last one is a requirement of IDNA, but honestly it hasn't gone well in the past 💀.

Because fixing problems found by this part of the review can take a while, even when the fix is just adding extra contextual rules or disallowing one or more code points, there will be times when it makes sense to publish the results of the algorithmic review anyway ✨. That means giving IANA the matching tables, with warnings about code points whose status is still uncertain until the IETF reaches consensus on how to proceed 🚦. Those affected code points should be considered unsafe and marked as "under review" in the IANA tables until final derived properties are assigned 🫣.

4. IDNA Assumptions and Current Practice 🌈

When the IDNA2008 documents were written, the assumption was that if new versions of Unicode introduced incompatible changes, the Standard would get updated to keep backward compatibility for IDNA users 🥹. Mostly, that would happen by adding to the table of exceptions tied to Rule G [RFC5892a].

Plot twist: that is not what happened in the reviews after Unicode 5.2 💅. Section 3 covers this. Incompatibilities showed up in Unicode 6.0 [RFC6452] and in the cumulative review that led to the tables for Unicode 11.0 [IDNA-Unicode11]. In every one of those cases, they chose to stay compatible with Unicode properties instead of prior versions of IDNA 🙃.

If an algorithmic review finds changes in Unicode after version 12.0 that would break compatibility with derived properties linked to earlier Unicode versions, or changes that would keep compatibility inside IDNA while departing from current Unicode specifications, those changes must be captured in documents that are expected to be published as Standards Track RFCs 📜. That way the IETF can review them and keep a historical record 🦋.

The community has now made decisions and updated tables for Unicode 6.0 [RFC6452], did the catch-up work between it and Unicode 11.0 [IDNA-Unicode11], and finished the review and tables for Unicode 12.0 [IDNA-Unicode12] 🎀. Those decisions were driven by keeping consistency with Unicode and Unicode property changes, for reasons best explained by the IAB [IAB-Unicode-2018] ✨. But honestly? They went against the language in RFC 5892, and they also clashed with promises made to registry and user communities 😭. Those promises were that IDN labels that were once valid under IDNA2008 would stay valid, and labels that were previously invalid would stay invalid, except for labels that were invalid only because they contained unassigned code points 💜.

This document brings back and clarifies that original language and intent 🔥. Unless there is extremely strong evidence, on a per-code point basis, that preserving the validity status of possible existing (or prohibited) labels would cause significant harm, Unicode changes that would affect IDNA derived properties are supposed to be reflected in IDNA exceptions that preserve the status of those labels 🫶. There is one partial exception to this principle. If the new code point analysis (see Section 3.2) concludes that some code points or collections of code points should be analyzed further, then those code points, and labels including them, should be considered unsafe and used only with extreme caution 🚨. The reason is that the analysis's conclusions may change their derived property values and status 🌸.

5. Derived Tables Published by IANA 📋

As discussed above, RFC 5892 said that derived property tables should be provided through an IANA registry 🗂️. Maybe because most IANA registries are treated as normative and authoritative, that registry has caused a lot of confusion 🤯. That includes the wrong assumption that the lack of published tables for Unicode versions later than 6.0 meant IDNA couldn't be used with later versions 💀. That position came up in several ways, not all of them consistent, especially in the ICANN context [ICANN-LGR-SLA].

If the changes in this document don't significantly clear up the confusion about the status of the tables IANA publishes, then serious consideration should be given to eliminating those tables entirely 🥀.

6. Editorial Clarification to RFC 5892 ✏️

This section updates RFC 5892 to fix known applicable errata and omissions 💖. Specifically, verified RFC Editor Erratum 3312 [Err3312] clarifies Appendix A and A.1 in RFC 5892, and that clarification is folded in below 🧚.

  1. In Appendix A, add a new paragraph after the paragraph that begins "The code point...". The new paragraph should read:
|  For the rule to be evaluated to True for the label, it MUST be
|  evaluated separately for every occurrence of the code point in
|  the label; each of those evaluations must result in True.
  1. In Appendix A.1, replace the "Rule Set" by:
Rule Set:
  False;
  If Canonical_Combining_Class(Before(cp)) .eq. Virama
        Then True;
  If cp .eq. \u200C And
         RegExpMatch((Joining_Type:{L,D})(Joining_Type:T)*cp
    (Joining_Type:T)*(Joining_Type:{R,D})) Then True;

7. IANA Considerations 🏳️‍🌈

For the algorithmic review described in Section 3.1, the IESG is to appoint a designated expert [RFC8126] with appropriate expertise to conduct the review and supply derived property tables to IANA 🎯. As provided in Section 5.2 of the Guidelines for Writing IANA Considerations [RFC8126], the designated expert is expected to consult additional sources of expertise as needed 🔎. For the code point review, the expertise will be supplied by an IESG-designated expert team, as discussed in Section 3.2 and Appendix B 💜. In both cases, the experts should draw on the expertise of other members of the community as needed 🫶. Especially if the various roles don't overlap in people, coordination with the IAB-appointed liaison to the Unicode Consortium will be essential to mitigate possible errors due to confusion 🚦.

As discussed in Section 5, IANA has modified the IDNA tables collection [IANA-IDNA-Tables] by marking them clearly as non-normative 🙅‍♀️. That way, a "current" or "correct" version of those tables isn't implied, and the collection points to this document for an explanation 📖. IANA has published tables supplied by the IETF for all Unicode versions through 11.0, keeping all the older versions and making them available 🗃️. Newer tables will be constructed as specified in this document and then made available by IANA ✨. IANA has also changed the title of that registry from "IDNA Parameters", which was misleading, to "IDNA Rules and Derived Property Values" 🎀.

The "Note" in that registry says:

|  IDNA does not require that applications and libraries, either for
|  registration/storage or lookup, support any particular version of
|  Unicode.  Instead, they are required to use derived property
|  values based on calculations associated with whatever version of
|  Unicode they are using elsewhere in the application or library.
|  For the convenience of application and library developers and
|  others, the IETF has supplied, and IANA maintains, derived
|  property tables for several version of Unicode as listed below.
|  It should be stressed that these are not normative in that, in
|  principle, an application can do its own calculations and these
|  tables can change as IETF understanding evolves.  By contrast, the
|  list of code points requiring contextual rules and the associated
|  rules are normative and should be treated as updates to the list
|  in RFC 5892.

As long as the intent is preserved, IANA can change the text of that note in the future at its own discretion 🌷.

IANA's attention is called to the introduction, in Section 3.2, of a temporary "under review" category for the PVALID, DISALLOWED, etc., entries in the tables 🚨.

8. Security Considerations 🔐

Applying the procedures in this document, and understanding the clarifications it provides, should reduce confusion about IDNA requirements 🧠. Past confusion has given bad actors opportunities for bad behavior, so these changes should improve Internet security at least a little bit 🫡.

Because of the preference for keeping derived property values stable (as specified in RFC 5892 and discussed in Section 4), the algorithm used to calculate those derived properties does change, as explained in Section 3 ⚠️. If these changes aren't taken into account, the derived property value will change, and the implications might have negative consequences, in some cases with security implications 😱. For example, changes in the calculated derived property value for a code point from DISALLOWED to PVALID, or from PVALID to DISALLOWED, can change how labels are interpreted. That would be visible and confusing to end users, and it might enable attacks 💀🚨.

9. References 📚

9.1. Normative References

These are the must-read sources the whole spec leans on 📚✨

9.2. Informative References

These ones are the extra context, the "ohh, that's why" reading 🧚💖

Appendix A. Summary of Changes to RFC 5892

Okay, the changelog moment 💅 Apart from the editorial fix in Section 6, every change here is about the reviews for new Unicode versions, plus the IANA Considerations in Section 5 of [RFC5892], especially Section 5.1 of [RFC5892]. Whether each change is substantive or just a clarification is kinda in the eye of the beholder, so don't treat the list below as the full tea 🫶 At a high level, this document says two types of review were always meant to exist and splits them up so it's clear 🎀 It also brings back the original default for what happens when code point derived properties change, which nobody had actually been doing 😭 So this document basically replaces Section 5.1 of [RFC5892] and tweaks some text so the replacement makes better sense.

Here's the stuff that's giving "important" energy:

  • Split the new Unicode version review into two explicit parts, with different review methods and, potentially, asynchronous outcomes. 🦋
  • Specified a DE team, not a single designated expert, for the code point review. 👑
  • Got rid of the de facto rule that the (formerly single) designated expert had to be the same person as the IAB's liaison to the Unicode Consortium, but called out how important coordination still is. 🤝
  • Created the "Status" field in the IANA tables to tell the community about specific potentially problematic code points. This means info about those code points can go up before IETF review is done, instead of the review holding up use of the new Unicode version. 🚀
  • Unicode now runs on a regular one-year cycle instead of making major and minor versions whenever needed. To avoid overloading the IETF's internationalization resources and avoid generating and storing IANA tables for trivial changes (e.g., the single new code point in Unicode 12.1), the review procedure only applies to major versions of Unicode, unless exceptional circumstances come up and get identified. 🌸

Appendix B. Background and Rationale for Expert Review Procedure for New Code Point Analysis

Let's talk about why the review process is kinda unusual 🧠✨ The expert review procedure for new code point analysis in Section 3.2 is a little different from the examples in the Guidelines for Writing IANA Considerations [RFC8126]. This appendix explains that choice and gives the backstory.

Making specs for languages and writing systems other than English (and Latin script), so-called "internationalization" or "i18n", has always been a bit of a mess in the IETF 😭 Especially once the requirements go beyond simple character coding (e.g., RFC 3629 [RFC3629]) or simple language identification (e.g., RFC 3282 [RFC3282] and the earlier RFC 1766 [RFC1766]). A good amount of specialized knowledge is needed, and it comes from multiple fields and needs multiple perspectives. The work isn't obviously harder than routing, especially if you assume routing needs a solid base in graph theory or network optimization, or than security and cryptography. But people in those areas are drawn to the IETF, while people from the fields that bear on internationalization typically aren't 🙈

So we keep thinking we understand a problem, we write a spec or a set of specs, and then we get surprised by issues the IETF didn't anticipate 💀 Then we have to tune and often revise the spec. The language tag work that started with RFC 1766 is a great example: broader considerations and requirements led to later work and a much more complex, finer-grained system [RFC5646].

IDNs made things even harder 🌈 A lot of the decisions behind the current version of IDNA require understanding the DNS, its constraints, and, to at least some extent, the commercial market of domain names, including various ICANN efforts.

Put all of that together and it's extremely unlikely that the IESG will ever find one designated expert whose knowledge covers everything that's needed 😭💅

So Section 7 and other discussions in this document specify a DE team that's expected to have the broad perspective, expertise, and access to information and community needed to review new Unicode versions and make consensus recommendations that will serve the Internet well 👑 We expect the team will have one or more leaders, but its structure is different from the suggestions in Section 5.2 of the Guidelines for Writing IANA Considerations [RFC8126]. The team's formation and its consultation aren't left to the discretion of the designated expert, and the designated expert isn't solely accountable to the community. A team with multiple perspectives is needed, the team members are accountable as a group, and any nontrivial recommendations need team consensus. This also differs from the common IETF practice of "review teams" where a single member gets picked to do a review. The principle here is team effort 🫶✨

Acknowledgments 🫶

Okay so this doc was inspired by a LOT of conversations in the I18N Directorate of the IETF Applications and Real-Time (ART) area back in the first quarter of 2019 🌸 The whole goal was sorting out the reviews for Unicode 11.0 and 12.0, which is giving very big homework energy 💀

Joel Halpern did careful reviews, and Barry Leiba suggested text that led to some clarifications ✨

Thanks to Christopher Wood for catching editorial errors that stuck around pretty late in the document's life cycle 🙈 and to Benjamin Kaduk for catching and raising a bunch of questions during Last Call 💅 Some of the issues they brought up made it into the document. Others didn't seem like good changes after more discussion, but the questions were 100% worth raising and talking through 💜

Authors' Addresses 💖

John C Klensin
1770 Massachusetts Ave, Ste 322
Cambridge, MA 02140
United States of America

Phone: +1 617 245 1457
Email: john-ietf@jck.com
Patrik Fältström
Netnod
Greta Garbos Väg 13
SE-169 40 Solna
Sweden

Phone: +46 70 6059051
Email: paf@netnod.se
📦 where this page comes from