RFC 4690
Review and Recommendations for Internationalized Domain Names (IDNs)
RFC 4690: Review and Recommendations for Internationalized Domain Names (IDNs) 🎀
okay so this one is an Informational RFC from September 2006, and it's written by a crew 💅 J. Klensin and P. Faltstrom (both from Cisco Systems), C. Karp (from the Swedish Museum of Natural History), and the IAB itself. It came out of the Network Working Group 🦋
- Number and category: RFC 4690, Informational
- Date: September 2006
- Obsoletes or updates: the title block doesn't list any
Table of Contents 📚
1. Introduction ....................................................3
1.1. The Role of IDNs and This Document .........................3
1.2. Status of This Document and Its Recommendations ............4
1.3. The IDNA Standard ..........................................4
1.4. Unicode Documents ..........................................5
1.5. Definitions ................................................5
1.5.1. Language ............................................6
1.5.2. Script ..............................................6
1.5.3. Multilingual ........................................6
1.5.4. Localization ........................................7
1.5.5. Internationalization ................................7
1.6. Statements and Guidelines ..................................7
1.6.1. IESG Statement ......................................8
1.6.2. ICANN Statements ....................................8
2. General Problems and Issues ....................................11
2.1. User Conceptions, Local Character Sets, and Input issues ..11
2.2. Examples of Issues ........................................13
2.2.1. Language-Specific Character Matching ...............13
2.2.2. Multiple Scripts ...................................13
2.2.3. Normalization and Character Mappings ...............14
2.2.4. URLs in Printed Form ...............................16
2.2.5. Bidirectional Text .................................17
2.2.6. Confusable Character Issues ........................17
2.2.7. The IESG Statement and IDNA issues .................19
3. Migrating to New Versions of Unicode ...........................20
3.1. Versions of Unicode .......................................20
3.2. Version Changes and Normalization Issues ..................21
3.2.1. Unnormalized Combining Sequences ...................21
3.2.2. Combining Characters and Character Components ......22
3.2.3. When does normalization occur? .....................23
4. Framework for Next Steps in IDN Development ....................24
4.1. Issues within the Scope of the IETF .......................24
4.1.1. Review of IDNA .....................................24
4.1.2. Non-DNS and Above-DNS Internationalization
Approaches .........................................25
4.1.3. Security Issues, Certificates, etc. ................25
4.1.4. Protocol Changes and Policy Implications ...........27
4.1.5. Non-US-ASCII in Local Part of Email Addresses ......27
4.1.6. Use of the Unicode Character Set in the IETF .......27
4.2. Issues That Fall within the Purview of ICANN ..............28
4.2.1. Dispute Resolution .................................28
4.2.2. Policy at Registries ...............................28
4.2.3. IDNs at the Top Level of the DNS ...................29
5. Specific Recommendations for Next Steps ........................29
5.1. Reduction of Permitted Character List .....................29
5.1.1. Elimination of All Non-Language Characters .........30
5.1.2. Elimination of Word-Separation Punctuation .........30
5.2. Updating to New Versions of Unicode .......................30
5.3. Role and Uses of the DNS ..................................31
5.4. Databases of Registered Names .............................31
6. Security Considerations ........................................31
7. Acknowledgements ...............................................32
8. References .....................................................32
8.1. Normative References ......................................32
8.2. Informative References ....................................33Status of This Memo 📜
this memo is just here to give info to the Internet community 🫶 it does not specify an Internet standard of any kind, and it can be shared as much as you want.
Copyright Notice ©
Copyright (C) The Internet Society (2006).
Abstract 💌
this note is about the issues that popped up when people started deploying and using Internationalized Domain Names (IDNs) 🌈 it covers problems both when you're registering a name and when you're using that name in the DNS. the big recommendation is that the IETF should update the RFCs about IDNs, and it should also come up with a framework for doing that. the note also sums up and points out some work that needs to happen outside the IETF. specifically, it suggests investigating some changes to the Internationalizing Domain Names in Applications (IDNA) standard and its supporting tables, based on what we've learned since those standards were finished ✨
1. Introduction ✨
1.1. The Role of IDNs and This Document
okay so IDNs have been pitched as the answer to basically every problem 😭 but this document takes a more chill approach: IDNs are no more and no less than DNS names 🍓 they have the same requirements for use, stability, and accuracy as regular "hostnames," just with a way bigger collection of allowed characters. IDNs are a step toward an Internet that's equally accessible from all languages and scripts, but at best they only tackle a small piece of that super broad goal. there's been controversy since IDNs were first suggested about how important they'll actually turn out to be, and that drama will probably keep going 💅 still, being accessible in all languages really matters, so our standards and definitions for IDNs need to be smoothly adaptable as new scripts get added to the Unicode character set 🌸
the utility of IDNs has to be judged by how users apply them and how they work in protocols. being able to put a name into the DNS and get it back out isn't, on its own, the important part 🧚 IDNs are useful and effective if they give you stable and predictable references, and those references should be just as stable, predictable, and secure as their ASCII counterparts 🔥
this mix of goals and criteria has proven really hard to pull off 😩 experience developing the IDNA standard, plus its early years of implementation and deployment, suggests it may be impossible to fully satisfy all of them, so engineering compromises are needed to get something workable, even if it's not totally satisfactory. based on that experience and the issues people have raised, it's now time to look at some implications of IDNs, the decisions made when defining them, and the foundation they rest on, and then figure out whether changes are needed and, if so, which ones 👑
the design of the DNS itself adds some extra constraints 🎀 if the DNS is going to stay globally interoperable, there are specific characteristics that no implementation of IDNs (or the DNS more generally) can change. for example, the DNS is a global hierarchal administrative namespace with only a single name at any given node, so there is one and only one owner of each domain name 💖 also, when strings get looked up in the DNS, positive responses can only reflect exact matches. if there's no exact match, you get an error reply, not a list of near matches or other extra info. searches and approximate matching just aren't possible 🙈
last one, and it's a big one: the DNS is a distributed system where any server might cache responses and then use those cached responses to answer queries before a global lookup is done. because of that, every server must use the same matching criteria 🫶
1.2. Status of This Document and Its Recommendations
this document looks at the IDN landscape from an IETF point of view, and it presents the recommendations and conclusions of the IAB 🦋 those are partly based on input from an ad hoc committee that was charged with reviewing IDN issues and the path forward (see Section 7). the recommendations are advice to the IETF, or in a few cases to other bodies, about topics to investigate and actions to take, if those bodies decide those actions make sense after they've looked into things 🧠
1.3. The IDNA Standard
during 2002, the IETF finished the following RFCs that, together, define IDNs 💜
- RFC 3454, "Preparation of Internationalized Strings" ("Stringprep") [RFC3454]. Stringprep is a generic mechanism for taking a Unicode string and converting it into a canonical format. on its own it's just a collection of rules, tables, and operations. any protocol or algorithm that uses it must define a "Stringprep profile," which specifies which of those rules get applied, how they get applied, and with which characteristics 🌈
- RFC 3490, "Internationalizing Domain Names in Applications (IDNA)" [RFC3490]. IDNA is the base spec in this group 👑 it says Nameprep is used as the Stringprep profile for domain names, and that Punycode is the relevant encoding for generating an ASCII-compatible ("ACE") form of the name. it also applies some extra conversions and character filtering that aren't part of Nameprep ✨
- RFC 3491, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)" [RFC3491]. Nameprep is built to meet the specific needs of IDNs, and in particular to support case-folding for scripts that have what are traditionally called upper- and lowercase forms of the same letters 💅 the result of the Nameprep algorithm is a string containing a subset of the Unicode Character set, normalized and case-folded so that case-insensitive comparison can happen 🎀
- RFC 3492, "Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)" [RFC3492]. Punycode is a mechanism for encoding a Unicode string in ASCII characters 🦋 the characters it uses are the same subset that's allowed in the hostname definition of DNS, i.e., the "letter, digit, and hyphen" characters, sometimes known as "LDH" 🍓
1.4. Unicode Documents ✨
okay so Unicode is the base, and the defining, character set for IDNs 💖 It's standardized by the Unicode Consortium and kept in sync with ISO to make ISO/IEC 10646 [ISO10646]. When the RFCs mentioned earlier were written, Unicode was at Version 3.2. For reasons we'll get into later, the IDNA folks had to pick one particular, then-current, version of Unicode when IDNA got adopted 🎀. So the RFCs are explicitly tied to Unicode Version 3.2 [Unicode32]. There's currently no established way to update the IDNA RFCs to newer Unicode versions (see Section 3.1) 🥺
Unicode is a seriously huge and complicated character set 😭 (quick note: "character set" or "charset" means something kinda peculiar to the IETF, so it might not match how other bodies and contexts use it 🙈). The Unicode Standard and its related docs are made and looked after by the Unicode Technical Committee (UTC), which is one of the committees of the Unicode Consortium 🌸.
The Consortium first published The Unicode Standard [Unicode10] in 1991, and it's still building on that original work 👑. Unicode gets developed alongside the International Organization for Standardization, and it shares its character repertoire with ISO/IEC 10646. Unicode and ISO/IEC 10646 work the same way as character encodings, but The Unicode Standard has way more info for implementers, covering -- in depth -- stuff like bitwise encoding, collation, and rendering 🦋. It also lists a multitude of character properties, including the ones needed for supporting bidirectional text 💅. Just so you know, the Unicode Consortium and ISO standards use slightly different terminology 🫶.
1.5. Definitions ✨
these terms are super important for understanding the rest of this document and for chatting about IDNs more generally 💜. They're derived from [RFC3536], which has more discussion on some of them 🧚.
1.5.1. Language 🌈
a language is a way that humans interact 💖. Language shows up in lots of forms, like speech, writing, and signing ✨.
Some languages have a tight link between written and spoken forms, while others have a looser one 🎀. RFC 3066 [RFC3066] goes deeper into languages and gives identifiers for languages to use in Internet protocols 🦋. Computer languages are explicitly excluded from this definition 🙈. The most recent IETF work in this area, and on script identification (see below), is documented in [RFC4645] and [RFC4646] 💅.
1.5.2. Script 🌸
a script is a set of graphic characters used for the written form of one or more languages 👑. This is the definition used in [ISO10646].
Examples of scripts are Arabic, Cyrillic, Greek, Han (the so-called ideographs used in writing Chinese, Japanese, and Korean), and "Latin" 🧚. Arabic, Greek, and Latin are, of course, also names of languages 💖.
Historically, the script known as "Latin" in Unicode and most information technology standards contexts is called "Roman" or "Roman-derived" in the linguistic community 🎀. The latter term tells apart the Latin language and the characters used to write it, especially in Republican times, from the much richer and more decorated script derived and adapted from those characters ✨. Since IDNA is defined using Unicode, and that standard used the term "LATIN" in its character names and descriptions, this document uses that terminology too, except when "Roman-derived" is needed for clarity 💅. But if you're coming at this from a cultural or linguistic angle, just know that "Latin script" in this document refers to the entire collection of Roman-derived characters, not only the characters used to write the Latin language 🫶. Some other script identification issues and relationships with other standards are discussed in [RFC4646] 🦋.
1.5.3. Multilingual 🌈
"multilingual" has lots of wildly different definitions, so it is not recommended for use in standards 🙈. Some definitions are about handling international characters, others are about handling multiple charsets, and still others are about handling multiple languages 💀.
This term has been deprecated for IETF-related uses and doesn't otherwise show up in this document 😭. We still talk about it here, though, because it's still widely used in IDN discussions 🌸.
1.5.4. Localization 💖
localization is the process of adapting an internationalized application platform or application to a specific cultural environment 🎀. In localization, the same semantics are kept while the syntax or presentation forms may change ✨.
Localization is the act of tailoring an application for a different language, script, or culture 🦋. Some internationalized applications can handle a wide variety of languages, but typical users only understand a small number, so the program has to be tailored to talk to users in just the languages they know 👑.
Other groups outside the IETF define localization and internationalization (see below) a bit differently, so check out [W3C-Localization] for one example 💅.
1.5.5. Internationalization 🧚
in the IETF, "internationalization" describes adding or improving the handling of non-ASCII text in a protocol 💜. Other bodies use the term in other ways, often with subtle differences in meaning 🫶. It's often abbreviated "i18n" (and localization as "l10n") 🌈.
Lots of protocols that handle text only handle the characters for one script (often a subset of the characters used in writing English text), or they leave the question of which character set is used to local guesswork, which causes interoperability problems 😭. Adding non-ASCII text to such a protocol lets it handle more scripts, with the goal of including every script that's useful in the world 🌸. It is naive (sic) to believe that all English words can be written in ASCII, no matter what mythologies say 💀🎀.
1.6. Statements and Guidelines 🌸
when the IDNA RFCs were published, the IESG and ICANN made statements to guide deployment and future work 🦋. ICANN has since updated its statement, and others have made contributions too 💖. Honestly, understanding of internationalization issues as applied to the DNS has evolved a LOT over the last few years 🔥. Organizations that took specific positions a year or more ago might not make exactly the same statements today 👀.
1.6.1. IESG Statement 💜
The IESG made a statement on IDNA [IESG-IDN]:
IDNA, through its requirement of Nameprep [RFC3491], uses equivalence tables that are based only on the characters themselves; no attention is paid to the intended language (if any) for the domain name. However, for many domain names, the intended language of one or more parts of the domain name actually does matter to the users.
Similarly, many names cannot be presented and used without ambiguity unless the scripts to which their characters belong are known. In both cases, this additional information should be of concern to the registry.
the statement is longer than this, but these paragraphs are the important ones 🫶. The rest is explanations and examples 🧚.
1.6.2. ICANN Statements 🌈
1.6.2.1. Initial ICANN Guidelines 🎀
soon after the IDNA standards were adopted, ICANN produced an initial version of its "IDN Guidelines" [ICANNv1] ✨. It had two jobs 💅. The first was to provide a basis for releasing the Generic Top Level Domain (gTLD) registries that ICANN had set up from a contractual restriction on registering labels with hyphens in the third and fourth positions 🦋. The second was to provide a general framework for developing registry policies for implementing IDNs 👑.
One of the key parts of this framework required strict compliance with RFCs 3490, 3491, and 3492 🌸. With the framework, ICANN said IDNA was the only mechanism to be used in the DNS to represent IDNs 💖.
Limits on which characters could go into IDNs came from two mechanisms 🔥. The first was an "inclusion-based approach (meaning that code points that are not explicitly permitted by the registry are prohibited) for identifying permissible code points from among the full Unicode repertoire." 💜 The second required every IDN to be associated with a specific language, and additional policies were also language based 🫶:
"In implementing the IDN standards, top-level domain registries will (a) associate each registered internationalized domain name with one language or set of languages, (b) employ language-specific registration and administration rules that are documented and publicly available, such as the reservation of all domain names with equivalent character variants in the languages associated with the registered domain name, and, (c) where the registry finds that the registration and administration rules for a given language would benefit from a character variants table, allow registrations in that language only when an appropriate table is available. ... In implementing the IDN standards, top-level domain registries should, at least initially, limit any given domain label (such as a second-level domain name) to the characters associated with one language or set of languages only."
It was left to each TLD registry to define the character repertoire it would associate with any given language 🧚. That led to big variation from registry to registry, plus more differences in the underlying language-based IDN policies 😭. If the guidelines had made room for IDN policies based on script too, a lot of the resulting ambiguity could have been avoided 💀. They didn't, so the sequence of events leading to the present review of IDNA got triggered 🌈✨.
1.6.2.2. ICANN Version 2 Guidelines ✨
okay so the TLD registries were basically in full panic mode 😭, and one of their moves was to use the update process from the initial guidelines. That process said: "As the deployment of IDNs proceeds, ICANN and the IDN registries will review these Guidelines at regular intervals, and revise them as necessary based on experience." 🦋
The big must-have was changing the guidelines so they could allow script-based policies for IDNs 💅. People also worried about how registry policies would actually get passed down into the lower levels of their name trees 🌳, which is not a small ask. Beyond the expected tightening at the protocol level, one obvious idea was to swap the guidelines for a document with clear standing in the IETF's normative framework 👑. That's where a BCP came in as the right home for the longer-term work. The urgent stuff would get handled in the meantime with incremental edits to the guidelines, but once those edits were done, nobody saw a need for much more detailed development of the guidelines themselves 🎀.
The result was version 2.0 of the guidelines [ICANNv2] 🌸. The ICANN Board endorsed it on November 8, 2005, for a period of nine months. The Board also said it "tasks the IDN working group to continue its important work and return to the board with specific IDN improvement recommendations before the ICANN Meeting in Morocco" and "supports the working group's continued action to reframe the guidelines completely in a manner appropriate for further development as a Best Current Practices (BCP) document, to ensure that the Guideline directions will be used deeper into the DNS hierarchy and within TLD's where ICANN has a lesser policy relationship." Iconic behaviour from the Board, honestly 🙌.
Version 2.0 kept the inclusion-based approach from version 1.0, and the big new addition to the policy framework is this:
"All code points in a single label will be taken from the same script as determined by the Unicode Standard Annex #24: Script Names at http://www.unicode.org/reports/tr24. Exception to this is permissible for languages with established orthographies and conventions that require the commingled use of multiple scripts. In such cases, visually confusable characters from different scripts will not be allowed to coexist in a single set of permissible codepoints unless a corresponding policy and character table is clearly defined."
It also adds:
"Permissible code points will not include: (a) line symbol-drawing characters (as those in the Unicode Box Drawing block), (b) symbols and icons that are neither alphanumeric nor ideographic language characters, such as typographic and pictographic dingbats, (c) characters with well-established functions as protocol elements, (d) punctuation marks used solely to indicate the structure of sentences."
Some points still weren't properly covered (if at all) in version 2.0, but they ought to go into the policy framework without waiting for a "best practices" document to get produced and released 🧐. Heads up: the term "BCP" above doesn't necessarily mean an IETF consensus document 🫶.
In November 2005, the plan was to hand the recommended major revision to the ICANN Board before its Morocco meeting (late June 2006). The changes would be collected bit by bit and shipped in interim version 2.n releases of the guidelines 💜. The IAB's understanding is that there's been some progress, but other IDN issues came along and sucked up a lot of the energy that was supposed to go into the bigger rewrite 😭.
2. General Problems and Issues 🌈
This section is a mix of problem types 🧩. Each subsection describes something perceived as a problem or issue "with IDNs" that needs fixing. Some of these can be at least partly solved by changing parts of the IDNA protocol or its tables 🛠️. Others will stick around for as long as people expect IDNs to behave in ways that clash with the basic DNS architecture 🙈. It matters to map out the whole range, because users, registrants, and policy makers often don't understand the protocol or the technical stuff. They only see the gap between what they think happens (or should happen) and what really happens 👀. As long as that gap exists, people will keep asking for functionality or policy changes for IDNs. Some of those asks are less realistic than others, but even the realistic ones need to be understood in the same context as the rest 💅.
Most of the issues discussed here exist whether IDNA stays tied to Unicode 3.2 or whether a move to newer Unicode versions is on the table ✨. A migration path is needed to cover newly-coded scripts and to let the largest possible number of languages and scripts show up in domain names 🦋. Still, the migration issues are mostly separate from the ones about a single Unicode version or Version 3.2 specifically, so they got their own spot in this section and in Section 3 🎀.
2.1. User Conceptions, Local Character Sets, and Input issues 💭
DNS labels are just strings of characters, and they aren't inherently tied to any particular language 🌸. As mentioned briefly in the Introduction, DNS labels that couldn't be words in any language are possible, and they're honestly very common 🙃. There's no good reason to put protocol restrictions on IDNs that are tighter than what all-ASCII hostname labels have had to deal with 💅. That's also why calling DNS labels, or strings of them, "names" is kind of a misnomer 😬, and it's probably part of why users are confused about what to expect.
People normally use "words" when they're talking about things and want others to picture them too, like "orange", "tree", "restaurant" or "Acme Inc" 🍊. Words usually belong to a specific language, like English or Swedish 🇸🇪. As we just said, the character-string labels the DNS supports aren't inherently "words" 🧚. Using real words as DNS labels is useful, especially for mnemonic value or for identifying objects 💖, but other DNS constraints make it impossible to guarantee that every word in every language can be represented as a DNS label, internationalized or not 😭.
When you write or type a label (or word), you have to pick a script and a charset to go with it ⌨️. The user typically doesn't get to choose the charset per word or per document 🙅♀️. It depends on local input devices, keyboard or terminal drivers, or other decisions made by operating system or even hardware designers and implementers 🖥️.
If that charset (or the local charset used by the relevant operating system or application software) isn't Unicode, you need an extra conversion step to get to Unicode 🔁. How often this comes up depends on how widely Unicode is deployed as the native character set for hardware, operating systems, and applications 📊. The estimates vary a lot, but among other difficulties, note that:
- ISO 8859 versions [ISO.8859.2003] and even national variations of ISO 646 [ISO.646.1991] are still widely used in parts of Europe 🇪🇺;
- code-table switching methods, typically based on the techniques of ISO 2022 [ISO.2022.1986], are still in general use in many parts of the world, especially in Japan with Shift-JIS and its variations; and
- computing, systems, and communications in China tend to use one or more of the national "GB" standards rather than native Unicode 🐼.
Also, not all charsets define their characters the same way, and not every older coding system was brought into Unicode without changes 🤯. Sometimes local distinctions were made that Unicode doesn't make, or the other way around 🧩. So converting from other systems to Unicode can potentially lose information 😱.
The Unicode string that comes out of this processing is then used as input to IDNA 🦋. In a Unicode-native system that step is trivial, but in others it can be a big deal ✨.
2.2. Examples of Issues 🔍
A lot of the discussion below is framed in terms of Unicode codings and their rules 💅. Still, the IAB thinks some of these issues aren't really about the Unicode character set itself 🙃. They're about how distributed matching systems actually operate, and about what the DNS's distributed, delayed search for stored data implies for the mapping algorithms 🌐.
2.2.1. Language-Specific Character Matching 🇳🇴🇸🇪
Some words that look similar can be written in multiple languages 💜. Take the name Torbjorn in Norwegian and Swedish, for example 🧸. In Norwegian, the second syllable is spelled with U+00F8 (LATIN SMALL LETTER O WITH STROKE), while in Swedish it's spelled with U+00F6 (LATIN SMALL LETTER O WITH DIAERESIS) 🎀. The Unicode Standard and its Annexes don't treat those characters as equivalent 😤, even though most people who speak Swedish, Danish, or Norwegian probably think they are equivalent 🥹.
Making these characters equivalent worldwide is neither possible nor desirable 🙅♀️. For this example, it would sort out the situation in Sweden but cause a lot of confusion in Germany, because U+00F8 is never used in the German language 🇩🇪. The "variant" model introduced in [RFC3743] and [RFC4290] can help a registry prevent the worst of that confusion 🛡️. It does this by making sure either that both names are registered to the same party in a given domain, or that one of them is completely prohibited 🔒.
2.2.2. Multiple Scripts 🌍
Some languages can be written in more than one script 🌏. For instance, some Eastern European and Central Asian languages can be written in either Cyrillic or Latin (see Section 1.5.2) characters, and some African and Southeast Asian languages can be written in either Arabic or Latin characters 🔤. A few languages can even be written in three different scripts 🤹♀️. In other cases, the language is normally written with a mix of scripts (e.g., Kanji, Kana, and Romaji for Japanese; Hangul and Hanji for Korean) 🇯🇵🇰🇷. So the same word, in the same language, can be written in different ways 😵💫. For some languages, a single word is normally written in just one script; for others, mixed scripts are required; and for still others, special circumstances may call for mixing scripts in labels, even though that isn't normally done for "words" 🧠. For IDN purposes, all these variations make the definition of "script" super sensitive 🫠, especially since ICANN is now recommending that it be the primary basis for registry policies 📜. Mixed-script labels are essential to prohibit, but there also needs to be extra policy nuance for "languages with established orthographies and conventions that require the commingled use of multiple scripts" 💖.
2.2.3. Normalization and Character Mappings ✨
okay so Unicode has, like, a bunch of different ways of representing characters 💅 The Chinese (Han)-derived characters in the "CJK" (Chinese, Japanese, and Korean) languages are "unified", which means characters with a common origin and similar looks get the same code point 🎀. European characters from a Greek-Latin base are split into separate blocks for Latin, Greek, and Cyrillic, even when individual characters are identical in both form and meaning 🌸. Separate code points based only on font differences are generally prohibited 🙅♀️, but a huge pile of "mathematical" characters got their own code points anyway, even though they only differ from the basic ASCII characters by font attributes like "script", "bold", or "italic" 💀. Some characters that often show up together are treated as typographical digraphs with a specific code point for the combo 🦋, others need the two-character sequence, and some are available in both forms 🫶. Some Roman-derived letters with decorations (like diacritical marks) get individual code points, while others have to be built up as two or more character sequences using "combining characters" ✨.
Most of these differences exist because Unicode wanted to stay backward compatible while the standard evolved over time, so honestly, it makes sense 🥹. BUT the DNS needs precise knowledge of which codes and code sequences represent the same character and which ones don't 🧠. Limiting the trouble with confusable characters (see Section 2.2.6) needs even more knowledge about which characters might look alike in some fonts but not others 👀. These variations make it hard or impossible to use one set of rules for all of Unicode that makes everyone happy 🙃. So instead, more or less complex mapping tables, defined character by character, are required to "normalize" different representations of the same character into a single form so that matching is possible 🧚.
Unless normalization rules, like the ones underneath Nameprep, are applied, characters that are basically identical will NOT match in the DNS, and that opens the door to a ton of problems 😭. The most common one is that, because of the processing that happens (we talked about it above) before a word becomes a Unicode string, a single word can end up as several different Unicode strings 💀. Even with normalization applied, some strings that users consider identical still won't compare equal 🙈. That problem gets a deeper dive elsewhere in this document, especially in Section 3.2.1 🔍.
IDNA tries to fix all this by using a normalization algorithm defined by the Unicode Consortium 🌈. This algorithm can change a sequence of one or more Unicode characters into a different set of characters 💜. For example, the base character U+0061 (LATIN SMALL LETTER A) followed by U+0308 (COMBINING DIAERESIS) becomes the single character U+00E4 (LATIN SMALL LETTER A WITH DIAERESIS) 🍓.
This Unicode normalization process only handles simple character equivalences, NOT the ones that depend on language or script 🌍. Like we said above, U+00F8 (LATIN SMALL LETTER O WITH STROKE) and U+00F6 (LATIN SMALL LETTER O WITH DIAERESIS) are considered to match in Swedish (and some other languages), but NOT for every language that uses either one 🇸🇪. Treating these characters as equal in some contexts and not others needs decisions and mechanisms that depend way more on context than IDNA or the Unicode character-based normalization tables can give 🫠.
It gets even messier when the sequences are more complicated or when an attacker is deliberately trying to confuse the normalization process 😈. For example, if U+0069 U+0307 (LATIN SMALL LETTER I followed by COMBINING DOT ABOVE) shows up, the Unicode Normalization Method known as NFKC maps it to U+00EF (LATIN SMALL LETTER I WITH DIAERESIS), which is what you'd predict 🤓. But wait, consider U+0131 U+0308 (LATIN SMALL LETTER DOTLESS I and COMBINING DIAERESIS): is that the same character? And is U+0131 U+0307 U+0307 (dotless i and two combining dot-above characters) equivalent to U+00EF or U+0069, or to neither? NFKC doesn't seem to tell us, and the definition of U+0307 doesn't seem to say what happens when it gets combined with other "symbol above" arrangements 😵 (unlike some of the "accent above" combining characters, which more or less specify kerning). Similar questions come up when U+00EF is combined with various dot-above combining characters 🫣. Each of these questions creates chances for spoofing if different display implementations read the rules in different ways 🚨.
If we leave the Latin scripts and look at the ones based on Chinese characters, there's also a lack of specific, lexigraphic, rules for changing between Traditional and Simplified Chinese 📚. Even if those rules existed, the unification of Japanese and Korean characters with Chinese ones would make it impossible to normalize Traditional Chinese into Simplified Chinese without causing problems for the Japanese and Korean use of the same characters 🥲.
More generally, some mappings, like the ones between precomposed Latin script characters and the equivalent multiple code point composed character sequences, depend only on the characters themselves 🧩. But in many or most cases, like the Swedish example above, the mapping depends on the language or culture 🌏. People have discussed whether different canonicalization rules (in addition to or instead of Unicode normalization) should be, or could be, applied differently to different languages or scripts 💭. The fact that most scripts in Unicode were first brought in by copying an existing standard more or less intact has an impact on how these algorithms are optimized and on forward compatibility 🎀. Even when the language is known and language-specific rules can be defined, the dependencies on the language don't go away 😤. Canonicalization operations aren't possible unless they either depend only on short sequences of text or have significant context available that isn't obvious from the text itself 🙃. DNS lookups and many other operations don't have a way to capture and use the language or other information needed to provide that context 💔.
These differences between languages and user perceptions of characters make it hard or impossible to provide uniform algorithms for matching Unicode strings in a way where no end user is ever surprised by the result 😱. For closely-related scripts or characters, the surprises might even be frequent 🫠. HOWEVER, because uniform algorithms are required for the mappings applied when names are looked up in the DNS, the rules that get chosen will always be an approximation, more or less successful at minimizing those user surprises 🎯. The current Nameprep and Stringprep algorithms use mapping tables to "normalize" different representations of the same text into a single form so that matching is possible 💖.
You can find more details on how the normalization algorithms were created in the Unicode Specification and the associated Technical Reports [UTR] and Annexes 📖. Technical Report #36 [UTR36] and [UTR39] are specifically related to the IDN discussion 🌟.
2.2.4. URLs in Printed Form 🖨️
URLs and other identifiers show up not only in electronic forms that can (at least in principle) be copied accurately and "pasted", but also in printed forms that the user has to transcribe into the computer system 📝. This is often called the "side-of-the-bus problem" 🚌, because a particularly nasty version of it requires the user to observe and accurately remember a URL that they only glimpsed in a fleeting form: a billboard seen while driving, a sign on the side of a passing vehicle, a TV ad that isn't repeated often or on screen for very long, and so on 🏎️.
The difficulty, in short, is that two Unicode strings that are actually different might look exactly the same, especially when there's no time to study them 👀. This happens because, for example, some glyphs in Cyrillic, Greek, and Latin do look the same, but they've been assigned different code points in Unicode 🤯. Worse, you need to be reasonably familiar with a script and how it's used to understand how much characters can reasonably vary because of artistic fonts and typography 🎨. For example, there are a few fonts for Latin characters that are so highly ornamented that an observer might easily confuse some of the characters with characters in Thai script 😭. Uppercase ITC Blackadder (a registered trademark of International Typeface Corporation) and Curlz MT are two pretty obvious examples 👑; these fonts use loops at the ends of serifs, which creates a resemblance to Thai (in some fonts) for some characters 🙈.
2.2.5. Bidirectional Text ➡️⬅️
Some scripts (and because of that, some words in some languages) are written not left to right, but right to left 🔄. And to make things more complicated, you might have something written in Arabic script, read right to left, that includes some characters that are read from left to right, like European-style digits 🤪. This means some texts might have a mixed left-to-right AND right-to-left order (even though in most implementations, and in IDNA, all texts have a major direction, with the other being the exception) 🌈.
IDNA allows European digits in a label that is otherwise a sequence of right-to-left characters, but it prohibits most other mixed-directional (or bidirectional) strings 🚫. This prohibition can cause other problems, like rejecting some strings that are otherwise linguistically and culturally sensible 💔. As Unicode and the conventions for handling so-called bidirectional ("BIDI") strings evolve, the prohibition in IDNA should be reviewed and reevaluated 🔍✨.
2.2.6. Confusable Character Issues 🎀
Okay so lookalike characters are a whole thing, bestie 😭 They can cause real problems on the Internet, because people can get sent to the wrong host or mailbox, deliberately or by accident, since they think they're typing or clicking on one character but the domain name or reference actually has a different one 💀 See Section 4.1.3 for more on this.
IDNs make it even messier 🌸 They add a ton of extra characters that look alike enough to get mixed up, and they also bring new policy questions. Like, if a language can be written in two different scripts, is a label made from a word in one script the same as a label made from that same word in the other script? And is the answer the same for two words in different languages that translate into each other? 🤔
It's generally understood now that, besides the collision problem with words that might be equivalent (and so labels that might be equivalent), you can use look-alike characters, "confusable" characters, to spoof names and mislead or scam users 🚨 That's the phishing problem basically, and it means registries have to work harder to prevent these issues than people used to realise 🫶
One commonly proposed approach is for the registry to set restrictions on which characters, and which combinations of characters, it will allow in a string being registered as a label. Using the Swedish top-level domain, .SE, as an example, a rule might say the registry "only accepts registrations in Swedish, using Latin script, and because of this, Unicode characters Latin-a, -b, -c,...". But there's no 1:1 mapping between country and language, so even a Country Code Top Level Domain (ccTLD) like .SE might have to accept registrations in other languages ✨ For example, there might be a requirement for Finnish (the second most-used language in Sweden). So what rules and code points get defined for Finnish? Does it have special mappings that collide with the ones defined for Swedish? And what about countries that use more than one script? (Finnish and Swedish use the same script.) In every case, the fight is ultimately about whether two strings are the same (or confusingly similar). That leads to the bigger question of how you define "what is the same" and "what is similar enough to be a problem" 💅
Another example came up recently 👀 If someone used Cyrillic characters to write the country code for Russia in a localized version of the ccTLD label, those characters would be indistinguishable from the Latin characters "P" and "Y" (in either lowercase or uppercase) in most fonts. We presume this might cause some consternation in Paraguay 💀
These difficulties can never be completely eliminated by algorithmic means 😭 Some of the problem can be addressed by tuning the protocols and their tables, some by registry actions to reduce confusion and conflicts, and some by careful design of user interfaces in application programs. But ultimately, some responsibility to avoid being tricked or harmfully confused will rest with the user 🫶
Another registry technique that's been explored a lot looks at confusable characters and confusion between complete labels, restricting the labels that can be registered based on what's already registered. Registries that take this approach might establish special mapping rules such as:
- If you register something with code point A, domain names with B instead of A will be blocked from registration by others (where B is a character at a separate code point that has a confusingly similar appearance to A).
- If you register something with code point A, you also get domain name with B instead of A.
These approaches are discussed in more detail for "CJK" characters in RFC 3743 [RFC3743] and more generally in RFC 4290 [RFC4290] 📚
2.2.7. The IESG Statement and IDNA issues 🧚
The issues above, at least as they were understood at the time, were the background for the IESG statement in Section 1.6.1 (which, in turn, was part of the basis for the initial ICANN Guidelines). That statement says a registry should have a policy about the scripts, languages, code points and text directions for which registrations will be accepted 📜 "Accept all" might be an acceptable policy, but it implies there's also a dispute resolution process that takes the problems listed above into account. That process has to be designed for dealing with all types of potential disputes 💜 For example, issues might come up between a registrant and a registry over a registry decision about collisions with already registered domain names, or between a registrant and a trademark holder (that a domain name infringes on a trademark). In both cases, the parties disagreeing have different views on whether two strings are "equivalent" or not. They might believe a string that isn't allowed to be registered is actually different from one that's already registered. Or they might believe two strings are the same, even though the rules the registry adopted to prevent confusion define them as two different domain names 🙈
3. Migrating to New Versions of Unicode 🦋
3.1. Versions of Unicode 🌈
Opinions differ about how important these issues are in practice, but IDNA's use of Unicode and its supporting tables seems to be way more sensitive to subtle changes than typical Unicode applications are 😳 That might be partly because lots of other applications are internally sensitive only to how characters look and not to how they're represented. Or those applications may be able to take real advantage of script, language, or character class identification. The working group that developed IDNA concluded that trying to encode any ancillary character information into the DNS label would be impractical and unwise, and the IAB, partly based on the comments from the ad hoc committee, saw no reason to review that decision 🫶
The Unicode Consortium has sometimes used how likely a combination of characters is to actually appear in a natural language as a test for whether a possible change is safe. But, as discussed above, DNS names are often fabrications: abbreviations, strings deliberately formed to be unusual, members of a series sequenced by numbers or other characters, and so on 🔥 So a test that considers a change safe if it wouldn't be visible in properly-constructed running text isn't helpful for DNS purposes. A change that's safe under that test could still be quite problematic for the DNS 💀
This sensitivity to changes makes it really hard to migrate IDNA from one version of Unicode to the next if any changes aren't strictly additive 😭 A change in a code point assignment or definition can be extremely disruptive if a DNS label was defined using the earlier form and any of its previous components has been moved from one table position or normalization rule to another. Unicode normalization tables, tables of scripts or languages and the characters that belong to them, and even tables of confusable characters (as an add-on to security recommendations) may be really helpful for designing registry restrictions on registrations and for application provisions that avoid or identify suspicious names. Ironically, they also extend the sensitivity of IDNA and its implementations to all forms of change between one version of Unicode and the next. So they make Unicode version migration harder 🙃
An example of a change that looks like a small correction from one perspective but can be problematic from another was the correction to the normalization definition in 2004 [Unicode-PR29]. Community input suggested the change would cause problems for Stringprep, but the Unicode Technical Committee decided, on balance, that the change was worth it. Because of difficulties with consistency, some deployed implementations adopted the change and others didn't, which led to subtle incompatibilities 👀
This leads to a real dilemma 🥺 On one hand, it's completely unacceptable to freeze IDNA at a Unicode version that excludes more recently-defined characters and scripts that matter to the people who use them. On the other hand, it's equally unacceptable to migrate from one version of Unicode to the next if that migration might invalidate an existing registered DNS name or some of its registered properties, or might make the string or representation of that name ambiguous. If IDNA is going to be modified to accommodate new versions of Unicode, the IETF will need to work with the Unicode Consortium and other bodies to find an appropriate balance here. But progress will only be possible if all the relevant parties can fairly consider and discuss decisions that may be very difficult and unpalatable 🌸
It would also be useful if, during that dialog, the need for Unicode Consortium concern with security issues in applications of the Unicode character set could be clarified. From almost every perspective considered here, it would be unfortunate if such matters slowed the inclusion of as yet unencoded scripts ✨
3.2. Version Changes and Normalization Issues 💖
3.2.1. Unnormalized Combining Sequences 🧩
okay so one of the big perks of Unicode's combining character model (same energy as the old character overstriking tricks) is that you can stack code points together to make characters that never got their own spot in the character set 🎀. but here's the tea: unless something blocks sequences that weren't explicitly provided for (like the normalization tables can), those combining sequences can cause two related dangers 😬
- The first is another risk of character confusion, especially if the relationship between a combining character and the characters it combines with isn't precisely defined, or if unexpected combinations of combining characters get used. We dig into that more, with an example, in Section 2.2.3 🔍
- These same issues also inherently impact the stability of the normalization tables 💀. Picture this: somewhere in the world there's a character that looks like a Roman-derived lowercase "i", but with three (not one or two) dots above it 🙈. Now say the users of that character agree to represent it by combining a traditional
i(U+0069) with a combining diaeresis (U+0308). So far, no problem ✨. But later, a broader need for this character is discovered, and it gets coded into Unicode either as a single precomposed character or, more likely under the existing rules, by introducing a three-dot-above combining character. In either case, that version of Unicode should include a rule in NFKC that maps the "i"-plus-diaeresis sequence into the new, approved, one. If one doesn't do that, then there is arguably a normalization that should occur but doesn't. If one does do that, then strings that were valid and normalized (even if nobody anticipated them) under the previous versions of Unicode become unnormalized under the new version 😭. That, in turn, would impact IDNA comparisons, because effectively it would introduce a change in the matching rules 🫠
It would be useful to consider rules that avoid or minimize these problems, with the understanding that, for reasons given elsewhere, simply minimizing them may not be good enough for IDNA 🧐. One partial solution might be to ban any combination of a base character and a combining character that doesn't appear in a hypothetical "anticipated combinations" table from being used in a domain name label 📋. The next subsection discusses a more radical, if impractical, view of the problem and its solutions 🌈
3.2.2. Combining Characters and Character Components 🧱
For several reasons, including those discussed above, one thing that makes IDNA more complex and increases the need for normalization is that combining characters are permitted 💅. Without them, complexity might be reduced enough to permit easier transitions to new versions. The community should consider the impact of entirely prohibiting combining characters from IDNs 🤔. While it's almost certainly unfeasible to introduce this change into Unicode as it's now defined (and it would be extremely disruptive even if it were feasible), the thought experiment can be helpful for understanding both the issues and the implications of the paths not taken 🦋. For example, one consequence of this, of course, is that each new language or script, and several existing ones, would require that all of its characters have Unicode assignments to specific, precomposed, code points 🍓
Note that this is not currently permitted within Unicode for Latin scripts 🙅♀️. For non-Latin scripts, some such code points have been defined. The decisions that govern the assignment of such code points are managed entirely within the Unicode Consortium 🏛️. Were the IETF to choose to reduce IDNA complexity by excluding combining characters, there would no doubt be additional input to the Unicode Consortium from users and proponents of scripts that precomposed characters be required 📣. The IAB and the IETF should examine whether it's appropriate to press the Unicode Consortium to revise these policies, or otherwise to recommend actions that would reduce the need for normalization and the related complexities 🧐. However, we've been told that the Technical Committee does not believe it's reasonable or feasible to add all possible precomposed characters to Unicode 😬. If Unicode can't be modified to contain the precomposed characters necessary to support existing languages and scripts, much less new ones, then this option for IDN restrictions won't be feasible 💔
3.2.3. When does normalization occur? ⏰
In many Unicode applications, the preferred solution is to pick a style of normalization and require that all text that's stored or transmitted gets normalized to that form 📝. (This is the approach taken in ongoing work in the IETF on a standard Unicode text form [net-utf8] 🫶.) IDNA doesn't impose this requirement. Text is normalized and case-reduced at registration time, and only the normalized version is placed in the DNS 🌸. However, there's no requirement that applications show only the native (and lower-case where appropriate) characters associated with the normalized form in discussions or references such as URLs 👀. If conventions used for all-ASCII DNS labels are to be extended to internationalized forms, such a requirement would be unreasonable, since it would prohibit the use of mixed-case references for clarity or market identification 🏷️. It might even be culturally inappropriate ✨. But without that restriction, the comparison that will ultimately be made in the DNS will be between strings normalized at different times and under different versions of Unicode ⚠️. The assertion that a string in normalized form under one version of Unicode will still be in normalized form under all future versions is not sufficient 🙅♀️. Normalization at different times also requires that a given source string always normalizes to the same target string, regardless of the version under which it's normalized. That criterion is much more difficult to fulfill 😩. The discussion above suggests that it may even be impossible 💀
Ignoring these issues with combining characters entirely, as IDNA effectively does today, may leave us "stuck" at Unicode 3.2 🪤. That could lead either to incompatibility differences in applications that otherwise use a modern version of Unicode (while IDN remains at Unicode 3.2), or to painful transitions to new versions 🥲. If decisions are made quickly, it may still be possible to make a one-time version upgrade to Version 4.1 or Version 5 of Unicode 🚀. However, unless we can impose sufficient global restrictions to permit smooth transitions, upgrading to versions beyond that one is likely to be painful (e.g., potentially requiring changing strings already in the DNS, or even a new Punycode prefix) or impossible 💔
4. Framework for Next Steps in IDN Development 🗺️
4.1. Issues within the Scope of the IETF 🧭
4.1.1. Review of IDNA 🔎
The IETF should consider reviewing RFCs 3454, 3490, 3491, and/or 3492, and update, replace, or supplement them to meet the criteria of this paragraph (one or more of them may prove impractical after further study) 🤷♀️. Any new versions or additional specifications should be adapted to the version of Unicode that's current when they're created. Ideally, they should specify a path for adapting to future versions of Unicode (some suggestions below may facilitate this) 🛤️. The IETF should also consider whether there are significant advantages to mapping some groups of characters, such as code points assigned to font variations, into others, or whether clarity and comprehensibility for the user would be better served by simply prohibiting those characters 🎭. More generally, it appears worthwhile for the IETF to review whether the Unicode normalization rules now invoked by the Stringprep profile in Nameprep are optimal for the DNS, or whether more restrictive rules, or an even more restrictive set of permitted character combinations, would provide better support for DNS internationalization 💖
The IAB has concluded that there is a consensus within the broader community that lists of code points should be specified by the use of an inclusion-based mechanism (i.e., identifying the characters that are permitted), rather than by excluding a small number of characters from the total Unicode set as Stringprep and Nameprep do today 🫧. That conclusion should be reviewed by the IETF community, and action taken as appropriate 📚
We suggest that the individuals doing the review of the code points should work as a specialized design team 👑. To the extent possible, that work should be done jointly by people with experience from the IETF and deep knowledge of the constraints of the DNS and application design, participants from the Unicode Consortium, and other people necessary to reach a generally-accepted result 🤝. Because any work along these lines would be modifications and updates to standards-track documents, final review and approval of any proposals would necessarily follow normal IETF processes 📋
It's worth noting that sufficiently extreme changes to IDNA would require a new Punycode prefix, probably with long-term support for both the old prefix and the new one in both registration arrangements and applications 🔮. An alternative, which is almost certainly impractical, would be some sort of "flag day", i.e., a date on which the old rules are simultaneously abandoned by everyone and the new ones adopted 🗓️. However, preliminary analysis indicates that few, if any, of the changes recommended for consideration elsewhere in this document would require this type of version change 🥳. For example, suppose additional restrictions, such as those implied above, are imposed on what can be registered. Those restrictions might require policy decisions about how labels are to be disposed of if they conformed to the earlier rules but not to the new ones. But they would not inherently require changes in the protocol or prefix 🌷
4.1.2. Non-DNS and Above-DNS Internationalization Approaches ✨
Okay bestie, so the IETF should once again look at how much sense it makes to fix internationalization problems through the DNS 🌐, and what place all the "keyword systems" and other Internet navigational techniques might have 🧚. Those techniques can be designed to impose fewer constraints, or at least different constraints, than IDNA and the DNS do. As discussed elsewhere in this document, IDNA cannot support information about scripts, languages, or Unicode versions on lookup 🙃. Because DNS lookups work the way they do, characters and labels either match or don't match. A near-match is simply not a possible concept in the DNS 💀. Meanwhile, near-matching is super common in human communication and in the matching people do themselves, especially when they have a particular script or language context in mind 🫶.
The DNS is also extra constrained by its fairly rigid internal aliasing system (via CNAME and DNAME resource records) 🎀, and some applications of international naming may need more flexibility. Then there's the rigid hierarchy of the DNS, and the tendency in practice for it to go flat at the levels nearest the root 👑. Add in the need for names to be unique, and you get something that's more suitable for some purposes than others. It may not be a great match for some of the purposes people want IDNs for 😭.
Each of these constraints can be relaxed or changed by one or more systems that provide alternatives to using the DNS directly 🦋. Some of the issues are discussed further in Section 5.3, and various ideas have been talked about in detail in the IETF or IRTF 💅. Many of those ideas have even been written up in Internet Drafts or other documents 📚. As experience with IDNs and with expectations for them builds up, it will probably become appropriate for the IETF or IRTF to revisit these underlying questions and possibilities 🌈.
4.1.3. Security Issues, Certificates, etc. 🔐
Some characters look like others, often because they share common origins 👀. The problem with these "confusable" characters, which often get incorrectly called homographs, has always existed whenever characters are shown to humans who look at what's displayed and then make decisions based on what they see 🙈. This isn't a problem that only exists with internationalized domain names, but IDNs make it worse 💀. A survey explaining what the problems are might be interesting 📝. Lots of these issues are covered in Unicode Technical Report #36 [UTR36].
In this and other IDN issues, precise terminology matters, or even more confusion can result 🧠. The definition of "homograph" that normally shows up in dictionaries and linguistic texts says homographs are different words that are spelled identically. For example, the adjective 'brief' meaning short, the noun 'brief' meaning a document, and the verb 'brief' meaning to inform 📖. By definition, letters in two different alphabets are not the same, no matter how similar they look ✨. So sequences of letters from two different scripts that appear identical on a computer display cannot be homographs in the accepted sense, even if they're both words in the dictionary of some language 🙅♀️. Say there's a language written in Cyrillic script where "cap" is a word. Whatever it means, it is not a homograph of the Latin-script English word "cap" 🧢.
When the security implications of visually confusable characters came to the forefront in 2005, the term homograph got used for any instance of graphic similarity, even when comparing individual characters 😤. That usage is not only incorrect, it risks adding even more confusion, so it should be avoided 🚫. The current preferred terminology is to describe these similar-looking characters as "confusable characters" or even "confusables" 💖.
Lots of people have suggested that confusable characters are a problem that must be addressed, at least partly, directly in the user interfaces of application software 💻. While that should almost certainly be part of a complete solution, it comes with its own set of difficulties 😬. For example, a user switching between systems, or even between applications on the same system, may be surprised by different types of behavior and different levels of protection 🫠. It's also unclear how a secure setup for the end user should be designed 🤔. Today, in the web browser, a padlock is a traditional way of showing some level of security to the end user 🔒. Is this binary signaling enough? Should there be any connection between the risk of a displayed string with confusable characters and the padlock or similar signaling to the user? 👀
Many web browsers have adopted a convention based on a "whitelist" or similar technique. It restricts the display of native characters to subdomains of top-level domains that are deemed to have safe practices for registering potentially confusable labels 🛡️. IDNs in other domains get displayed as Punycode 🧩. These techniques may not be sensitive enough to differences in policies among top-level domains and their subdomains, so while they're clearly helpful, they may not be adequate 🥺. Are other methods of dealing with confusable characters possible? Would other ways of identifying and listing policies about avoiding confusing registrations be feasible and helpful? 🤷♀️
It would be interesting to see a more coordinated effort to establish guidelines for user interfaces 🎀. If nothing else, the current whitelists are browser specific, and they can and do differ between implementations 🙃.
4.1.4. Protocol Changes and Policy Implications ⚖️
Some potential protocol or table changes raise important policy issues about what to do with existing, registered names 📛. If such changes are needed, their impact must be carefully evaluated in the IETF, ICANN, and possibly other forums 🧐. In particular, protocol or policy changes that would not permit existing names to be registered under the newer rules should be considered carefully. That means balancing their importance against possible disruption, and weighing the issues of invalidating older names against the importance of consistency as seen by the user 💭.
4.1.5. Non-US-ASCII in Local Part of Email Addresses 📧
Work is going on in the IETF related to the local part of email addresses 🌍. The local part of email addresses has much different syntax and constraints than a domain name label, so you can't directly apply IDNA to the local part 🚫. It's not possible, bestie 💀.
4.1.6. Use of the Unicode Character Set in the IETF 🌏
Unicode and the closely related ISO 10646 are the only coded character sets that aspire to include all of the world's characters 🌸. Because of that, they let you use international characters without having to identify particular character coding standards or tables ✨. Having a single character set is especially important for use with the DNS, since there's no place to put character set identification 🫶. The decision to use Unicode as the base for future IETF protocols is discussed in [RFC2277]. The IAB doesn't see any reason to revisit the decision to use Unicode in IETF protocols 👑.
4.2. Issues That Fall within the Purview of ICANN 🏛️
4.2.1. Dispute Resolution ⚔️
IDNs create new types of collisions between trademarks and domain names, as well as collisions between domain names themselves 💥. These have an impact on the dispute resolution processes used by registries and others 📋. It's important that IDN deployment evolves in parallel with the review and updating of ICANN or registry-specific dispute resolution processes 🔄.
4.2.2. Policy at Registries 📜
The IAB recommends that registries use an inclusion-based model when choosing which characters to allow at registration time 🎯. That list of characters is in turn supposed to be a subset of what's allowed according to the updated IDNA standard 🧩. The IAB further recommends that registries develop their inclusion-based models in parallel with the dispute resolution process at the registry itself 💅.
Most established policies for dealing with claimed or apparent confusion or conflicts of names are based on dispute resolution ⚖️. Decisions about the legitimate use or registration of one or more names get resolved at or after registration, case by case, using policies specific to the particular DNS zone or jurisdiction involved 🗂️. These policies have generally not been extended below the level of the DNS that the top-level registry directly controls 🙈.
Because of how many conflicts the larger number of available and confusable characters in Unicode can generate, we recommend that registration-restriction and dispute resolution policies be developed to constrain IDN registration, and that zone administrators at all levels of the DNS tree do the same 🌳. Of course, many of these policies will be less formal than others, and there's no requirement for complete global consistency 🤍. Still, the arguments for reducing confusable characters and other issues in TLDs should apply to all zones below that specific TLD 💜.
Consistency across all zones can obviously only be accomplished by changes to the protocols 🔧. Such changes should be considered by the IETF if particular restrictions are identified that are important and consistent enough to be applied globally 🌐.
Some potential protocol changes or changes to character-mapping tables might, if adopted, have profound registry policy implications 🫨. See Section 4.1.4.
4.2.3. IDNs at the Top Level of the DNS 🔝
The IAB has concluded that IDNs at the top level of the DNS (IDN TLDs) aren't one issue, but at least three very separate ones 🦄:
- If IDNs are to be entered in the root zone, decisions must first be made about how these TLDs are to be named and delegated. These decisions fall within the traditional IANA scope and are ICANN issues today 🏷️.
- There has been discussion of permitting some or all existing TLDs to be referenced by multiple labels, with those labels presumably representing some understanding of the "name" of the TLD in different languages 🌺. If actual aliases of this type are desired for existing domains, the IETF may need to consider whether the use of DNAME records in the root is appropriate to meet that need, what constraints, if any, are needed, whether alternate approaches, such as those of [RFC4185], are appropriate, or whether further alternatives should be investigated 🔍. But to the extent that aliases are considered desirable and feasible, decisions presumably must be made about which root IDN labels, if any, should be associated with DNAME records, and which ones should be handled by normal delegation records or other mechanisms 🧭. That decision is one of DNS root-level namespace policy, so it falls to ICANN, although we would expect ICANN to pay careful attention to any technical, operational, or security recommendations that other bodies may produce 🙏.
- Finally, if IDN labels are to be placed in the root zone, there are issues associated with how they are to be encoded and deployed 📦. This area may have implications for work that has been done, or should be done, in the IETF 🌟.
5. Specific Recommendations for Next Steps ✨
okay so, following the framework from above, the IAB has some recommendations 💅 for the groups they're meant for. let's get into it 🫶
5.1. Reduction of Permitted Character List 🎀
basically, take the old "hostname" rules and stretch them to cover non-ASCII characters, but only let in as few characters as possible to get the job done 🧚. that's a restrictive model for IDN labels, which is kind of the opposite of how the original IDNA/Nameprep tables were built. back then the idea was to include every Unicode character there wasn't a clear reason to leave out 😭.
the specific recommendation is to spec out these internationalized hostnames. the IETF would be the one to do that, but making the list of allowed characters is going to take effort both in the IETF and outside it 🌸. it should be as linguistically and culturally sensitive as possible, but smooth, effective DNS operation, including keeping complexity down, should be the top priorities 👑. these are the possible mechanisms for getting to a sensible minimum number of characters:
5.1.1. Elimination of All Non-Language Characters 💖
Unicode characters that aren't needed to write words or numbers in any of the world's languages should get cut from the list of allowed DNS label characters 🙅. that includes stuff like box-drawing characters and sentence punctuation, and it should also exclude punctuation for word structure and other delimiters 🍓. DNS labels can totally be used for words a lot of the time, but the goal isn't to spell out words, sentences, or phrases. it's to make labels that are unambiguous and have good mnemonic value ✨.
5.1.2. Elimination of Word-Separation Punctuation 🦋
the hyphen is in the original hostname rules because of a historical artifact from an older, flat namespace 🙈. the community should think about whether it's right to treat it as a simple legacy property of ASCII names and just not try to generalize it to other scripts. for example, we might not allow claimed equivalents of the hyphen from other scripts to be used in IDNs 💅. we could even consider banning the hyphen itself in non-ASCII strings or, less strictly, in strings that contain non-Latin characters 💀.
5.2. Updating to New Versions of Unicode 🌈
as new scripts keep getting added to Unicode to support new languages, it's super important that IDNA keeps up 📚. if it doesn't, and it stays "stuck" at 3.2 or some single later version, then labels derived from words in languages that need characters only available in later versions just can't go in the DNS 😭. upgrades are hard, and they'll keep being hard as long as new versions need more than added characters. we're talking changes to canonicalization conventions, normalization tables, or matching procedures (see Section 3.1) 🥲. anything that can lower complexity and simplify forward transitions should be seriously considered 🔥.
5.3. Role and Uses of the DNS 🎀
reminder, bestie: the DNS has boundaries 🙅♀️. it was designed and built to serve some specific purposes. it does some things well, some things badly, and some things it can't do at all 💅. no amount of protocol work on IDNs will solve problems with alternate spellings, near-matches, searching for appropriate names, and so on 😩. registration restrictions and carefully designed user interfaces can reduce the risk and pain when those attempts go wrong, and they can also reduce the risks of various kinds of deliberate bad behavior 🛡️. but past a certain point, using the DNS just because it's available becomes a bad tradeoff 🙃. the tradeoff can be especially unfortunate when IDNs don't actually solve the proposed problem 💀. for example, internationalization of DNS names doesn't eliminate the ASCII protocol identifiers and structure of URIs [RFC3986] or even IRIs [RFC3987] 🧃. so DNS internationalization itself, at any or all levels of the DNS tree, is not a sufficient response to people wanting to use the Internet entirely in their own languages and the characters associated with those languages 🌍💜.
these issues are discussed at more length, and alternatives are presented, in [RFC2825], [RFC3467], [INDNS], and [DNS-Choices] 📖.
5.4. Databases of Registered Names 📇
IDNs cause issues in other places domain names get used, not just in the DNS 👀. mainly, the design and content of the databases that bind registered names to info about the registrant (usually called "whois" databases) will need review and updating ✏️. for example, the whois protocol itself [RFC3912] has no standard capability for handling non-ASCII text. you can't consistently search for, or report, a DNS name or contact info that isn't in ASCII characters 😤. that might give some extra push toward switching to IRIS [RFC3981] [RFC3982], but it also raises a bunch of other questions about what information, and in what languages and scripts, should be included or permitted in those databases 🤔.
6. Security Considerations 🚨
okay so this doc is just a discussion of IDNs and IDNA issues, which means it raises no new security concerns 🙌. but, and this is a big but 😳, if some of its recommendations get followed and prove successful (to reduce IDNA complexity, the number of available characters, and various approaches to constraining the use of confusable characters), the risks of name spoofing and other problems may go down 🛡️✨.
7. Acknowledgements 🫶
huge thanks to the IAB-IDN ad hoc committee for their contributions to this report 💖. not every member of that group endorses every comment and suggestion here, though 🙃. and this report does not claim to reflect the views of the Unicode Consortium as a whole, or those of particular participants in that Consortium's work 📣.
the members of the ad hoc committee were: Rob Austein, Leslie Daigle, Tina Dam, Mark Davis, Patrik Faltstrom, Scott Hollenbeck, Cary Karp, John Klensin, Gervase Markham, David Meyer, Thomas Narten, Michael Suignard, Sam Weiler, Bert Wijnen, Kurt Zeilenga, and Lixia Zhang 🌸.
thanks are due to Tina Dam and others associated with the ICANN IDN Working Group for contributing considerable specific text, to Marcos Sanz and Paul Hoffman for careful late-stage reading and extensive comments, and to Pete Resnick for many contributions and comments, both while on IAB service and afterward 🙌. Olaf M. Kolkman took over IAB leadership for this document after Patrik Faltstrom and Pete Resnick stepped down in March 2006 👑.
members of the IAB at the time this document was approved were: Bernard Aboba, Loa Andersson, Brian Carpenter, Leslie Daigle, Patrik Faltstrom, Bob Hinden, Kurtis Lindqvist, David Meyer, Pekka Nikander, Eric Rescorla, Pete Resnick, Jonathan Rosenberg and Lixia Zhang 🌟.
8. References 📚
8.1. Normative References 🎀
[ISO10646] International Organization for Standardization,
"Information Technology - Universal Multiple-
Octet Coded Character Set (UCS) - Part 1:
Architecture and Basic Multilingual Plane"",
ISO/IEC 10646-1:2000, October 2000.
[RFC3454] Hoffman, P. and M. Blanchet, "Preparation of
Internationalized Strings ("stringprep")",
RFC 3454, December 2002.
[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello,
"Internationalizing Domain Names in Applications
(IDNA)", RFC 3490, March 2003.
[RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A
Stringprep Profile for Internationalized Domain
Names (IDN)", RFC 3491, March 2003.
[RFC3492] Costello, A., "Punycode: A Bootstring encoding of
Unicode for Internationalized Domain Names in
Applications (IDNA)", RFC 3492, March 2003.
[Unicode32] The Unicode Consortium, "The Unicode Standard,
Version 3.0", 2000.
(Reading, MA, Addison-Wesley, 2000. ISBN
0-201-61633-5). Version 3.2 consists of the
definition in that book as amended by the Unicode
Standard Annex #27: Unicode 3.1
(http://www.unicode.org/reports/tr27/) and by the
Unicode Standard Annex #28: Unicode 3.2
(http://www.unicode.org/reports/tr28/).8.2. Informative References ✨
okay bestie, this is the reading list 📚 these are the "for context" references, so they're not required to implement the thing, but they're where the whole story comes from 🦋 every single one is listed below, exactly as written, so nothing gets lost 💅
[DNS-Choices] Faltstrom, P., "Design Choices When Expanding
DNS", Work in Progress, June 2005.
[ICANNv1] ICANN, "Guidelines for the Implementation of
Internationalized Domain Names, Version 1.0",
March 2003, <http://www.icann.org/general/
idn-guidelines-20jun03.htm>.
[ICANNv2] ICANN, "Guidelines for the Implementation of
Internationalized Domain Names, Version 2.0",
November 2005, <http://www.icann.org/general/
idn-guidelines-20sep05.htm>.
[IESG-IDN] Internet Engineering Steering Group (IESG), "IESG
Statement on IDN", IESG Statements IDN Statement,
February 2003, <http://www.ietf.org/IESG/
STATEMENTS/IDNstatement.txt>.
[INDNS] National Research Council, "Signposts in
Cyberspace: The Domain Name System and Internet
Navigation", National Academy Press ISBN 0309-
09640-5 (Book) 0309-54979-5 (PDF), 2005, <http://
www7.nationalacademies.org/cstb/pub_dns.html>.
[ISO.2022.1986] International Organization for Standardization,
"Information Processing: ISO 7-bit and 8-bit
coded character sets: Code extension techniques",
ISO Standard 2022, 1986.
[ISO.646.1991] International Organization for Standardization,
"Information technology - ISO 7-bit coded
character set for information interchange",
ISO Standard 646, 1991.
[ISO.8859.2003] International Organization for Standardization,
"Information processing - 8-bit single-byte coded
graphic character sets - Part 1: Latin alphabet
No. 1 (1998) - Part 2: Latin alphabet No. 2
(1999) - Part 3: Latin alphabet No. 3 (1999) -
Part 4: Latin alphabet No. 4 (1998) - Part 5:
Latin/Cyrillic alphabet (1999) - Part 6: Latin/
Arabic alphabet (1999) - Part 7: Latin/Greek
alphabet (2003) - Part 8: Latin/Hebrew alphabet
(1999) - Part 9: Latin alphabet No. 5 (1999) -
Part 10: Latin alphabet No. 6 (1998) - Part 11:
Latin/Thai alphabet (2001) - Part 13: Latin
alphabet No. 7 (1998) - Part 14: Latin alphabet
No. 8 (Celtic) (1998) - Part 15: Latin alphabet
No. 9 (1999) - Part 16: Part 16: Latin alphabet
No. 10 (2001)", ISO Standard 8859, 2003.
[RFC2277] Alvestrand, H., "IETF Policy on Character Sets
and Languages", BCP 18, RFC 2277, January 1998.
[RFC2825] IAB and L. Daigle, "A Tangled Web: Issues of
I18N, Domain Names, and the Other Internet
protocols", RFC 2825, May 2000.
[RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001.
[RFC3467] Klensin, J., "Role of the Domain Name System
(DNS)", RFC 3467, February 2003.
[RFC3536] Hoffman, P., "Terminology Used in
Internationalization in the IETF", RFC 3536,
May 2003.
[RFC3743] Konishi, K., Huang, K., Qian, H., and Y. Ko,
"Joint Engineering Team (JET) Guidelines for
Internationalized Domain Names (IDN) Registration
and Administration for Chinese, Japanese, and
Korean", RFC 3743, April 2004.
[RFC3912] Daigle, L., "WHOIS Protocol Specification",
RFC 3912, September 2004.
[RFC3981] Newton, A. and M. Sanz, "IRIS: The Internet
Registry Information Service (IRIS) Core
Protocol", RFC 3981, January 2005.
[RFC3982] Newton, A. and M. Sanz, "IRIS: A Domain Registry
(dreg) Type for the Internet Registry Information
Service (IRIS)", RFC 3982, January 2005.
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter,
"Uniform Resource Identifier (URI): Generic
Syntax", STD 66, RFC 3986, January 2005.
[RFC3987] Duerst, M. and M. Suignard, "Internationalized
Resource Identifiers (IRIs)", RFC 3987,
January 2005.
[RFC4185] Klensin, J., "National and Local Characters for
DNS Top Level Domain (TLD) Names", RFC 4185,
October 2005.
[RFC4290] Klensin, J., "Suggested Practices for
Registration of Internationalized Domain Names
(IDN)", RFC 4290, December 2005.
[RFC4645] Ewell, D., "Initial Language Subtag Registry",
RFC 4645, September 2006.
[RFC4646] Phillips, A. and M. Davis, "Tags for Identifying
Languages", BCP 47, RFC 4646, September 2006.
[UTR] Unicode Consortium, "Unicode Technical Reports",
<http://www.unicode.org/reports/>.
[UTR36] Davis, M. and M. Suignard, "Unicode Technical
Report #36: Unicode Security Considerations",
November 2005, <http://www.unicode.org/draft/
reports/tr36/tr36.html>.
[UTR39] Davis, M. and M. Suignard, "Unicode Technical
Standard #39 (proposed): Unicode Security
Considerations", July 2005, <http://
www.unicode.org/draft/reports/tr39/tr39.html>.
[Unicode-PR29] The Unicode Consortium, "Public Review Issue #29:
Normalization Issue", Unicode PR 29,
February 2004.
[Unicode10] The Unicode Consortium, "The Unicode Standard,
Version 1.0", 1991.
[W3C-Localization] Ishida, R. and S. Miller, "Localization vs.
Internationalization", W3C International/
questions/qa-i18n.txt, December 2005.
[net-utf8] Klensin, J. and M. Padlipsky, "Unicode Format for
Network Interchange", Work in Progress,
April 2006.Authors' Addresses 🎀
here's who wrote it, so you know who to thank (or politely message) 🫶 contact details are copied exactly below
John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA 02140
USA
Phone: +1 617 491 5735
EMail: john-ietf@jck.com
Patrik Faltstrom
Cisco Systems
EMail: paf@cisco.com
Cary Karp
Swedish Museum of Natural History
Box 50007
Stockholm SE-10405
Sweden
Phone: +46 8 5195 4055
EMail: ck@nrm.museum
IAB
EMail: iab@iab.orgFull Copyright Statement 🌈
the legal bit, kept word for word so nobody gets confused 🙈
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.Intellectual Property 💜
more legal fine print, also verbatim 💅
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.Acknowledgement 🧚
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).and that's the whole document, bestie!! 🌈✨ thank u for being the most patient reading buddy 🫶💖
📦 where this page comes from
- ietf: rfc4690 2006-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms