RFC 5895

Mapping Characters for Internationalized Domain Names in Applications (IDNA) 2008

📅 published September 2010, from ietf

RFC 5895: Mapping Characters for Internationalized Domain Names in Applications (IDNA) 2008 ✨

Stream: Independent Submission 🌸 RFC number: 5895 Authors: P. Resnick (Qualcomm Incorporated) and P. Hoffman (VPN Consortium) 👯 Category: Informational 💖 ISSN: 2070-1721 Date: September 2010 📅 Obsoletes / Updates: the header doesn't list either one 🦋

Abstract

okay so back in the original IDNA protocol 🌈, any Unicode code points that came from user input got mapped into a set of code points that "made sense" 🤷‍♀️, and then they were encoded and passed to the domain name system (DNS). 💅 The IDNA2008 protocol (described in RFCs 5890, 5891, 5892, and 5893) assumes the input comes from a set of "permitted" code points 🎀, and then it encodes those and passes them to the DNS. BUT it doesn't say what to do with the result of user input, so this doc fills that gap ✨. It describes the actions an implementation can take between getting user input and handing permitted code points over to the new IDNA protocol 🫶.

Status of This Memo

This document is not an Internet Standards Track specification 🙈. It's published for informational purposes 📚.

This is a contribution to the RFC Series, independent of any other RFC stream 🦋. The RFC Editor picked this document to publish at their discretion, and they make no statement about how valuable it is for implementation or deployment 🤷‍♀️. Documents the RFC Editor approves for publication are not a candidate for any level of Internet Standard, see Section 2 of RFC 5741 💀.

Info about the current status of this document, any errata, and how to give feedback on it can be found at http://www.rfc-editor.org/info/rfc5895 🔗.

Copyright (c) 2010 IETF Trust and the persons identified as the document authors 💜. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date this document was published 📜. Please read these carefully, because they spell out your rights and restrictions for this document 🫶.

1. Introduction ✨

This document describes the operations that can be applied to user input so that it ends up in a form the Internationalized Domain Names in Applications (IDNA) protocol will accept [IDNA2008protocol] 💖. It also includes a general implementation procedure for mapping 🗺️.

Now, it's worth noting that this document does NOT describe the behavior of a protocol that shows up "on the wire" 🙅‍♀️. It describes an operation applied to user input to get that input ready for use in an "on the network" protocol 🌐. I know, it's kinda unusual for an Internet protocol doc 😭, but it's necessary to spell this out for implementers who designed around the original IDNA protocol (called IDNA2003 here) 🧩. IDNA2003 mashed this user-input operation right into the protocol itself 🍓.

Also, super important: there are LOTS of valid ways to map characters from user input 🌸. The mapping in this document is the basis for other mappings, and it's not likely to be useful without tweaks 🛠️. Any useful mapping will be designed to reduce surprise for users, and it's likely to be a bit (or sometimes a LOT) different depending on the user's locale, the kind of input they're using (typing, copy-and-paste, voice, and so on), the kind of application, and more 🌍. Most common mappings will probably give similar results for the same input, but there will be subtle differences between applications 👀.

1.1. The Dividing Line between User Interface and Protocol 🦋

The user interface of applications is way more complicated than most network implementers realize 🤯. When we say "the user enters an internationalized domain name in the application," we're talking about a super complex process 💅. It goes from the user figuring out the name and picking which symbols to express it with, to the user typing those symbols in with some input method (a keyboard, a stylus, or even a voice recognition program) 🎙️, to the computer interpreting that input (keyboard scan codes, a graphical representation, or digitized sounds) into some representation of those symbols 💻. Finally, those symbols get normalized into a particular character repertoire, in an encoding that IDNA processes and the domain name system can recognize 🎯.

Designing a user interface for internationalized domain names means taking culture, context, and locale into account for each user 🌏. A simple, well-known example is lowercasing LATIN CAPITAL LETTER I (U+0049) in Turkish and other languages 🇹🇷. A capital "I" in Turkish is properly lowercased to LATIN SMALL LETTER DOTLESS I (U+0131), NOT to LATIN SMALL LETTER I (U+0069) 😱. This lowercasing clearly depends on the locale of the system and/or the locale of the user 🫠. If you use a single context-free mapping without considering user interface properties, you can end up doing exactly the wrong thing for the user 💀.

The original version of IDNA mashed user interface processing and protocol together 🥴. It took whatever characters the user produced, in whatever encoding the application used, assumed some conversion to Unicode code points 🔢, and then, without caring about context, locale, or what the user actually meant 🙃, mapped them into a particular set of other characters. Then it re-encoded them in Punycode, so the whole operation could live inside the protocol 📦. Ignoring context, locale, and user preference in the IDNA protocol made life way simpler for application developers 🧑‍💻, but the cost was violating the principle of "least user surprise" for the people consuming and producing domain names 😤.

In IDNA2008, the dividing line between "user interface" and "protocol" is clear ✨. The IDNA2008 specification defines the protocol part of IDNA, and it explicitly does NOT deal with the user interface 🙅‍♀️. Mappings like the one in this document explicitly deal with the user interface and not the protocol 🎀. So a mapping is only applied BEFORE a string of characters gets treated as a domain name (in the "user interface"), and it's never applied DURING domain name processing (in the "protocol") 🚫.

1.2. The Design of This Mapping 🧚

The user interface mapping in this document is a set of expansions to IDNA2008 🧩. Its goals are to be sensible, friendly, and mostly obvious to people all over the world when they use typical applications with hand-entered domain names 🌍. It's also designed to let applications stay mostly backwards compatible with IDNA2003 🔙. Obviously, it can't hit every one of those goals for everyone 🥲, and it's known to miss some of them for pretty large groups of people 💔.

A good mapping in the real world might share the "sensible, friendly, and mostly obvious" goal but come up with a different algorithm 💡. Lots of algorithms will get results close to what's described here, but they'll differ in their assumptions about how users think or type 🤔. That said, some mappings will probably be WAY different 😳. For example, a mapping might apply to a spoken user interface instead of a typed one 🎙️. Another example: a mapping might differ for users who are typing versus users copying and pasting from different applications 📋. And a user interface that allows typed input transliterated from Latin characters could have very different mappings than one for typing in other character sets. This would be typical of a Pinyin input method for Chinese characters 🇨🇳🥢.

2. The General Procedure ✨

okay bestie, this is the recipe section 🍓 It's basically "here's the algorithm apps ought to follow," and the vibe is a general, pretty chill mapping built to be acceptable to the widest user community possible 💖. Just remember the plot twist from before: it doesn't account for any particular context, culture, or locale 🙈. Anyway, the algorithm an application (or the input method from the OS) ought to use is honestly pretty straightforward:

  1. Uppercase characters get mapped to their lowercase equivalents using the Unicode case mapping algorithm. This was picked because the output will behave more like ASCII host names do 💅
  2. Fullwidth and halfwidth characters (the ones with Decomposition Types <wide> and <narrow>) get mapped to their decomposition mappings as shown in the Unicode character database. This step exists because lots of input mechanisms, especially in Asia, don't let you easily type the form IDNA2008 uses 😭 And even when they do, the user might not even know which form they're typing.
  3. All characters get mapped using Unicode Normalization Form C (NFC) 🦋 This one combines sequences of combining characters into their canonical composed form. Same energy as the last step: users usually aren't aware of the exact form they're entering, and IDNA2008 requires that only the canonical composed forms from NFC be used.
  4. [IDNA2008protocol] is specified so the protocol acts on the individual labels of a domain name. 🧚 If your implementation also splits a domain name into labels using the FULL STOP character (U+002E), the IDEOGRAPHIC FULL STOP character (U+3002) can be mapped to the FULL STOP before label separation happens. Other characters are used as "full stops" too, and you could consider mapping them as label separators, but their use as such hasn't been investigated thoroughly. This step exists because some input mechanisms don't let users easily enter proper label separators. Only U+3002 gets added here, because the authors haven't fully investigated the other characters or the environments where they should and shouldn't count as domain name label separators.

Note that the steps above are ordered, so don't shuffle them 🙅‍♀️

Definitions for the rules in this algorithm can be found in [Unicode52]. Specifically:

If the mappings in this document are applied to versions of Unicode later than Unicode 5.2, the later versions of the Unicode Standard should be consulted 📚

These form a minimal set of mappings that an application should strongly consider doing. Of course, there are many others that might be done ✨

3. Implementing This Mapping

If you're implementing a mapping for an application or operating system by using exactly the four steps in Section 2, the authors of this document have a request: please don't. We mean it. 🙅‍♀️ Section 2 does not describe a universal mapping algorithm because, as we said, there is no universally-applicable mapping algorithm.

If you read the material in Section 2 without reading Section 1, go back and carefully read all of Section 1. In many ways, Section 1 is more important than Section 2 🥹 Also, you can probably think of user interface considerations that we didn't list in Section 1. And if you did read Section 1 but still decided the algorithm in Section 2 is completely correct for your intended users, you're probably not thinking hard enough about those users 💀

4. Security Considerations 🚨

This document suggests creating mappings that might cause confusion for some users while alleviating confusion in other users. Such confusion is not covered in any depth in this document (nor in the other IDNA-related documents).

5. Acknowledgements 🫶

This document is the product of many contributions from numerous people in the IETF 💜

6. Normative References 📚

[IDNA2008protocol] Klensin, J., "Internationalized Domain Names in Applications (IDNA): Protocol", RFC 5891, August 2010.

[TR44] The Unicode Consortium, "Unicode Technical Report #44: Unicode Character Database", September 2009, <http://www.unicode.org/reports/tr44/tr44-4.html>.

[Unicode-UAX15] The Unicode Consortium, "Unicode Standard Annex #15: Unicode Normalization Forms, Revision 31", September 2009, <http://www.unicode.org/reports/tr15/tr15-31.html>.

[Unicode52] The Unicode Consortium. The Unicode Standard, Version 5.2.0, defined by: "The Unicode Standard, Version 5.2.0", (Mountain View, CA: The Unicode Consortium, 2009. ISBN 978-1-936213-00-9). <http://www.unicode.org/versions/Unicode5.2.0/>.

Authors' Addresses 💌

Peter W. Resnick
Qualcomm Incorporated
5775 Morehouse Drive
San Diego, CA  92121-1714
US

Phone: +1 858 651 4478
EMail: presnick@qualcomm.com
URI:   http://www.qualcomm.com/~presnick/


Paul Hoffman
VPN Consortium
127 Segre Place
Santa Cruz, CA  95060
US

Phone: 1-831-426-9827
EMail: paul.hoffman@vpnc.org

and that's the whole spec, bestie 🌈🦋👑

📦 where this page comes from