RFC 6055
IAB Thoughts on Encodings for Internationalized Domain Names
IAB Thoughts on Encodings for Internationalized Domain Names 🎀
Okay bestie, this one is an Informational doc from the Internet Architecture Board (IAB) 🧚, published as RFC 6055 in February 2011. The authors are D. Thaler (Microsoft), J. Klensin, and S. Cheshire (Apple). It updates RFC 2130, and its header doesn't list anything it obsoletes. It's not a standard, so it's giving "IAB shares its thoughts" 💅. ISSN: 2070-1721.
Abstract ✨
This doc looks at the problems with Internationalized Domain Names (IDNs) that come from using different encoding schemes, like UTF-8 and the ASCII-Compatible Encoding made by the Punycode algorithm. The big point is that agreeing on a single encoding matters, and things get super messy when we use different encodings today 😭.
Status of This Memo 📋
This doc is not an Internet Standards Track specification; it's published for informational purposes.
This doc is a product of the Internet Architecture Board (IAB), and it holds information the IAB thinks is valuable to keep as a permanent record. Docs the IAB approves for publication are not a candidate for any level of Internet Standard, see Section 2 of RFC 5741 🔥.
Info about the current status of this doc, any errata, and how to give feedback on it can be found at http://www.rfc-editor.org/info/rfc6055.
Copyright Notice ©️
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they're in effect on the date of publication. Please read these carefully, because they explain your rights and restrictions for this doc 🙈.
Table of Contents 🗂️
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2. Use of Non-DNS Protocols . . . . . . . . . . . . . . . . . . . 9
3. Use of Non-ASCII in DNS . . . . . . . . . . . . . . . . . . . 10
3.1. Examples . . . . . . . . . . . . . . . . . . . . . . . . . 14
4. Recommendations . . . . . . . . . . . . . . . . . . . . . . . 16
5. Security Considerations . . . . . . . . . . . . . . . . . . . 18
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 19
7. IAB Members at the Time of Approval . . . . . . . . . . . . . 19
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.1. Normative References . . . . . . . . . . . . . . . . . . . 20
8.2. Informative References . . . . . . . . . . . . . . . . . . 201. Introduction 🦋
The goal here is to dig into what we can learn from some current struggles with implementing Internationalized Domain Names (IDNs) 🧠.
A domain name is a sequence of labels, usually written separated by dots. An IDN is a domain name that has one or more labels which contain one or more non-ASCII characters. Just like plain ASCII domain names, each IDN label has to be encoded with some mechanism before it can be sent in network packets, stored in memory, stored on disk, etc. These encodings need to be reversible, but they don't have to store domain names the same way humans write them on paper. For example, when domain name labels are sent over the network in DNS packets, they are not separated with dots 👀.
Internationalized Domain Names for Applications (IDNA), which we'll talk about later, is the standard that defines how internationalized domain names are used and coded for the public Internet [RFC5890] 💖. An earlier version of IDNA [RFC3490] is now being phased out. Except where noted, the two versions are pretty much the same for the issues in this doc. Some explanations showed up in the earlier docs but were no longer considered useful once the later revision came out, so they're quoted here from the docs where they appear. Also, the terminology of the two versions is a little different, and this doc uses the terminology of the current version ✨.
Unicode [Unicode] is a list of characters (including non-spacing marks that are used to form some other characters), where each character is given an integer value, called a code point 🌈. Simply put, a Unicode string is a string of integer code point values in the range 0 to 1,114,111 (10FFFF in base 16). These integer code points also have to be encoded with some mechanism before they can be sent in network packets, stored in memory, stored on disk, etc. Common ways of encoding these integer code point values in computer systems include UTF-8, UTF-16, and UTF-32. Besides the stuff below, those forms and the tradeoffs between them are discussed in Chapter 2 of The Unicode Standard [Unicode] 📚.
UTF-8 is a mechanism for encoding a Unicode code point in a variable number of 8-bit octets, where an ASCII code point is kept as-is. Those octets encode a string of integer code point values, which represent a string of Unicode characters. The authoritative definition of UTF-8 is in Sections 3.9 and 3.10 of The Unicode Standard [Unicode], but a description of UTF-8 encoding can also be found in RFC 3629 [RFC3629]. Descriptions and formulae can also be found in Annex D of ISO/IEC 10646-1 [10646] 🫶.
UTF-16 is a mechanism for encoding a Unicode code point in one or two 16-bit integers, described in detail in Sections 3.9 and 3.10 of The Unicode Standard [Unicode]. A UTF-16 string encodes a string of integer code point values that represent a string of Unicode characters 💅.
UTF-32 (formerly UCS-4), also described in Sections 3.9 and 3.10 of The Unicode Standard [Unicode], is a mechanism for encoding a Unicode code point in a single 32-bit integer. So a UTF-32 string is a string of 32-bit integer code point values, which represent a string of Unicode characters 🍓.
Note that UTF-16 gives some all-zero octets when code points occur early in the Unicode sequence, and UTF-32 always has all-zero octets 🙈.
IDNA specifies the validity of a label, like what characters it can contain, how they relate to each other, and so on, in Unicode terms 🔥. Valid labels can be in either "U-label" or "A-label" form, and the right one is determined by particular protocols or by context. U-label form is a direct representation of the Unicode characters using one of the encoding forms above. This doc talks about UTF-8 strings in lots of places. While all U-labels can be represented by UTF-8 strings, not all UTF-8 strings are valid U-labels (see Section 2.3.2 of the IDNA Definitions document [RFC5890] for a discussion of these distinctions) 🤓. A-label form uses a compressed, ASCII-compatible encoding (an "ACE" in IDNA and other terminology) made by an algorithm called Punycode. U-labels and A-labels are duals of each other: going from one to the other doesn't lose information. The transformation mechanisms are specified in the IDNA Protocol document [RFC5891] 💜.
Punycode [RFC3492] is a mechanism for encoding a Unicode string in an ASCII-compatible encoding, i.e., using only letters, digits, and hyphens from the ASCII character set 🧚. When a Unicode label that is valid under the IDNA rules (a U-label) is encoded with Punycode for IDNA purposes, it gets the prefix xn--, and the result is called an A-label. The prefix convention assumes that no other DNS labels (at least no other DNS labels in IDNA-aware applications) are allowed to start with these four characters. So when A-label encoding is assumed, any DNS label beginning with xn-- now means something different (the Punycode encoding of a label with one or more non-ASCII characters), or has no defined meaning at all (for labels that aren't IDNA-compliant, i.e., aren't well-formed A-labels) 😬.
ISO-2022-JP [RFC1468] is a mechanism for encoding a string of ASCII and Japanese characters, where an ASCII character is kept as-is. ISO-2022-JP is stateful: special sequences are used to switch between character coding tables. Because of that, if characters in a character stream are lost or mangled, it's extremely difficult to recover the original stream after a lost encoding shift 💀.
Comparing Unicode strings isn't as easy as comparing ASCII strings. First, there are a multitude of ways to represent a string of Unicode characters. Second, in many languages and scripts, the actual definition of "same" is very context-dependent. Because of this, comparing two Unicode strings has to take into account how they're encoded. Regardless of the encoding, though, you can't just compare the encoded Unicode strings byte by byte. That only works when both strings have been mapped into some canonical form and encoded the same way 😤.
In 1996 the IAB sponsored a workshop on character sets and encodings [RFC2130]. This doc adds to that discussion and focuses on the importance of agreeing on a single encoding and how complicated things get because we use different encodings today 🌸.
Different applications, APIs, and protocols use different encoding schemes today. Many of them were first defined to use only ASCII. Internationalizing Domain Names in Applications (IDNA) [RFC5890] defines a mechanism that requires changes to applications, but to avoid changing APIs or servers, it specifies that the A-label format is to be used in many contexts. In some ways, this could be seen as not changing the existing APIs, since the strings passed to and from the APIs still look like ASCII strings. In other ways it's a very big change to the existing APIs, because even though those strings are still syntactically valid ASCII strings, they no longer mean what they used to. What looks like a plain ASCII string to one piece of software or library could be seen by another piece of software or library (using out-of-band information) as actually being an encoding of a Unicode string 🫣.
Section 1.3 of the original IDNA specification [RFC3490] says:
The IDNA protocol is contained completely within applications. It is not a client-server or peer-to-peer protocol: everything is done inside the application itself. When used with a DNS resolver library, IDNA is inserted as a "shim" between the application and the resolver library. When used for writing names into a DNS zone, IDNA is used just before the name is committed to the zone.
Figure 1 depicts a simplistic architecture that a naive reader might assume from the paragraph quoted above. (A variant of this same picture appears in Section 6 of the original IDNA specification [RFC3490], further strengthening this assumption.) 🧐
+-----------------------------------------+
|Host |
| +-------------+ |
| | Application | |
| +------+------+ |
| | |
| +----+----+ |
| | DNS | |
| | Resolver| |
| | Library | |
| +----+----+ |
| | |
+-----------------------------------------+
|
_________|_________
/ \
/ \
/ \
| Internet |
\ /
\ /
\___________________/
Simplistic Architecture
Figure 1Okay so this is the "simplistic" picture of how a host talks to the Internet, and it's a little too cute to be true 🙈. There are, however, two problems with it that make it differ from reality 💅.
First, resolver APIs on Operating Systems (OSs) today (Mac OS, Windows, Linux, etc.) are not DNS-specific 🦋. They usually give apps a layer of indirection, so the app can work no matter how the name gets resolved. That could be DNS, mDNS [DNS-MULTICAST], LLMNR [RFC4795], NetBIOS-over-TCP [RFC1001][RFC1002], the hosts table [RFC0952], NIS [NIS], or literally anything else 💀. For example, "Basic Socket Interface Extensions for IPv6" [RFC3493] specifies the getaddrinfo() API and has lines like "For example, when using the DNS" and "any type of name resolution service (for example, the DNS)". The main point is that DNS only shows up as an example, and the application has no idea whether DNS or some other protocol is going to be used 🫶.
Second, even with the DNS protocol, private namespaces (sometimes including private uses of the DNS) don't necessarily use the same character set encoding scheme as the public Internet namespace 🌸.
We're going to dig into each of these issues in later sections ✨. For reference, Figure 2 shows a more realistic setup on typical hosts today (which don't have IDNA slotted in as a shim right above the DNS resolver library) 🧚. More generally, a host might be attached to one or more local networks, and each of those may or may not be connected to the public Internet, and may or may not have a private namespace 👑.
+-----------------------------------------+
|Host |
| +-------------+ |
| | Application | |
| +------+------+ |
| | |
| +------+------+ |
| | Generic | |
| | Name | |
| | Resolution | |
| | API | |
| +------+------+ |
| | |
| +-----+------+---+--+-------+-----+ |
| | | | | | | |
| +-+-++--+--++--+-++---+---++--+--++-+-+ |
| |DNS||LLMNR||mDNS||NetBIOS||hosts||...| |
| +---++-----++----++-------++-----++---+ |
| |
+-----------------------------------------+
|
______|______
/ \
/ \
/ local \
\ network /
\ /
\_____________/
|
_________|_________
/ \
/ \
/ \
| Internet |
\ /
\ /
\___________________/
Realistic Architecture
Figure 21.1. APIs ✨
Section 6.2 of the original IDNA specification [RFC3490] says this (where ToASCII and ToUnicode refer to conversions using the Punycode algorithm) 🎀:
It is expected that new versions of the resolver libraries in the future will be able to accept domain names in other charsets than ASCII, and application developers might one day pass not only domain names in Unicode, but also in local script to a new API for the resolver libraries in the operating system. Thus the ToASCII and ToUnicode operations might be performed inside these new versions of the resolver libraries.
Okay so resolver APIs like getaddrinfo() and its older sibling gethostbyname() were designed to take C-Language "char " arguments 💖. That means they take a string of bytes that ends with a NULL (0) byte. Because a NULL octet marks the end of the string, this works fine for ASCII strings (including A-labels), and even for ISO-2022-JP [RFC1468] and UTF-8 strings (unless an implementation artificially blocks them) 🌈. It does *not work for UTF-16 or UTF-32 strings, because a NULL octet can show up right in the middle of those encodings 😭. Several operating systems that were historically used in Japan will accept (and expect) ISO-2022-JP strings in these APIs 🍓. Some platforms used worldwide have newer versions of the APIs (for example, GetAddrInfoW() on Windows) that accept other encodings like UTF-16 🔥.
It's worth knowing that an API using C-Language "char " arguments can tell apart conventional ASCII "hostname" labels, A-labels, ISO-2022-JP, and UTF-8 labels in names, *if the coding is known to be one of those four and the label is intact (no lost or mangled characters) 🔍. If a stateful encoding like ISO-2022-JP is used, applications pulling labels out of text have to take special precautions to make sure the right state-setting characters are included in the string passed to the API ⚠️.
Applications pulling labels out of text must take special precautions with stateful encodings, or the API may not decode them correctly.
Here's an example method for telling these codings apart 🦋:
- if the label contains an ESC (0x1B) byte, the label is ISO-2022-JP; otherwise,
- if any byte in the label has the high bit set, the label is UTF-8; otherwise,
- if the label starts with "xn--", then it is presumed to be an A-label; otherwise,
- the label is ASCII (and therefore, by definition, the label is also UTF-8, since ASCII is a subset of UTF-8).
Again, this assumes that ASCII labels never start with "xn--", and that UTF-8 strings never contain an ESC character 🙈. Also, the above is just an illustration. UTF-8 can be detected and told apart from other 8-bit encodings with good accuracy [MJD] 💜.
It's harder, or flat-out impossible, to tell the ISO 8859 character sets [ISO8859] apart from each other, because they differ in up to about 90 characters that have exactly the same encodings. A short string is very unlikely to contain enough characters for a receiver to figure out which character set it is 🥺. Likewise, it isn't possible in general to tell ISO-2022-JP apart from any other encoding based on ISO 2022 code table switching 🌀.
So yes, you can sometimes figure out an encoding when it isn't explicitly specified (like in the example above) 🔎. But honestly, it's cleaner to have the encodings specified explicitly, like specifying UTF-16 for GetAddrInfoW(), or specifying exactly which APIs expect UTF-8 strings 👑.
2. Use of Non-DNS Protocols ✨
As we said earlier, typical name resolution libraries aren't DNS-specific 💅. On top of that, some protocols are defined to use encoding forms other than IDNA A-labels. For example, mDNS [DNS-MULTICAST] specifies that UTF-8 be used 🌸. And the IETF policy on character sets and languages [RFC2277] (which came after the 1996 IAB-sponsored workshop [RFC2130]) says:
Protocols MUST be able to use the UTF-8 charset, which consists of the ISO 10646 coded character set combined with the UTF-8 character encoding scheme, as defined in [10646] Annex R (published in Amendment 2), for all text. Protocols MAY specify, in addition, how to use other charsets or other character encoding schemes for ISO 10646, such as UTF-16, but lack of an ability to use UTF-8 is a violation of this policy; such a violation would need a variance procedure ([BCP9] section 9) with clear and solid justification in the protocol specification document before being entered into or advanced upon the standards track. For existing protocols or protocols that move data from existing datastores, support of other charsets, or even using a default other than UTF-8, may be a requirement. This is acceptable, but UTF-8 support MUST be possible.
Now here's where it gets messy 😭. Applications that convert an IDN to A-label form before calling getaddrinfo() will get name resolution failures if the Punycode name is used directly in these protocols 💀. Having libraries or protocols convert from A-labels to whatever encoding the protocol uses (like UTF-8) would mean changing APIs and/or servers, and avoiding exactly that kind of change was what IDNA was meant to do 🧚.
So applications that assume non-ASCII names get resolved using the public DNS, and blindly convert them to A-labels without knowing which protocol the name resolution library will pick, are going to have problems 🙈. Plus, name resolution libraries often try several protocols until one succeeds, because they're all defined to use a common namespace 🔁. For example, the hosts file [RFC0952], NetBIOS-over-TCP [RFC1001], and DNS [RFC1034] are all defined to be able to share a common syntax. That means when an application passes in a name to resolve, the resolution may actually be attempted with multiple protocols, each with a potentially different encoding scheme 🌈. For this to work, the name has to be converted to the right encoding scheme only after the protocol is chosen. And in general, the application can't do that itself, since the application doesn't get to choose the protocol 🫶.
3. Use of Non-ASCII in DNS ✨
okay so bestie, there's this super common myth that DNS only does names made of letters, digits, and hyphens 🙈 and honestly? it's so not true, and I need you to see why 💅
The myth actually comes from 1985, when the "hosts" file [RFC0952] defined an "Internet hostname" (and net, gateway, and domain name) as only letters, digits, and hyphens, with uppercase and lowercase treated as identical. Then the DNS specification [RFC1034], Section 3.5 entitled "Preferred name syntax" repeated that definition in 1987, saying that this "syntax will result in fewer problems with many applications that use domain names (e.g., mail, TELNET)".
So the confusion was like, is that "preferred" syntax actually a mandatory rule in DNS, or is it just preferred 🤔
In 1989 the definition of an Internet hostname got an update ([RFC1123], Section 2.1) to allow names starting with a digit 🌸 But it didn't fix the bigger mess of whether every DNS name counts as a "hostname", or whether a "hostname" is just one special case of a DNS name.
By 1997 things were getting seriously confusing, so "Clarifications to the DNS Specification" [RFC2181], Section 11 stepped in and said:
The DNS itself places only one restriction on the particular labels that can be used to identify resource records. That one restriction relates to the length of the label and the full name. The length of any one label is limited to between 1 and 63 octets. A full domain name is limited to 255 octets (including the separators). The zero length full name is defined as representing the root of the DNS tree, and is typically written and displayed as ".". Those restrictions aside, any binary string whatever can be used as the label of any resource record. Similarly, any binary string can serve as the value of any record that includes a domain name as some or all of its value (SOA, NS, MX, PTR, CNAME, and any others that may be added). Implementations of the DNS protocols must not place any restrictions on the labels that can be used.
So that settles it 💖 The letters-digits-hyphens thing does NOT apply to DNS names in general, or to records that include "domain names". That means the "preferred" name syntax in the original DNS spec [RFC1034] is genuinely just preferred, not mandatory. Iconic behavior from the spec, honestly 👑
Since there's no restriction even to ASCII, let alone letter-digit-hyphen, DNS doesn't break the later IETF requirement to allow UTF-8 [RFC2277]. We're good 🦋
Now, using UTF-16 or UTF-32 wouldn't be ideal for DNS packets or C-Language "char *" APIs. Why? Because existing software already uses ASCII, and UTF-16 and UTF-32 strings can contain all-zero octets that existing software will read as the end of the string 😭 And if you did go with UTF-16 or UTF-32, you'd need some way to know whether a string was ASCII, UTF-16, or UTF-32, and for UTF-16 or UTF-32, whether it was big-endian or little-endian. UTF-8, on the other hand, works so well because any 7-bit ASCII string is also a UTF-8 string representing the same characters ✨ Main character encoding energy.
If a private namespace uses UTF-8 (and not UTF-16 or UTF-32), there's no need for any mechanism to tell ASCII from UTF-8. For any string that can be written in ASCII, the two encodings are exactly the same, and for any string that can't be written in ASCII, it's obviously UTF-8 🎀 Also, ASCII and UTF-8 are both byte-oriented, so the big-endian versus little-endian question doesn't even apply.
Here's the nuance though 🧚 DNS implementations must not place any restrictions on the labels that can be used, but applications that use DNS are free to impose whatever restrictions they like, and many do. The rules above do allow weird characters like embedded spaces, which lots of applications think is a bad idea. For example, the original SMTP specification [RFC0821] (SMTP is [RFC5321]) constrains the character set usable in email addresses. There's now an effort underway to define an SMTP extension for internationalized email addresses and headers. For more on that, check out the EAI framework [RFC4952] 💜
Next up, shortly after the DNS Clarifications [RFC2181] and the IETF character sets and languages policy [RFC2277] came out, people needed internationalized names inside private namespaces (aka within enterprises). The current practice (and the practice from before IDNA and the prefixed ACE conventions existed) is for enterprises that support other languages to put UTF-8 names in their internal DNS servers in a private namespace. For example, "Using the UTF-8 Character Set in the Domain Name System" [UTF8-DNS] was first written in 1997, and then got widely deployed in Windows. Mac OS did the same thing, and the reason it just worked is that applications blindly passed UTF-8 strings to the name resolution APIs, those APIs blindly passed the strings to the DNS servers, and the DNS servers correctly answered the queries 🙌 From the user's side, everything worked with no special new code, except that ASCII is matched case-insensitively while UTF-8 isn't. (Some enterprise DNS servers reportedly try case-insensitive matching on UTF-8 in private namespaces, but that causes other problems and breaks a later prohibition [RFC4343], so ngl, that's a hard no 🙅) Within a private namespace, and especially given the IETF UTF-8 policy [RFC2277], it was reasonable to assume binary strings were UTF-8.
Also, there are issues with mapping strings to a canonical form, no matter the encoding 🌈 This document doesn't go into detail on those. They're discussed to some extent in, for example, Section 3 of "Unicode Format for Network Interchange" [RFC5198], and they're left as opportunities for other documents to elaborate on.
A few years after UTF-8 was already being used in private DNS namespaces, the strategy of using a reserved prefix and an ASCII-compatible encoding (ACE) was developed for IDNA 💅 That strategy included the Punycode algorithm, which was developed between 2002 [IDN-PUNYCODE] and 2003 [RFC3492] for use in the public DNS namespace. There were a few reasons for this. One was that the prefixed ACE strategy was picked for the public namespace because other encodings like ISO 8859-1 were also in use in DNS, and those encodings weren't necessarily distinguishable from each other. Another was concern about whether IDNA's details, including the Punycode algorithm, were an adequate solution to the problems being posed. If either Punycode or fundamental parts of character handling turned out to be wrong and had to change to something incompatible, it would be possible to switch to a new prefix or adopt a totally different model. Only the part of the public DNS namespace that starts a label with "xn--" would be affected 🦋
Today the algorithm is seen as about as good as it can realistically be. So moving to a different encoding (UTF-8, as suggested in this document), which can be viewed as "native", wouldn't be as risky as it would have been in 2002 🍓
Anyway, publishing Punycode [RFC3492] and the dependencies on it in the IDNA Protocol document [RFC5891] and the earlier IDNA specification [RFC3490] meant that different encodings had to be used for different namespaces (even though UTF-8 was already deployed in private namespaces). So, going back to Figure 2, a different encoding scheme may be in use on the Internet than on a local network 🌸
In general, a host may be connected to zero or more networks using private namespaces, plus potentially the public namespace. Applications that convert a U-label form IDN to an A-label before calling getaddrinfo() will get name resolution failures if the name is actually registered in a private namespace using some other encoding (e.g., UTF-8). Having libraries or protocols convert A-labels into the encoding a private namespace uses (e.g., UTF-8) would require changes to APIs and/or servers, which is exactly what IDNA was meant to avoid 😤
Also, an FQDN to be resolved might come straight from an application, or the DNS resolver might build it itself from a single label an application provided, by using a configured suffix search list. That resulting FQDN can even use multiple encodings across its labels 🤯 For more on the suffix search list, see Section 6 of "Common DNS Implementation Errors and Suggested Fixes" [RFC1536], the DHCP Domain Search Option [RFC3397], and Section 4 of "DNS Configuration options for DHCPv6" [RFC3646].
As noted in Section 6 of "Common DNS Implementation Errors and Suggested Fixes" [RFC1536], the community has had bad experiences (e.g., security problems [RFC1535]) with "searching" for domain names by trying multiple variations or appending different suffixes 💀 That kind of searching can give inconsistent results depending on the order the alternatives are tried in. But the practice is widespread, so nonetheless, it must be considered 🫶
Searching for names, whether through a suffix search list or by searching different namespaces, can give inconsistent results. For example, even when a suffix search list is only used for an application-provided name with no dots, two clients with different configured suffix search lists can get different answers. The same client could also get different answers at different times if it changes its configuration (e.g., when moving to another network) 😵💫 A deeper discussion of this topic is outside the scope of this document.
3.1. Examples ✨
Okay bestie, here are some examples of stuff that already happens in real implementations today 💅 (where {non-ASCII} is some user-typed string with non-ASCII characters in it):
- User types in
{non-ASCII}.{non-ASCII}.com, and the application passes it, as a UTF-8 string, togetaddrinfo()orgethostbyname()or equivalent.- The DNS resolver passes the (UTF-8) string unmodified to a DNS server.
- User types in
{non-ASCII}.{non-ASCII}.com, and the application passes it to a name resolution API that accepts strings in some other encoding such as UTF-16, e.g.,GetAddrInfoW()on Windows.- The name resolution API decides to pass the string to DNS (and possibly other protocols).
- The DNS resolver converts the name from UTF-16 to UTF-8 and passes the query to a DNS server.
- User types in
{non-ASCII}.{non-ASCII}.com, but the application first converts it to A-label form such that the name that is passed to name resolution APIs is (say)xn--e1afmkfd.xn--80akhbyknj4f.com.- The name resolution API decides to pass the string to DNS (and possibly other protocols).
- The DNS resolver passes the string unmodified to a DNS server.
- If the name is not found in DNS, the name resolution API decides to try another protocol, say mDNS.
- The query goes out in mDNS, but since mDNS specified that names are to be registered in UTF-8, the name isn't found since it was encoded as an A-label in the query. 😭
- User types in
{non-ASCII}, and the application passes it, in the form of a UTF-8 string, togetaddrinfo()or equivalent.- The name resolution API decides to pass the string to DNS (and possibly other protocols).
- The DNS resolver will append suffixes in the suffix search list, which may contain UTF-8 characters if the local network uses a private namespace.
- Each FQDN in turn will then be sent in a query to a DNS server, until one succeeds.
- User types in
{non-ASCII}, but the application first converts it to an A-label, such that the name that is passed togetaddrinfo()or equivalent is (say)xn--e1afmkfd.- The name resolution API decides to pass the string to DNS (and possibly other protocols).
- The DNS stub resolver will append suffixes in the suffix search list, which may contain UTF-8 characters if the local network uses a private namespace, resulting in (say)
xn--e1afmkfd.{non-ASCII}.com - Each FQDN in turn will then be sent in a query to a DNS server, until one succeeds.
- Since the private namespace in this case uses UTF-8, the above queries fail, since the A-label version of the name was not registered in that namespace. 💀
- User types in
{non-ASCII1}.{non-ASCII2}.{non-ASCII3}.com, where{non-ASCII3}.comis a public namespace using IDNA and A-labels, but{non-ASCII2}.{non-ASCII3}.comis a private namespace using UTF-8, which is accessible to the user. The application passes the name, in the form of a UTF-8 string, togetaddrinfo()or equivalent.- The name resolution API decides to pass the string to DNS (and possibly other protocols).
- The DNS resolver tries to locate the authoritative server, but fails the lookup because it cannot find a server for the UTF-8 encoding of
{non-ASCII3}.com, even though it would have access to the private namespace. (To make this work, the private namespace would need to include the UTF-8 encoding of{non-ASCII3}.com.)
When users use multiple applications, some of which do A-label conversion prior to passing a name to name resolution APIs, and some of which do not, odd behavior can result which at best violates the Principle of Least Surprise, and at worst can result in security vulnerabilities. Yikes. 🙈
First consider two competing applications, such as web browsers, that are designed to achieve the same task. If the user types the same name into each browser, one may successfully resolve the name (and hence access the desired content) because the encoding scheme is correct, while the other may fail name resolution because the encoding scheme is incorrect. Hence the issue can incent users to switch to another application (which in some cases means switching to an IDNA application, and in other cases means switching away from an IDNA application). Main character drama, but make it DNS. 👑
Next consider two separate applications where one is designed to be launched from the other, for example a web browser launching a media player application when the link to a media file is clicked. If both types of content (web pages and media files in this example) are hosted at the same IDN in a private namespace, but one application converts to A-labels before calling name resolution APIs and the other does not, the user may be able to access a web page, click on the media file causing the media player to launch and attempt to retrieve the media file, which will then fail because the IDN encoding scheme was incorrect. Or even worse, if an attacker is able to register the same name in the other encoding scheme, the user may get the content from the attacker's machine. This is similar to a normal phishing attack, except that the two names represent exactly the same Unicode characters. 🚨 Not okay, and we are NOT doing this. 🥺
4. Recommendations ✨
On many platforms, the name resolution library will automatically use a variety of protocols to search a variety of namespaces, which might be using UTF-8 or other encodings. In addition, even when only the DNS protocol is used, in many operational environments, a private DNS namespace using UTF-8 is also deployed and is automatically searched by the name resolution library. 🌈
As explained earlier, using multiple canonical formats, and multiple encodings in different protocols or even in different places in the same namespace creates problems. Because of this, and the fact that both IDNA A-labels and UTF-8 are in use as encoding mechanisms for domain names today, we make the recommendations described below. 🫶
It is inappropriate for an application that calls a general-purpose name resolution library to convert a name to an A-label unless the application is absolutely certain that, in all environments where the application might be used, only the global DNS that uses IDNA A-labels actually will be used to resolve the name. 🙅♀️
Instead, conversion to A-label form, or any other special encoding required by a particular name-lookup protocol, should be done only by an entity that knows which protocol will be used (e.g., the DNS resolver, or getaddrinfo() upon deciding to pass the name to DNS), rather than by general applications that call protocol-independent name resolution APIs. (Of course, applications that store strings internally in a different format than that required by those APIs, need to convert strings from their own internal format to the format required by the API.) Similarly, even if an application can know that DNS is to be used, the conversion to A-labels should be done only by an entity that knows which part of the DNS namespace will be used. ✨
That is, a more intelligent DNS resolver would be more liberal in what it would accept from an application and be able to query for both a name in A-label form (e.g., over the Internet) and a UTF-8 name (e.g., over a corporate network with a private namespace) in case the server only recognizes one. However, we might also take into account that the various resolution behaviors discussed earlier could also occur with record updates (e.g., with Dynamic Update [RFC2136]), resulting in some names being registered in a local network's private namespace by applications doing conversion to A-labels, and other names being registered using UTF-8. Hence, a name might have to be queried with both encodings to be sure to succeed without changes to DNS servers. 🔥
Similarly, a more intelligent stub resolver would also be more liberal in what it would accept from a response as the value of a record (e.g., PTR) in that it would accept either UTF-8 (U-labels in the case of IDNA) or A-labels and convert them to whatever encoding is used by the application APIs to return strings to applications. 🦋
Indeed the choice of conversion within the resolver libraries is consistent with the quote from Section 6.2 of the original IDNA specification [RFC3490] stating that conversion using the Punycode algorithm (i.e., to A-labels) "might be performed inside these new versions of the resolver libraries". 🧚
That said, some application-layer protocols (e.g., EPP Domain Name Mapping [RFC5731]) are defined to use A-labels rather than simply using UTF-8 as recommended by the IETF character sets and languages policy [RFC2277]. In this case, an application may receive a string containing A-labels and want to pass it to name resolution APIs. Again the recommendation that a resolver library be more liberal in what it would accept from an application would mean that such a name would be accepted and re-encoded as needed, rather than requiring the application to do so. 🍓
It is important that any APIs used by applications to pass names specify what encoding(s) the API uses. For example, GetAddrInfoW() on Windows specifies that it accepts UTF-16 and only UTF-16. In contrast, the original specification of getaddrinfo() [RFC3493] does not, and hence platforms vary in what they use (e.g., Mac OS uses UTF-8 whereas Windows uses Windows code pages). 💜
Finally, the question remains about what, if anything, a DNS server should do to handle cases where some existing applications or hosts do IDNA queries using A-labels within the local network using a private namespace, and other existing applications or hosts send UTF-8 queries. It is undesirable to store different records for different encodings of the same name, since this introduces the possibility for inconsistency between them. Instead, a new DNS server serving a private namespace using UTF-8 could potentially treat encoding-conversion in the same way as case-insensitive comparison which a DNS server is already required to do, as long the DNS server has some way to know what the encoding is. Two encodings are, in this sense, two representations of the same name, just as two case-different strings are. However, whereas case comparison of non-ASCII characters is complicated by ambiguities (as explained in the IAB's Review and Recommendations for Internationalized Domain Names [RFC4690]), encoding conversion between A-labels and U-labels is unambiguous. ⭐
5. Security Considerations 🔐
okay so this section is SERIOUS bestie 😭 like, converting names to the prefixed ACE format (the A-labels 🏷️) before you call name resolution can actually open up security holes 🚨. If the name gets resolved by protocols or zones that registered their records with some other encoding scheme, an attacker can claim the A-label version of the same name and trick the victim into going to a totally different destination 💀. And this can happen for ANY non-ASCII name, even when there's zero possible confusion from case, language, or other stuff 🙈. The other kinds of confusion that go beyond just the encoding choice are covered in "Review and Recommendations for IDNs" [RFC4690] 📚, so go read that one too 🫶.
designers and users of encodings that represent Unicode strings in terms of ASCII should also think about whether trademark protection or phishing could be issues 🎣. Like, if one name gets encoded in a way that naturally looks associated with another organization or product, that's a whole mess 😬✨.
6. Acknowledgements 🌸
the authors want to thank Patrik Faltstrom, Martin Duerst, JFC Morfin, Ran Atkinson, S. Moonesamy, Paul Hoffman, and Stephane Bortzmeyer for their careful review and helpful suggestions 💖. It's also iconic to note that none of the first three individuals' names above can be spelled out and written correctly in ASCII text 🤭. Furthermore, one of the IAB member's names below (Andrei Robachevsky) cannot be written in the script as it appears on his birth certificate 📜😭.
7. IAB Members at the Time of Approval 👑
- Bernard Aboba
- Marcelo Bagnulo
- Ross Callon
- Spencer Dawkins
- Vijay Gill
- Russ Housley
- John Klensin
- Olaf Kolkman
- Danny McPherson
- Jon Peterson
- Andrei Robachevsky
- Dave Thaler
- Hannes Tschofenig
8. References 📚
8.1. Normative References
[10646] International Organization for Standardization,
"Information Technology - Universal Multiple-octet
coded Character Set (UCS)".
ISO/IEC Standard 10646, comprised of ISO/IEC 10646-
1:2000, "Information technology -- Universal
Multiple-Octet Coded Character Set (UCS) -- Part 1:
Architecture and Basic Multilingual Plane", ISO/IEC
10646-2:2001, "Information technology -- Universal
Multiple-Octet Coded Character Set (UCS) -- Part 2:
Supplementary Planes" and ISO/IEC 10646- 1:2000/Amd
1:2002, "Mathematical symbols and other characters".
[Unicode] The Unicode Consortium. The Unicode Standard,
Version 5.1.0, defined by: "The Unicode Standard,
Version 5.0", Boston, MA, Addison-Wesley, 2007, ISBN
0-321-48091-0, as amended by Unicode 5.1.0
(http://www.unicode.org/versions/Unicode5.1.0/).8.2. Informative References
[DNS-MULTICAST] Cheshire, S. and M. Krochmal, "Multicast DNS", Work
in Progress, February 2011.
[IDN-PUNYCODE] Costello, A., "Punycode version 0.3.3", Work
in Progress, January 2002.
[ISO8859] International Organization for Standardization,
"Information technology -- 8-bit single-byte coded
graphic character sets".
ISO/IEC Standard 8859, comprised of ISO/IEC 8859-
1:1998, Part 1: Latin alphabet No. 1 - ISO/IEC 8859-
2:1999, Part 2: Latin alphabet No. 2 - ISO/IEC 8859-
3:1999, Part 3: Latin alphabet No. 3 - ISO/IEC 8859-
4:1998, Part 4: Latin alphabet No. 4 - ISO/IEC 8859-
5:1999, Part 5: Latin/Cyrillic alphabet - ISO/IEC
8859-6:1999, Part 6: Latin/Arabic alphabet - ISO/IEC
8859-7:2003, Part 7: Latin/Greek alphabet - ISO/IEC
8859-8:1999, Part 8: Latin/Hebrew alphabet - ISO/IEC
8859-9:1999, Part 9: Latin alphabet No. 5 - ISO/IEC
8859-10:1998, Part 10: Latin alphabet No. 6 - ISO/
IEC 8859-11:2001, Part 11: Latin/Thai alphabet -
ISO/IEC 8859-13:1998, Part 13: Latin alphabet No. 7
- ISO/IEC 8859-14:1998, Part 14: Latin alphabet No.
8 (Celtic) - ISO/IEC 8859-15:1999, Part 15: Latin
alphabet No. 9 - ISO/IEC 8859-16:2001, Part 16:
Latin alphabet No. 10.
[MJD] Duerst, M., "The Properties and Promizes of UTF-8",
11th International Unicode Conference, San Jose ,
September 1997, <http://www.ifi.unizh.ch/mml/
mduerst/papers/PDF/IUC11-UTF-8.pdf>.
[NIS] Sun Microsystems, "System and Network
Administration", March 1990.
[RFC0821] Postel, J., "Simple Mail Transfer Protocol", STD 10,
RFC 821, August 1982.
[RFC0952] Harrenstien, K., Stahl, M., and E. Feinler, "DoD
Internet host table specification", RFC 952,
October 1985.
[RFC1001] NetBIOS Working Group, "Protocol standard for a
NetBIOS service on a TCP/UDP transport: Concepts and
methods", STD 19, RFC 1001, March 1987.
[RFC1002] NetBIOS Working Group, "Protocol standard for a
NetBIOS service on a TCP/UDP transport: Detailed
specifications", STD 19, RFC 1002, March 1987.
[RFC1034] Mockapetris, P., "Domain names - concepts and
facilities", STD 13, RFC 1034, November 1987.
[RFC1123] Braden, R., "Requirements for Internet Hosts -
Application and Support", STD 3, RFC 1123,
October 1989.
[RFC1468] Murai, J., Crispin, M., and E. van der Poel,
"Japanese Character Encoding for Internet Messages",
RFC 1468, June 1993.
[RFC1535] Gavron, E., "A Security Problem and Proposed
Correction With Widely Deployed DNS Software",
RFC 1535, October 1993.
[RFC1536] Kumar, A., Postel, J., Neuman, C., Danzig, P., and
S. Miller, "Common DNS Implementation Errors and
Suggested Fixes", RFC 1536, October 1993.
[RFC2130] Weider, C., Preston, C., Simonsen, K., Alvestrand,
H., Atkinson, R., Crispin, M., and P. Svanberg, "The
Report of the IAB Character Set Workshop held 29
February - 1 March, 1996", RFC 2130, April 1997.
[RFC2136] Vixie, P., Thomson, S., Rekhter, Y., and J. Bound,
"Dynamic Updates in the Domain Name System (DNS
UPDATE)", RFC 2136, April 1997.
[RFC2181] Elz, R. and R. Bush, "Clarifications to the DNS
Specification", RFC 2181, July 1997.
[RFC2277] Alvestrand, H., "IETF Policy on Character Sets and
Languages", BCP 18, RFC 2277, January 1998.
[RFC3397] Aboba, B. and S. Cheshire, "Dynamic Host
Configuration Protocol (DHCP) Domain Search Option",
RFC 3397, November 2002.
[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello,
"Internationalizing Domain Names in Applications
(IDNA)", RFC 3490, March 2003.
[RFC3492] Costello, A., "Punycode: A Bootstring encoding of
Unicode for Internationalized Domain Names in
Applications (IDNA)", RFC 3492, March 2003.
[RFC3493] Gilligan, R., Thomson, S., Bound, J., McCann, J.,
and W. Stevens, "Basic Socket Interface Extensions
for IPv6", RFC 3493, February 2003.
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, November 2003.
[RFC3646] Droms, R., "DNS Configuration options for Dynamic
Host Configuration Protocol for IPv6 (DHCPv6)",
RFC 3646, December 2003.
[RFC4343] Eastlake, D., "Domain Name System (DNS) Case
Insensitivity Clarification", RFC 4343,
January 2006.
[RFC4690] Klensin, J., Faltstrom, P., Karp, C., and IAB,
"Review and Recommendations for Internationalized
Domain Names (IDNs)", RFC 4690, September 2006.
[RFC4795] Aboba, B., Thaler, D., and L. Esibov, "Link-local
Multicast Name Resolution (LLMNR)", RFC 4795,
January 2007.
[RFC4952] Klensin, J. and Y. Ko, "Overview and Framework for
Internationalized Email", RFC 4952, July 2007.
[RFC5198] Klensin, J. and M. Padlipsky, "Unicode Format for
Network Interchange", RFC 5198, March 2008.
[RFC5321] Klensin, J., "Simple Mail Transfer Protocol",
RFC 5321, October 2008.
[RFC5731] Hollenbeck, S., "Extensible Provisioning Protocol
(EPP) Domain Name Mapping", STD 69, RFC 5731,
August 2009.
[RFC5890] Klensin, J., "Internationalized Domain Names for
Applications (IDNA): Definitions and Document
Framework", RFC 5890, August 2010.
[RFC5891] Klensin, J., "Internationalized Domain Names in
Applications (IDNA): Protocol", RFC 5891,
August 2010.
[UTF8-DNS] Kwan, S. and J. Gilroy, "Using the UTF-8 Character
Set in the Domain Name System", Work in Progress,
November 1997.Authors' Addresses 💌
Dave Thaler
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
USA
Phone: +1 425 703 8835
EMail: dthaler@microsoft.com
John C Klensin
1770 Massachusetts Ave, Ste 322
Cambridge, MA 02140
Phone: +1 617 245 1457
EMail: john+ietf@jck.com
Stuart Cheshire
Apple Inc.
1 Infinite Loop
Cupertino, CA 95014
Phone: +1 408 974 3207
EMail: cheshire@apple.comThat's the whole RFC, bestie! 🌈🦋✨ Thanks for hanging with me through all five parts 🫶💜
📦 where this page comes from
- ietf: rfc6055 2011-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms