RFC 3987
Internationalized Resource Identifiers (IRIs)
Internationalized Resource Identifiers (IRIs) 🦋
Okay bestie, this is RFC 3987 and it's all about Internationalized Resource Identifiers (IRIs) 🌈 It was written by M. Duerst from the W3C and M. Suignard from Microsoft Corporation. It's a Standards Track doc from the Network Working Group, dated January 2005. The title block doesn't list anything it obsoletes or updates, so no drama there 💅
Status of This Memo
This one is an Internet standards track protocol for the whole Internet community, and it's literally asking people to talk about it and send suggestions for improvements 🗣️ For the current standardization state and status, check the current edition of the "Internet Official Protocol Standards" (STD 1) 📚 And distribution of this memo is unlimited, so share it wherever you want 💌
Copyright Notice
Copyright (C) The Internet Society (2005). 🎀
Abstract ✨
So the big idea here is a brand new protocol element called the Internationalized Resource Identifier (IRI), which is basically a complement to the Uniform Resource Identifier (URI) 🦋 An IRI is a sequence of characters from the Universal Character Set (Unicode/ISO 10646), and there's a defined mapping from IRIs to URIs. That means IRIs can be used instead of URIs wherever that makes sense for identifying resources 🫶
They didn't go with extending or changing the definition of URIs. They made a new element instead, so there's a clear distinction and no incompatibilities with software that already exists 💖 Guidelines are also provided for using and deploying IRIs in the various protocols, formats, and software components that currently deal with URIs 🧚
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Overview and Motivation . . . . . . . . . . . . . . . . 3
1.2. Applicability . . . . . . . . . . . . . . . . . . . . . 3
1.3. Definitions . . . . . . . . . . . . . . . . . . . . . . 4
1.4. Notation . . . . . . . . . . . . . . . . . . . . . . . . 5
2. IRI Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1. Summary of IRI Syntax . . . . . . . . . . . . . . . . . 6
2.2. ABNF for IRI References and IRIs . . . . . . . . . . . . 7
3. Relationship between IRIs and URIs . . . . . . . . . . . . . . 10
3.1. Mapping of IRIs to URIs . . . . . . . . . . . . . . . . 10
3.2. Converting URIs to IRIs . . . . . . . . . . . . . . . . 14
3.2.1. Examples . . . . . . . . . . . . . . . . . . . . 15
4. Bidirectional IRIs for Right-to-Left Languages. . . . . . . . 16
4.1. Logical Storage and Visual Presentation . . . . . . . . 17
4.2. Bidi IRI Structure . . . . . . . . . . . . . . . . . . . 18
4.3. Input of Bidi IRIs . . . . . . . . . . . . . . . . . . . 19
4.4. Examples . . . . . . . . . . . . . . . . . . . . . . . . 19
5. Normalization and Comparison . . . . . . . . . . . . . . . . . 21
5.1. Equivalence . . . . . . . . . . . . . . . . . . . . . . 22
5.2. Preparation for Comparison . . . . . . . . . . . . . . . 22
5.3. Comparison Ladder . . . . . . . . . . . . . . . . . . . 23
5.3.1. Simple String Comparison . . . . . . . . . . . . 23
5.3.2. Syntax-Based Normalization . . . . . . . . . . . 24
5.3.3. Scheme-Based Normalization . . . . . . . . . . . 27
5.3.4. Protocol-Based Normalization . . . . . . . . . . 28
6. Use of IRIs . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.1. Limitations on UCS Characters Allowed in IRIs . . . . . 29
6.2. Software Interfaces and Protocols . . . . . . . . . . . 29
6.3. Format of URIs and IRIs in Documents and Protocols . . . 30
6.4. Use of UTF-8 for Encoding Original Characters .. . . . . 30
6.5. Relative IRI References . . . . . . . . . . . . . . . . 32
7. URI/IRI Processing Guidelines (informative) . . . . . . . . . 32
7.1. URI/IRI Software Interfaces . . . . . . . . . . . . . . 32
7.2. URI/IRI Entry . . . . . . . . . . . . . . . . . . . . . 33
7.3. URI/IRI Transfer between Applications . . . . . . . . . 33
7.4. URI/IRI Generation . . . . . . . . . . . . . . . . . . . 34
7.5. URI/IRI Selection . . . . . . . . . . . . . . . . . . . 34
7.6. Display of URIs/IRIs . . . . . . . . . . . . . . . . . . 35
7.7. Interpretation of URIs and IRIs . . . . . . . . . . . . 36
7.8. Upgrading Strategy . . . . . . . . . . . . . . . . . . . 36
8. Security Considerations . . . . . . . . . . . . . . . . . . . 37
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 39
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 40
10.1. Normative References . . . . . . . . . . . . . . . . . . 40
10.2. Informative References . . . . . . . . . . . . . . . . . 41
A. Design Alternatives . . . . . . . . . . . . . . . . . . . . . 44
A.1. New Scheme(s) . . . . . . . . . . . . . . . . . . . . . 44
A.2. Character Encodings Other Than UTF-8 . . . . . . . . . . 44
A.3. New Encoding Convention . . . . . . . . . . . . . . . . 44
A.4. Indicating Character Encodings in the URI/IRI . . . . . 45
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 45
Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 461. Introduction 🌸
1.1. Overview and Motivation
Okay so a Uniform Resource Identifier (URI) is defined in [RFC3986] as a sequence of characters picked from a limited subset of the US-ASCII [ASCII] characters 🔤
Here's the thing: URIs often use characters to spell out words from natural languages, and that's honestly so useful. Those URIs are easier to memorize, easier to interpret, easier to transcribe, easier to create, and easier to guess 🧠 But for most languages other than English, the natural script uses characters that aren't A - Z. For a lot of people, handling Latin characters is as hard as handling other scripts is for people who only use the Latin alphabet 😭 Plenty of languages with non-Latin scripts get transcribed with Latin letters, and those transcriptions are now used all over URIs. The catch is that they introduce extra ambiguities 🙈
The good news is that local scripts are now widely supported in local versions of operating system and application software. Software that can handle a wide variety of scripts and languages at the same time is getting more and more common 🌈 Plus, more and more protocols and formats can carry a wide range of characters 💜
This document defines a new protocol element called the Internationalized Resource Identifier (IRI) by extending the syntax of URIs to a much wider repertoire of characters 🦋 It also defines "internationalized" versions of other constructs from [RFC3986], such as URI references. The syntax of IRIs is in section 2, and the relationship between IRIs and URIs is in section 3.
Using characters outside of A - Z in IRIs does bring some difficulties 💀 Section 4 covers the special case of bidirectional IRIs, section 5 covers the various forms of equivalence between IRIs, and section 6 covers how IRIs get used in different situations. Section 7 adds extra informative guidelines, and section 8 is all about security considerations 🔐
1.2. Applicability
IRIs are designed to be compatible with recommendations for new URI schemes [RFC2718] ✨ That compatibility comes from a well-defined, deterministic mapping from the IRI character sequence to the functionally equivalent URI character sequence. Using IRIs (or IRI references) in place of URIs (or URI references) in practice depends on the following conditions being met:
- a. A protocol or format element should be explicitly designated to be able to carry IRIs. The intent is not to introduce IRIs into contexts that aren't defined to accept them. For example, XML schema [XMLSchema] has an explicit type "anyURI" that includes IRIs and IRI references. So IRIs and IRI references can be in attributes and elements of type "anyURI". On the other hand, in the HTTP protocol [RFC2616], the Request URI is defined as a URI, which means direct use of IRIs isn't allowed in HTTP requests 🙅♀️
- b. The protocol or format carrying the IRIs should have a mechanism to represent the wide range of characters used in IRIs, either natively or by some protocol- or format-specific escaping mechanism (for example, numeric character references in [XML1]) 🧩
- c. The URI corresponding to the IRI in question has to encode original characters into octets using UTF-8. For new URI schemes, this is recommended in [RFC2718]. It can apply to a whole scheme (e.g., IMAP URLs [RFC2192] and POP URLs [RFC2384], or the URN syntax [RFC2141]). It can apply to a specific part of a URI, such as the fragment identifier (e.g., [XPointer]). It can apply to a specific URI or part(s) thereof. For details, please see section 6.4. 🍓
1.3. Definitions
Okay, these are the definitions used in this document, and they follow the terms in [RFC2130], [RFC2277], and [ISO10646] 📖
- character: A member of a set of elements used for the organization, control, or representation of data. For example, "LATIN CAPITAL LETTER A" names a character. 🔠
- octet: An ordered sequence of eight bits considered as a unit. 🧃
- character repertoire: A set of characters (in the mathematical sense). 🌷
- sequence of characters: A sequence of characters (one after another). 🫧
- sequence of octets: A sequence of octets (one after another). 🎈
- character encoding: A method of representing a sequence of characters as a sequence of octets (maybe with variants). It's also a method of (unambiguously) converting a sequence of octets into a sequence of characters. 🔁
- charset: The name of a parameter or attribute used to identify a character encoding. 🏷️
- UCS: Universal Character Set. The coded character set defined by ISO/IEC 10646 [ISO10646] and the Unicode Standard [UNIV4]. 🌍
- IRI reference: Denotes the common usage of an Internationalized Resource Identifier. An IRI reference may be absolute or relative. However, the "IRI" that results from such a reference only includes absolute IRIs, so any relative IRI references get resolved to their absolute form. Note that in [RFC2396] URIs didn't include fragment identifiers, but in [RFC3986] fragment identifiers are part of URIs 🧭
- running text: Human text (paragraphs, sentences, phrases) with syntax according to the orthographic conventions of a natural language, as opposed to syntax defined for ease of processing by machines (e.g., markup, programming languages). 📝
- protocol element: Any portion of a message that affects processing of that message by the protocol in question. ⚙️
- presentation element: A presentation form corresponding to a protocol element; for example, using a wider range of characters. 🎨
- create (a URI or IRI): With respect to URIs and IRIs, the term is used for the initial creation. This may be the initial creation of a resource with a certain identifier, or the initial exposition of a resource under a particular identifier. 🐣
- generate (a URI or IRI): With respect to URIs and IRIs, the term is used when the IRI is generated by derivation from other information. ⭐
That's the whole first chunk of the spec, bestie, and we're just getting started 🌈💖
1.4. Notation
okay so RFCs and Internet Drafts currently can't use anything outside US-ASCII 🙈 so this doc has to use some special notations to show those characters in examples 💅
in regular text, we sometimes point at a non-ASCII character with a U+ prefix followed by four to six hex digits, like U+00A2 ✨
for examples, this doc uses two notations: XML Notation and Bidi Notation 🎀
XML Notation starts with &#x, ends with ;, and has the character's hex number in the UCS in the middle. so я means CYRILLIC CAPITAL LETTER YA 🌸 and to show an actual & you write & 🫶
Bidi Notation is for bidirectional examples. lowercase letters stand for Latin letters (or other left-to-right letters), and uppercase letters stand for Arabic or Hebrew letters that go right to left 🔥
and for actual octets in examples (not percent-encoded ones), the two hex digits go between < and >. so the octet people often write as 0xc9 shows up here as <c9> 🦋
now the key words!! in this document, "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are to be read as described in [RFC2119] 👑
2. IRI Syntax
this section defines the syntax of Internationalized Resource Identifiers (IRIs) 🧚
like URIs, an IRI is a sequence of characters, not a sequence of octets. this is because IRIs might get written on paper or read out over the radio, not just stored or sent digitally 📝 the same IRI can show up as different octet sequences in different protocols or docs if they use different character encodings (and/or transfer encodings). using the same character encoding as the containing protocol or doc means the characters in the IRI can get handled (searched, converted, displayed) the same way as everything else in there ✨
2.1. Summary of IRI Syntax
IRIs are defined a lot like URIs in [RFC3986], except the unreserved class gets extended by adding UCS characters (Universal Character Set, [ISO10646]) beyond U+007F. there are limits on that, given in the syntax rules below and in section 6.1 🌈
otherwise the syntax and the use of components and reserved characters is the same as in [RFC3986]. all the operations defined in [RFC3986], like resolving relative references, can be applied to IRIs by IRI-processing software in exactly the same way as URI-processing software does for URIs 💅
here's the big rule!! characters outside US-ASCII are not reserved, so they MUST NOT be used for syntactical purposes, like delimiting components in newly defined schemes 🚫 for example, U+00A2, CENT SIGN, can't be a delimiter in IRIs, because it's in the iunreserved category. it's kind of like how you can't use - as a delimiter in URIs, since it's in the unreserved category 💀
2.2. ABNF for IRI References and IRIs
it might be possible to define IRI references and IRIs just by transforming them into URI references and URIs, but they can also be accepted and processed directly. so there's an ABNF definition for IRI references (the most general concept and the start of the grammar) and IRIs right here 🦋 the ABNF syntax is described in [RFC2234]. character numbers come from the UCS without implying any actual binary encoding, and terminals in the ABNF are characters, not bytes 🔥
the grammar closely follows the URI grammar in [RFC3986], except the range of unreserved characters expands to include UCS characters, with the restriction that private UCS characters can only show up in query parts 🎀 the grammar is split into two parts: rules that differ from [RFC3986] because of that expansion, and rules that are the same as in [RFC3986]. for the rules that differ, the non-terminal names changed like this: if the name contains 'URI', that becomes 'IRI'. otherwise, an 'i' gets prefixed 🌸
the following rules are different from those in [RFC3986]:
IRI = scheme ":" ihier-part [ "?" iquery ]
[ "#" ifragment ]
ihier-part = "//" iauthority ipath-abempty
/ ipath-absolute
/ ipath-rootless
/ ipath-empty
IRI-reference = IRI / irelative-ref
absolute-IRI = scheme ":" ihier-part [ "?" iquery ]
irelative-ref = irelative-part [ "?" iquery ] [ "#" ifragment ]
irelative-part = "//" iauthority ipath-abempty
/ ipath-absolute
/ ipath-noscheme
/ ipath-empty
iauthority = [ iuserinfo "@" ] ihost [ ":" port ]
iuserinfo = *( iunreserved / pct-encoded / sub-delims / ":" )
ihost = IP-literal / IPv4address / ireg-name
ireg-name = *( iunreserved / pct-encoded / sub-delims )
ipath = ipath-abempty ; begins with "/" or is empty
/ ipath-absolute ; begins with "/" but not "//"
/ ipath-noscheme ; begins with a non-colon segment
/ ipath-rootless ; begins with a segment
/ ipath-empty ; zero characters
ipath-abempty = *( "/" isegment )
ipath-absolute = "/" [ isegment-nz *( "/" isegment ) ]
ipath-noscheme = isegment-nz-nc *( "/" isegment )
ipath-rootless = isegment-nz *( "/" isegment )
ipath-empty = 0<ipchar>
isegment = *ipchar
isegment-nz = 1*ipchar
isegment-nz-nc = 1*( iunreserved / pct-encoded / sub-delims
/ "@" )
; non-zero-length segment without any colon ":"
ipchar = iunreserved / pct-encoded / sub-delims / ":"
/ "@"
iquery = *( ipchar / iprivate / "/" / "?" )
ifragment = *( ipchar / "/" / "?" )
iunreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar
ucschar = %xA0-D7FF / %xF900-FDCF / %xFDF0-FFEF
/ %x10000-1FFFD / %x20000-2FFFD / %x30000-3FFFD
/ %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD
/ %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD
/ %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD
/ %xD0000-DFFFD / %xE1000-EFFFD
iprivate = %xE000-F8FF / %xF0000-FFFFD / %x100000-10FFFDsome productions are ambiguous 🙃 the "first-match-wins" (a.k.a. "greedy") algorithm applies here. for details, check [RFC3986] 🧠
the following rules are the same as those in [RFC3986]:
scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
port = *DIGIT
IP-literal = "[" ( IPv6address / IPvFuture ) "]"
IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
IPv6address = 6( h16 ":" ) ls32
/ "::" 5( h16 ":" ) ls32
/ [ h16 ] "::" 4( h16 ":" ) ls32
/ [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
/ [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
/ [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
/ [ *4( h16 ":" ) h16 ] "::" ls32
/ [ *5( h16 ":" ) h16 ] "::" h16
/ [ *6( h16 ":" ) h16 ] "::"
h16 = 1*4HEXDIG
ls32 = ( h16 ":" h16 ) / IPv4address
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
dec-octet = DIGIT ; 0-9
/ %x31-39 DIGIT ; 10-99
/ "1" 2DIGIT ; 100-199
/ "2" %x30-34 DIGIT ; 200-249
/ "25" %x30-35 ; 250-255
pct-encoded = "%" HEXDIG HEXDIG
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="and last but not least, this syntax does not support IPv6 scoped addressing zone identifiers 🌙
3. Relationship between IRIs and URIs
IRIs are meant to replace URIs when identifying resources for protocols, formats, and software components that use a UCS-based character repertoire ✨ those protocols and components might never need to touch URIs directly, especially when the resource identifier is only used for identification 🫶 but when it's used for resource retrieval, it's often necessary to figure out the associated URI, because most retrieval mechanisms are currently only defined for URIs 🥹 in that case, IRIs can act as presentation elements for URI protocol elements. an example would be the address bar in a Web user agent 🌐 (more rationale is in section 3.1, bestie 💖)
3.1. Mapping of IRIs to URIs ✨
okay so this section is all about how to turn an IRI into a URI, and it's giving very precise instructions 💅 Everything in here also applies to IRI references and URI references, plus their pieces, like fragment identifiers 🧚
This mapping has two purposes, bestie 🎀
Syntaxical. Lots of URI schemes and components have extra syntax rules that aren't captured in section 2.2. To apply those scheme-specific rules to an IRI, you convert the IRI to a URI and then check that URI against the rules 🔍
Interpretational. URIs identify resources in different ways, and IRIs identify resources too ✨ When an IRI is only used for identification, you don't have to map it to a URI (see section 5). But when an IRI is used to retrieve a resource, the resource it locates is the same one the URI locates after you convert the IRI using the procedure here 🦋 So there's no need to define resolution separately at the IRI level, which is honestly so clean 🫶
Applications MUST map IRIs to URIs using the following two steps 👑
Step 1. Generate a UCS character sequence from the original IRI format. This step has three variants, depending on the form of the input:
- a. If the IRI is written on paper, read aloud, or otherwise represented as a sequence of characters independent of any character encoding, represent the IRI as a sequence of characters from the UCS normalized according to Normalization Form C (NFC, [UTR15]).
- b. If the IRI is in some digital representation (e.g., an octet stream) in some known non-Unicode character encoding, convert the IRI to a sequence of characters from the UCS normalized according to NFC.
- c. If the IRI is in a Unicode-based character encoding (for example, UTF-8 or UTF-16), do not normalize (see section 5.3.2.2 for details). Apply step 2 directly to the encoded Unicode character sequence.
Step 2. For each character in ucschar or iprivate, apply steps 2.1 through 2.3 below.
- 2.1. Convert the character to a sequence of one or more octets using UTF-8 [RFC3629].
- 2.2. Convert each octet to
%HH, whereHHis the hexadecimal notation of the octet value. Note that this is identical to the percent-encoding mechanism in section 2.1 of [RFC3986]. To reduce variability, the hexadecimal notation SHOULD use uppercase letters 🌸 - 2.3. Replace the original character with the resulting character sequence (i.e., a sequence of
%HHtriplets).
The mapping above from IRIs to URIs produces URIs that fully conform to [RFC3986] 🎉 It's also an identity transformation for URIs and it's idempotent, meaning applying it a second time won't change anything. Every URI is by definition an IRI, so it all clicks together 💖
Systems accepting IRIs MAY convert the ireg-name component of an IRI as follows (before step 2 above) for schemes known to use domain names in ireg-name, if the scheme definition does not allow percent-encoding for ireg-name:
Replace the ireg-name part of the IRI with the part converted using the ToASCII operation specified in section 4.1 of [RFC3490] on each dot-separated label, using U+002E (FULL STOP) as a label separator. The flag UseSTD3ASCIIRules is set to TRUE, and the flag AllowUnassigned is set to FALSE when creating IRIs and set to TRUE otherwise 🧚
The ToASCII operation may fail, but that would mean the IRI cannot be resolved 😭 This conversion SHOULD be used when the goal is to maximize interoperability with legacy URI resolvers. For example, this IRI:
"http://résumé.example.org"may be converted to:
"http://xn--rsum-bpad.example.org"instead of:
"http://r%C3%A9sum%C3%A9.example.org"An IRI with a scheme that is known to use domain names in ireg-name, but where the scheme definition does not allow percent-encoding for ireg-name, meets scheme-specific restrictions if either the straightforward conversion or the conversion using the ToASCII operation on ireg-name results in a URI that meets those restrictions 🦋
Such an IRI resolves to the URI obtained after converting the IRI and using the ToASCII operation on ireg-name. Implementations don't have to do this conversion, as long as they produce the same result 💅
Note: The difference between variants b and c in step 1 (using normalization with NFC, versus not using any normalization) accounts for the fact that in many non-Unicode character encodings, some text cannot be represented directly. For example, the word "Vietnam" is natively written "Việt Nam" (containing a LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW) in NFC, but a direct transcoding from the windows-1258 character encoding leads to "Việt Nam" (containing a LATIN SMALL LETTER E WITH CIRCUMFLEX followed by a COMBINING DOT BELOW). Direct transcoding of other 8-bit encodings of Vietnamese may lead to other representations.
Note: The uniform treatment of the whole IRI in step 2 is important to make processing independent of URI scheme. See [Gettys] for an in-depth discussion.
Note: In practice, whether the general mapping (steps 1 and 2) or the ToASCII operation of [RFC3490] is used for ireg-name won't be noticed if mapping from IRI to URI and resolution are tightly integrated (e.g., carried out in the same user agent). But conversion using [RFC3490] may be able to better deal with backwards compatibility issues in case mapping and resolution are separated, as in the case of using an HTTP proxy.
Note: Internationalized Domain Names may be contained in parts of an IRI other than theireg-namepart. It is the responsibility of scheme-specific implementations (if the Internationalized Domain Name is part of the scheme syntax) or of server-side implementations (if the Internationalized Domain Name is part ofiquery) to apply the necessary conversions at the appropriate point. Example: Trying to validate the Web page at http://résumé.example.org would lead to an IRI of
http://validator.w3.org/check?uri=http%3A%2F%2Frésumé.example.orgwhich would convert to a URI of
http://validator.w3.org/check?uri=http%3A%2F%2Fr%C3%A9sum%C3%A9.example.orgThe server-side implementation would be responsible for making the necessary conversions to be able to retrieve the Web page 🌈
Systems accepting IRIs MAY also deal with the printable characters in US-ASCII that are not allowed in URIs, namely "<", ">", '"', space, "{", "}", "|", "\", "^", and "`", in step 2 above. If these characters are found but are not converted, then the conversion SHOULD fail. Please note that the number sign ("#"), the percent sign ("%"), and the square bracket characters ("[", "]") are not part of the above list and MUST NOT be converted 🙈 Protocols and formats that have used earlier definitions of IRIs including these characters MAY require percent-encoding of these characters as a preprocessing step to extract the actual IRI from a given field. This preprocessing MAY also be used by applications that allow the user to enter an IRI.
Note: In this process (in step 2.3), characters allowed in URI references and existing percent-encoded sequences are not encoded further. (This mapping is similar to, but different from, the encoding applied when arbitrary content is included in some part of a URI.) For example, an IRI of"http://www.example.org/red%09rosé#red"(in XML notation) is converted to"http://www.example.org/red%09ros%C3%A9#red", not to something like"http%3A%2F%2Fwww.example.org%2Fred%2509ros%C3%A9%23red".
Note: Some older software transcoding to UTF-8 may produce illegal output for some input, in particular for characters outside the BMP (Basic Multilingual Plane). As an example, for the IRI with non-BMP characters (in XML notation)"http://example.com/𐌀𐌁𐌂", which contains the first three letters of the Old Italic alphabet, the correct conversion to a URI is"http://example.com/%F0%90%8C%80%F0%90%8C%81%F0%90%8C%82"😭✨
3.2. Converting URIs to IRIs 🦋
sometimes you want to turn a URI into an equivalent IRI, and this section gives you the procedure for that 🧚 The conversion here will always produce an IRI that maps back to the URI you started with (except for possible case differences in percent-encoding and for percent-encoded unreserved characters). But the resulting IRI might not be exactly the same as the original IRI, if there ever was one 💅
URI-to-IRI conversion removes percent-encodings, but not all of them can be eliminated. There are a few reasons why:
- Some percent-encodings are necessary to distinguish percent-encoded and unencoded uses of reserved characters.
- Some percent-encodings cannot be interpreted as sequences of UTF-8 octets.
(Note: The octet patterns of UTF-8 are highly regular. Therefore, there is a very high probability, but no guarantee, that percent-encodings that can be interpreted as sequences of UTF-8 octets actually originated from UTF-8. For a detailed discussion, see [Duerst97].)
- The conversion may result in a character that is not appropriate in an IRI. See sections 2.2, 4.1, and 6.1 for further details.
Conversion from a URI to an IRI is done using the following steps (or any other algorithm that produces the same result) 🎀
- Represent the URI as a sequence of octets in US-ASCII.
- Convert all percent-encodings ("%" followed by two hexadecimal digits) to the corresponding octets, except those corresponding to "%", characters in "reserved", and characters in US-ASCII not allowed in URIs.
- Re-percent-encode any octet produced in step 2 that is not part of a strictly legal UTF-8 octet sequence.
- Re-percent-encode all octets produced in step 3 that in UTF-8 represent characters that are not appropriate according to sections 2.2, 4.1, and 6.1.
- Interpret the resulting octet sequence as a sequence of characters encoded in UTF-8.
This procedure converts as many percent-encoded characters as possible into characters in an IRI 🌸 Because there are some choices when step 4 is applied (see section 6.1), results may vary.
Conversions from URIs to IRIs MUST NOT use any character encoding other than UTF-8 in steps 3 and 4, even if it might be possible to guess from the context that another character encoding than UTF-8 was used in the URI 🙅♀️ For example, the URI "http://www.example.org/r%E9sum%E9.html" might, with some guessing, be interpreted to contain two e-acute characters encoded as iso-8859-1. It must not be converted to an IRI containing these e-acute characters. Otherwise, in the future the IRI will be mapped to "http://www.example.org/r%C3%A9sum%C3%A9.html", which is a different URI from "http://www.example.org/r%E9sum%E9.html" 😭💜
3.2.1. Examples 🎀
okay bestie, let's see the URI-to-IRI conversions in action! 🦋 each example shows what happens after steps 1 through 5, and the final result is written in XML notation. octets get written as "<" + two hex digits + ">", so <c3> just means that byte. 💅
first up, "%C3%BC" is a strictly legal UTF-8 sequence, so it gets turned into the real character U+00FC, LATIN SMALL LETTER U WITH DIAERESIS (aka u-umlaut). ✨ slay 💖
- http://www.example.org/D%C3%BCrst
- http://www.example.org/D<c3><bc>rst
- http://www.example.org/D<c3><bc>rst
- http://www.example.org/D<c3><bc>rst
- http://www.example.org/Dürst
next, "%FC" might mean U+00FC in iso-8859-1, but it could mean other stuff in other encodings! 🌸 for example, the octet <fc> in iso-8859-5 is U+045C, CYRILLIC SMALL LETTER KJE. 🧚 since <fc> isn't part of a strictly legal UTF-8 sequence, it gets re-percent-encoded in step 3. no cap, that's the whole plot twist 😭
- http://www.example.org/D%FCrst
- http://www.example.org/D<fc>rst
- http://www.example.org/D%FCrst
- http://www.example.org/D%FCrst
- http://www.example.org/D%FCrst
ok now this one is giving main character energy 👑 "%e2%80%ae" is the percent-encoded UTF-8 for U+202E, RIGHT-TO-LEFT OVERRIDE. section 4.1 forbids using that character directly in an IRI, so those octets get re-percent-encoded in step 4. 🙈 this example also shows that the case of letters in percent-encodings may not be preserved (see how lowercase turns into uppercase 👀). the punycode label xn--99zt52a isn't converted here, btw 🫶
- http://xn--99zt52a.example.org/%e2%80%ae
- http://xn--99zt52a.example.org/<e2><80><ae>
- http://xn--99zt52a.example.org/<e2><80><ae>
- http://xn--99zt52a.example.org/%E2%80%AE
- http://xn--99zt52a.example.org/%E2%80%AE
implementations with scheme-specific knowledge MAY convert punycode-encoded domain name labels to the matching characters using the ToUnicode procedure. 🔥 so for the example above, the label "xn--99zt52a" MAY become U+7D0D U+8C46 (Japanese Natto), giving the full IRI "http://納豆.example.org/%E2%80%AE". it's so iconic when a domain gets its own glow-up 🍓✨
- Bidirectional IRIs for Right-to-Left Languages 🌈
okay so some UCS characters, like the ones in Arabic and Hebrew, are inherently right-to-left (rtl) 🌙 IRIs with these characters (called bidirectional IRIs or Bidi IRIs) need extra attention, because the logical order (used for storage and spelling) and the visual order (used for display and printing) aren't simply the same thing 🙃
since the logical representation, the visual representation, and the IRI syntax all interact in tricky ways, we need to balance a few goals. the main requirements are:
- user-predictable conversion between visual and logical representation;
- the ability to include a wide range of characters in various parts of the IRI; and
- minor or no changes or restrictions for implementations.
4.1. Logical Storage and Visual Presentation 💫
when bidirectional IRIs are stored or transmitted in digital form, they MUST be in full logical order and MUST conform to the IRI syntax rules (including the rules for their scheme). 💅 this way, Bidi IRIs get processed the same way as every other IRI, periodt.
bidirectional IRIs MUST be rendered with the Unicode Bidirectional Algorithm [UNIV4], [UNI9]. they MUST be rendered the same way they would be inside a left-to-right embedding, i.e. as if they were preceded by U+202A, LEFT-TO-RIGHT EMBEDDING (LRE), and followed by U+202C, POP DIRECTIONAL FORMATTING (PDF). 🎀 the embedding direction can also be set by a higher-level protocol (for example, the dir='ltr' attribute in HTML).
there's no need to use that embedding if the display stays the same without it. for example, a bidirectional IRI in text with left-to-right base directionality (like English or Cyrillic) that's surrounded by whitespace and strong left-to-right characters doesn't need one. 🦋 likewise, a bidirectional relative IRI reference that only has strong right-to-left and weak characters, starts and ends with a strong right-to-left character, sits in text with right-to-left base directionality (like Arabic or Hebrew), and is surrounded by whitespace and strong characters, doesn't need an embedding either ✨
in some other cases, U+200E, LEFT-TO-RIGHT MARK (LRM), may be enough to get the right display 🧚 but the Unicode Bidirectional algorithm is honestly kind of a lot 💀 so implementers are strongly advised to err on the side of caution and use embedding in every case where they aren't completely sure the display is unaffected without it. 🫶
the Unicode Bidirectional Algorithm ([UNI9], section 4.3) lets higher-level protocols influence bidirectional rendering. those changes MUST NOT be used if they change how IRIs render 🙅♀️
the bidirectional formatting characters you might use before or after an IRI to get the right display aren't part of the IRI itself 🌸 IRIs MUST NOT contain bidirectional formatting characters (LRM, RLM, LRE, RLE, LRO, RLO, and PDF). they change how the IRI looks but never show up themselves, so it would be impossible to type an IRI with them in it correctly 😭
4.2. Bidi IRI Structure 🧩
the Unicode Bidirectional Algorithm is mainly built for running text, so to keep it from messing up Bidi IRIs too much, we need some restrictions. these are described in terms of delimiters (structural punctuation like "@", ".", ":", and "/") and components (usually made of letters and digits) 💖
the following syntax rules from section 2.2 count as components for Bidi behavior: iuserinfo, ireg-name, isegment, isegment-nz, isegment-nz-nc, ireg-name, iquery, and ifragment. 🍓
specifications that define the syntax of any of those components MAY split them further and define smaller parts to be components under this document. for instance, the rules of [RFC3490] on bidirectional domain names treat each label of a domain name as a component for schemes where ireg-name is a domain name. even when components aren't formally defined, it can help to think of some syntax in those terms and apply the restrictions. for example, for the usual name/value syntax in query parts, it's convenient to treat each name and each value as a component. 🌈 and the extensions in a resource name can be treated as separate components too 🎀
for each component, these restrictions apply:
- A component SHOULD NOT use both right-to-left and left-to-right characters.
- A component using right-to-left characters SHOULD start and end with right-to-left characters.
these restrictions are written as SHOULDs, not MUSTs. for IRIs that are never shown visually, they don't matter. but for IRIs in general, they're super important for keeping conversion between visual presentation and logical representation consistent in both directions 🔥
Note: In some components, the above restrictions may actually be strictly enforced. For example, [RFC3490] requires that these restrictions apply to the labels of a host name for those schemes where ireg-name is a host name. In some other components (for example, path components) following these restrictions may not be too difficult. For other components, such as parts of the query part, it may be very difficult to enforce the restrictions because the values of query parameters may be arbitrary character sequences.
if the restrictions above can't be satisfied any other way, the affected component can always be mapped to URI notation as described in section 3.1. 🦋 just note that the whole component has to be mapped (see also Example 9 below) 💅
4.3. Input of Bidi IRIs ⌨️
Bidi input methods MUST generate Bidi IRIs in logical order while rendering them according to section 4.1. 🫶 during input, rendering SHOULD be updated after every new character is typed so end-users don't get confused 💜
4.4. Examples 🎀
Okay bestie, these are the bidi examples and they're giving chaos in the best way 🌈. Each one shows a legal IRI with its logical and visual representations, so you can see how things that look weird to people who don't read Arabic or Hebrew actually make total sense to people who do 💅. There's also a peek at what happens when you break the rules from section 4.2 🙈. You can find the examples at [BidiEx] in Arabic, Hebrew, and Bidi Notation versions ✨.
To read the bidi text, go left to right through the visual version until you hit a block of rtl text. Then read that whole block right to left, slashes and special characters included, and pick up again at the next unread ltr character 🦋.
Example 1: A single component with rtl characters is inverted 💖
Logical representation: "http://ab.CDEFGH.ij/kl/mn/op.html"
Visual representation: "http://ab.HGFEDC.ij/kl/mn/op.html"You can read the components one at a time, and each one reads in its own natural direction 🧚.
Example 2: More than one consecutive component with rtl characters is inverted as a whole 🔥
Logical representation: "http://ab.CDE.FGH/ij/kl/mn/op.html"
Visual representation: "http://ab.HGF.EDC/ij/kl/mn/op.html"A run of rtl components gets read rtl, same as a run of rtl words in regular bidi text. Very consistent, very iconic 👑.
Example 3: All components of an IRI (except for the scheme) are rtl. All rtl components are inverted overall 🍓
Logical representation: "http://AB.CD.EF/GH/IJ/KL?MN=OP;QR=ST#UV"
Visual representation: "http://VU#TS=RQ;PO=NM?LK/JI/HG/FE.DC.BA"The whole IRI, minus the scheme, gets read rtl 😭. Delimiters between rtl components stay between their own components, and delimiters between ltr and rtl components don't move at all 💜.
Example 4: Each of several sequences of rtl components is inverted on its own 🌸
Logical representation: "http://AB.CD.ef/gh/IJ/KL.html"
Visual representation: "http://DC.BA.ef/gh/LK/JI.html"Every rtl sequence gets read rtl on its own, just like each rtl word sequence in an ltr text 🫶.
Example 5: Example 2, applied to components of different kinds ✨
Logical representation: "http://ab.cd.EF/GH/ij/kl.html"
Visual representation: "http://ab.cd.HG/FE/ij/kl.html"Okay, the domain label and the path getting flipped might feel unexpected, but it's totally consistent with the rest of bidi behavior 🙈. If you want reassurance that the domain really is "ab.cd.EF", read the visual version out loud with the bidi algorithm in mind. After "http://ab.cd." you read the RTL block "E-F-slash-G-H", and that matches the logical version 🧠.
Example 6: Same as Example 5, with more rtl components 🎀
Logical representation: "http://ab.CD.EF/GH/IJ/kl.html"
Visual representation: "http://ab.JI/HG/FE.DC/kl.html"The flipped domain labels and path components are easier to spot here, because the delimiters move too 🦋.
Example 7: A single rtl component includes digits 💅
Logical representation: "http://ab.CDE123FGH.ij/kl/mn/op.html"
Visual representation: "http://ab.HGF123EDC.ij/kl/mn/op.html"Numbers always get written ltr, but the bidi algorithm treats them as an extra embedding inside a run of rtl characters. This is totally normal bidi text, so no cap 💖.
Example 8 (not allowed): Numbers are at the start or end of an rtl component 💀
Logical representation: "http://ab.cd.ef/GH1/2IJ/KL.html"
Visual representation: "http://ab.cd.ef/LK/JI1/2HG.html"Here the sequence "1/2" gets read by the bidi algorithm as a fraction, which splits up the components and causes a whole mess 😭. There are other characters that get special treatment near numbers too, specifically "+", "-", "#", "$", "%", ",", ".", and ":" 🙈.
Example 9 (not allowed): The numbers in the previous example are percent-encoded 🌈
Logical representation: "http://ab.cd.ef/GH%31/%32IJ/KL.html",
Visual representation (Hebrew): "http://ab.cd.ef/%31HG/LK/JI%32.html"
Visual representation (Arabic): "http://ab.cd.ef/31%HG/%LK/JI32.html"Whether the uppercase letters are Arabic or Hebrew changes what the visual version looks like, so the same logical IRI can end up looking different depending on the script 👑.
Example 10 (allowed but not recommended):
Logical representation: "http://ab.CDEFGH.123/kl/mn/op.html"
Visual representation: "http://ab.123.HGFEDC/kl/mn/op.html"Components made up of only numbers are allowed, because banning them would be pretty hard, but they can interact with neighboring rtl components in ways that are hard to predict 🍓.
5. Normalization and Comparison 🔥
Note: The structure and much of the material for this section is taken from section 6 of [RFC3986]; the differences are due to the specifics of IRIs.
Comparing IRIs is one of the most common things that happens with them 💖. It's about figuring out whether two IRIs are equivalent without actually using the IRIs, or the URIs they map to, to reach their resources 🦋. Comparisons happen all the time: when a response cache gets accessed, when a browser checks its history to color a link, or when an XML parser handles tags inside a namespace ✨. Spiders and indexing engines might do a lot of normalization before comparing, so they can prune a search space or cut down on duplicate requests and stored responses 🧚.
Comparison is always done for some specific purpose 🎀. Protocols or implementations that compare IRIs for different reasons will usually have different trade-offs about how much effort they put into reducing aliased identifiers 💅. This section walks through the different ways IRIs can be compared, the trade-offs between them, and the kinds of applications that might use each one 👑.
5.1. Equivalence 🌸
Since IRIs exist to identify resources, you'd think they should count as equivalent whenever they identify the same resource 🫶. But that definition isn't super useful in practice, because an implementation can't compare two resources unless it has full knowledge or control of them 🙈. So deciding whether IRIs are equivalent or different is based on string comparison, maybe with some extra rules from URI scheme definitions 🧠. We use the terms "different" and "equivalent" to describe the possible results of these comparisons, but there are lots of application-dependent versions of equivalence 💜.
Even when you can tell that two IRIs are equivalent, comparing IRIs still isn't enough to tell whether they identify different resources 😭. For example, one owner of two different domain names could decide to serve the same resource from both, which gives you two different IRIs 💀. That's why comparison methods are designed to minimize false negatives while strictly avoiding false positives 🎀.
When you're testing for equivalence, applications should not directly compare relative references 🦋. Instead, convert the references to their respective target IRIs before comparing ✨. When IRIs are compared to select (or avoid) a network action, such as retrieving a representation, fragment components (if any) should be excluded from the comparison 🍓.
Applications that use IRIs as identity tokens with no relationship to a protocol MUST use the Simple String Comparison (see section 5.3.1). All other applications MUST select one of the comparison practices from the Comparison Ladder (see section 5.3 or, after IRI-to-URI conversion, select one of the comparison practices from the URI comparison ladder in [RFC3986], section 6.2) 👑
5.2. Preparation for Comparison 💖
Any kind of IRI comparison REQUIRES that all escapings or encodings in the protocol or format that carries an IRI are resolved 🌈. This usually happens when the protocol or format gets parsed 🫶. Examples of these escapings or encodings are entities and numeric character references in [HTML4] and [XML1] ✨. For instance, "http://example.org/rosé" (in HTML), "http://example.org/rosé" (in HTML or XML), and "http://example.org/rosé" (in HTML or XML) all get resolved into what this document denotes (see section 1.4) as "http://example.org/rosé" (the "é" here stands for the actual e-acute character, since this document can't contain non-ASCII characters) 🦋.
Similar considerations apply to encodings like Transfer Codings in HTTP (see [RFC2616]) and Content Transfer Encodings in MIME ([RFC2045]) 🧚. In these cases, though, the encoding is based on octets instead of characters, so you need extra care to make sure that characters, and not just arbitrary octets, are compared (see section 5.3.1) 🔥.
5.3. Comparison Ladder 🧁
In practice, lots of different methods get used to test IRI equivalence 🎀. They fall along a range based on how much processing they need and how much they reduce the chance of false negatives 💅. Like we said above, false negatives can't be eliminated 🙈. In practice their probability can be reduced, but that takes more processing, and it isn't cost-effective for every application 💜.
Think of this range of comparison practices as a ladder 🪜. The discussion below climbs that ladder, starting with practices that are cheap but have a relatively higher chance of false negatives, and moving up to practices with higher computational cost and lower risk of false negatives 👑✨.
5.3.1. Simple String Comparison ✨
okay so if two IRIs are identical when you look at them as plain character strings, then it's safe to conclude they're equivalent 💖 this is the cheapest test there is, it's used everywhere (especially in parsing), and it gives a definite answer without needing the scheme or touching the network 🧚. XML Namespaces ([XMLNamespace]) is a super clear example of where this is exactly what you want.
Testing strings for equality still needs a few basic precautions 🫶. People call this a "bit-for-bit" or "byte-for-byte" comparison, and ngl that name is kind of misleading 🙈. Equality is normally checked by comparing pairs of characters, starting at the first one and moving on until both strings have run out and every pair matched, until one pair doesn't match, or until one string runs out before the other.
Each pair of characters has to be in a comparable encoding form first 😭. Say one IRI is stored as UTF-8 bytes and the other is UTF-16. If you compare those bytes naively, you'll get errors, full stop 💀. So it's better to talk about equality character-for-character instead of byte-for-byte or bit-for-bit 🌸. In practice, character-by-character comparisons should be done codepoint by codepoint after converting to a common character encoding form. And when you're comparing character by character, the comparison function MUST NOT map IRIs to URIs, because that mapping would create additional spurious equivalences 🚫. Which means an IRI SHOULD NOT be modified when it's being transported, if there's any chance it might get used as an identifier 💅.
False negatives come from IRI aliases being produced and used 🙃. You can cut down on the unnecessary aliases, no matter which comparison method you use, by consistently giving IRI references in an already normalized form (basically, the exact form you'd get after normalization, which we'll get to below) ✨. Protocols and data formats often limit some IRI comparisons to plain string comparison. The theory is that people and implementations will, in their own best interest, be consistent when writing IRI references, or at least consistent enough that any efficiency you'd get from extra normalization just isn't worth it 🎀.
5.3.2. Syntax-Based Normalization
implementations may use logic based on the definitions in this specification to reduce the probability of false negatives 🦋. This processing costs a bit more than character-for-character string comparison, but it's still moderate. For example, an app using this approach could reasonably consider these two IRIs equivalent:
example://a/b/c/%7Bfoo%7D/rosé
eXAMPLE://a/./b/../b/%63/%7bfoo%7d/ros%C3%A9web user agents, like browsers, typically apply this kind of IRI normalization when they're deciding whether a cached response is available 🌐. Syntax-based normalization includes techniques like case normalization, character normalization, percent-encoding normalization, and removal of dot-segments. We're about to go through all of them, bestie 👑.
5.3.2.1. Case Normalization
for all IRIs, the hexadecimal digits inside a percent-encoding triplet (like "%3a" versus "%3A") are case-insensitive, so they should be normalized to uppercase letters for the digits A - F 🔠.
when an IRI uses components of the generic syntax, the component syntax equivalence rules always apply. Basically, the scheme and US-ASCII-only host are case-insensitive, so they should be normalized to lowercase 💜. For example, the URI "HTTP://www.EXAMPLE.com/" is equivalent to "http://www.example.com/". Case equivalence for non-ASCII characters in IRI components that are IDNs is discussed in section 5.3.3 ✨. The other generic syntax components are assumed to be case sensitive, unless the scheme specifically defines them otherwise 🌈.
creating schemes that allow case-insensitive syntax components with non-ASCII characters should be avoided 🙅. Case normalization of non-ASCII characters can depend on culture and is always a complex operation 😵💫. The only exception is non-ASCII host names, where character normalization includes a mapping step derived from case folding 🫶.
5.3.2.2. Character Normalization
the Unicode Standard [UNIV4] defines various equivalences between sequences of characters for different purposes 📚. Unicode Standard Annex #15 [UTR15] defines the Normalization Forms for these equivalences. The big ones are Normalization Form C (NFC, Canonical Decomposition followed by Canonical Composition) and Normalization Form KC (NFKC, Compatibility Decomposition followed by Canonical Composition) 🧠.
equivalence of IRIs MUST rely on the assumption that IRIs are already pre-character-normalized, rather than applying character normalization while comparing two IRIs 🎯. The exceptions are conversion from a non-digital form, and conversion from a non-UCS-based character encoding to a UCS-based one. In those cases, NFC or a normalizing transcoder using NFC MUST be used for interoperability 🤝. To avoid false negatives and transcoding problems, IRIs SHOULD be created using NFC. Using NFKC may avoid even more problems. For example, you can pick half-width Latin letters instead of full-width ones, and full-width Katakana instead of half-width 🎀.
as an example, "http://www.example.org/résumé.html" (in XML Notation) is in NFC ✅. On the other hand, "http://www.example.org/résumé.html" is not in NFC ❌.
The first one uses precombined e-acute characters. The second one uses "e" characters followed by combining acute accents. Both usages are defined as canonically equivalent in [UNIV4] 🦋.
Note: Because it's unknown how a particular sequence of characters is being treated with respect to character normalization, it would be inappropriate to let third parties normalize an IRI arbitrarily. This doesn't contradict the recommendation that when a resource is created, its IRI should be as character normalized as possible (i.e., NFC or even NFKC). It's similar to the uppercase/lowercase problems. Some parts of a URI are case insensitive (domain name). For others, it's unclear whether they're case sensitive, case insensitive, or something in between (e.g., case sensitive, but with a multiple choice selection if the wrong case is used, instead of a direct negative result). The best recipe is that the creator uses reasonable capitalization and, when transferring the URI, the capitalization never gets changed 🫠.
various IRI schemes may allow Internationalized Domain Names (IDN) [RFC3490] either in the ireg-name part or somewhere else. Character normalization also applies to IDNs, as discussed in section 5.3.3 🌍.
5.3.2.3. Percent-Encoding Normalization
the percent-encoding mechanism (section 2.1 of [RFC3986]) is a frequent source of variance among IRIs that are otherwise identical 😤. On top of the case normalization issue we just covered, some IRI producers percent-encode octets that don't need it, which gives IRIs that are equivalent to their non-encoded counterparts 🙄. These IRIs should be normalized by decoding any percent-encoded octet sequence that corresponds to an unreserved character, as described in section 2.3 of [RFC3986] 🧹.
for actual resolution, differences in percent-encoding (except for the percent-encoding of reserved characters) MUST always result in the same resource 🎯. For example, "http://example.org/~user", "http://example.org/%7euser", and "http://example.org/%7Euser" must all resolve to the same resource 💜.
if this kind of equivalence is going to be tested, the percent-encoding of both IRIs being compared has to be lined up. For example, you can convert both IRIs to URIs (see section 3.1), eliminate the escape differences in the resulting URIs, and make sure the hexadecimal characters in the percent-encoding always use the same case (uppercase is preferred) 🧁. If the IRI is going to be passed to another application or used further in some other way, its original form MUST be preserved 🔒. The conversion described here should only be performed for local comparison 🏠.
5.3.2.4. Path Segment Normalization
the complete path segments "." and ".." are only meant for use within relative references (section 4.1 of [RFC3986]). They're removed as part of the reference resolution process (section 5.2 of [RFC3986]) 🗑️. However, some implementations may wrongly assume that reference resolution isn't necessary when the reference is already an IRI, so they fail to remove dot-segments when those show up in non-relative paths 😬. IRI normalizers should remove dot-segments by applying the remove_dot_segments algorithm to the path, as described in section 5.2.4 of [RFC3986] 🦋✨.
5.3.3. Scheme-Based Normalization ✨
Okay so every scheme has its own rulebook (the defining spec for that scheme), which means IRIs can act differently depending on which scheme they belong to 💅. Implementations are allowed to use scheme-specific rules to cut down on false negatives, even though it costs extra processing 🧚. For example, "http" has an authority component, a default port of "80", and treats an empty path as "/", so these four IRIs are all equivalent 🌸:
http://example.com
http://example.com/
http://example.com:/
http://example.com:80/In general, an IRI that uses the generic authority syntax with an empty path should be normalized to a path of "/" 🦋. Same vibe with ":port": an explicit one where the port is empty or the scheme's default is basically the same as leaving the port and its ":" delimiter out, so scheme-based normalization should remove it 💜. That means the second IRI in the list is the normal form for "http" 👑.
Another spot where normalization changes by scheme is an empty authority or empty host subcomponent. For a lot of schemes that's an error, and for others it means "localhost" or the end-user's own host 🙈. When a scheme has a default for authority and you want to reference that default, the reference should be normalized to an empty authority for uniformity, brevity, and internationalization ✨. BUT if the userinfo or port subcomponents aren't empty, the host should be given explicitly even when it matches the default, because that's the main character move here 🎀.
Normalization should not remove delimiters when their component is empty, unless the scheme specification says it's allowed to 🌈. So an IRI like http://example.com/? can't be assumed equivalent to any of the examples above 💀. The presence or absence of delimiters inside a userinfo subcomponent is usually a big deal for how it's interpreted 🍓. The fragment component isn't touched by scheme-based normalization at all, so two IRIs that differ only by a trailing "#" are considered different no matter the scheme 🫶.
Some IRI schemes may allow Internationalized Domain Names (IDN) [RFC3490], either in their ireg-name part or somewhere else 🌸. When IDNs show up in IRIs, they SHOULD be validated with the ToASCII operation from [RFC3490], using the flags "UseSTD3ASCIIRules" and "AllowUnassigned" 🔥. An IRI with an invalid IDN can't be resolved successfully, so that's a real dealbreaker 😭. Validated IDN components SHOULD be character normalized with the Nameprep process [RFC3491], but for legibility, they SHOULD NOT be converted into ASCII Compatible Encoding (ACE) 💅.
Scheme-based normalization may also treat IDN components and their punycode conversions as equivalent 🧚. For example, "http://résumé.example.org" may be considered equivalent to "http://xn--rsum-bpad.example.org" ✨. Other scheme-specific normalizations are possible too, so keep your eyes open 👀.
5.3.4. Protocol-Based Normalization 🦋
Putting real effort into reducing false negatives is often cost-effective for web spiders 🕷️. Because of that, they go even harder with IRI comparison, using more aggressive techniques 🔥. For example, if they notice that an IRI like this:
http://example.com/dataredirects to an IRI that only differs by the trailing slash:
http://example.com/data/they will likely treat the two as equivalent from then on 💜. This kind of trick is only appropriate when equivalence is clearly shown by BOTH the result of accessing the resources AND the common conventions of the scheme's dereference algorithm 👑. In this case, that convention is HTTP origin servers using redirection to avoid problems with relative references 🌈. Not every redirect is a free pass, bestie 🙈.
6. Use of IRIs 🎀
6.1. Limitations on UCS Characters Allowed in IRIs 💖
This section talks about limits on which characters and character sequences can be used in IRIs, beyond what's in section 2.2 and section 4.1 🌸. These considerations matter when IRIs are created and when URIs get converted into IRIs 🧚.
a. The set of characters allowed in each IRI component is limited by how that component is defined 💅. For example, the definition of the scheme component doesn't allow characters beyond US-ASCII 🙈.
(Note: In accordance with URI practice, generic IRI software cannot and should not check for such limitations.)
b. The UCS has tons of characters that look almost exactly like each other 😭. Since transcription errors are a real risk, these also should be avoided 💀. That includes full-width versions of Latin characters, half-width Katakana for Japanese, and many others 🍓. It also covers lookalikes of "space", "delims", and "unwise" characters, which are excluded in [RFC3491] 🫶.
Extra info is available from [UNIXML] 📚. [UNIXML] is written for running text rather than identifiers, but it still covers lots of the character categories that aren't a good fit for IRIs 🌈.
6.2. Software Interfaces and Protocols 💜
Okay so an IRI is defined as a sequence of characters, but software interfaces for URIs usually work on sequences of octets or other code units 🦋. That means software interfaces and protocols MUST define which character encoding is used 👑.
When intermediate software sits between IRI-capable components and URI-only components, it MUST map the IRIs per section 3.1 as they get transferred from the IRI-capable side to the URI-only side 🔥. That mapping SHOULD be applied as late as possible 💅. It SHOULD NOT be applied between components that are known to handle IRIs already 🧚.
6.3. Format of URIs and IRIs in Documents and Protocols 📄
Document formats that carry URIs might need an upgrade so they can carry IRIs too 🌸. If the document as a whole has a native character encoding, IRIs MUST also be encoded in that same encoding, and a parser or interpreter has to convert them accordingly 🎀. IRI characters that can't be written in the native encoding SHOULD be escaped using the document format's own escaping conventions, if those exist ✨. As an alternative, they MAY be percent-encoded according to section 3.1 💖. For example, in HTML or XML, numeric character references SHOULD be used 🌈. And if a document as a whole has a native character encoding that isn't UTF-8, then IRIs MUST NOT be placed into the document in UTF-8 🙅♀️.
Note: Some formats already accommodate IRIs, they just use different terminology 💅. HTML 4.0 [HTML4] treats the conversion from IRIs to URIs as error-avoiding behavior 🫶. XML 1.0 [XML1], XLink [XLink], XML Schema [XMLSchema], and specs built on them all allow IRIs 🔥. Also, it's expected that all relevant new W3C formats and protocols will be required to handle IRIs [CharMod] 👑.
6.4. Use of UTF-8 for Encoding Original Characters 🌈
This section gets into the details and examples for point c) in section 1.2 🧚. To use IRIs, the URI that corresponds to the IRI has to encode original characters into octets using UTF-8 💜. This can be specified for all URIs of a URI scheme, or it can apply to individual URIs for schemes that don't say how to encode original characters 🦋. It can cover the whole URI or just some part of it 💅. For background on encoding characters into URIs, check out section 2.5 of [RFC3986] 📚.
For new URI schemes, using UTF-8 is recommended in [RFC2718] 🌸. Schemes that already use UTF-8 include the URN syntax [RFC2141], IMAP URLs [RFC2192], and POP URLs [RFC2384] ✨. On the other hand, the HTTP URL scheme doesn't specify how to encode original characters, so only some HTTP URLs can have corresponding IRIs that are different from them 🙈.
For example, for a document with the URI "http://www.example.org/r%C3%A9sum%C3%A9.html", you can build a corresponding IRI (in XML notation, see section 1.4): "http://www.example.org/résumé.html". Here "é" stands for the e-acute character, and "%C3%A9" is the UTF-8 encoded and percent-encoded version of that character 🎀. But for a document with the URI "http://www.example.org/r%E9sum%E9.html", the percent-encoding octets can't be turned into actual characters in an IRI, because the percent-encoding isn't based on UTF-8 😭.
So for most URI schemes, there's no need to upgrade the scheme definition to work with IRIs 🫶. The main cases where upgrading makes sense are when a scheme definition, or a particular component of one, is strictly limited to US-ASCII characters with no way to include non-ASCII characters or octets through percent-encoding 💖. Upgrading also makes sense when a scheme currently uses highly scheme-specific provisions for encoding non-ASCII characters 🌈. The mailto: scheme [RFC2368] is an example of that 💌.
This specification doesn't upgrade any scheme specifications in any way, so that has to happen separately 🔥. Also, there's no such thing as an "IRI scheme" 🙅♀️. All IRIs use URI schemes, and all URI schemes can be used with IRIs, even if in some cases that only works by using URIs directly as IRIs, without any conversion 👑.
URI schemes can put restrictions on the syntax of scheme-specific URIs 💅. That means URIs that are fine under the generic URI syntax [RFC3986] might not be admissible because a scheme specification has narrower syntactic constraints 😭. URI scheme definitions can't broaden the syntactic restrictions of the generic URI syntax, because otherwise you could generate URIs that satisfy the scheme-specific constraints without satisfying the generic syntax constraints 🧚. However, the additional syntactic constraints a URI scheme specification imposes DO apply to IRIs 🔥. The corresponding URI produced by the mapping defined in section 3.1 MUST be a valid URI under the syntactic restrictions of the generic URI syntax, plus any narrower restrictions from the corresponding URI scheme specification 🎀.
The requirement to use UTF-8 applies to all parts of a URI, with a possible exception for the ireg-name part (see section 3.1) 🌸. But the ability of IRIs to directly represent a wide range of characters might only be used in some parts of an IRI (or IRI reference) 💖. The other parts of the IRI may only contain US-ASCII characters, or they may not be based on UTF-8 🙈. They could be based on another character encoding, or they could directly encode raw binary data (see also [RFC2397]) 🦋.
For example, it's possible to have a URI reference of "http://www.example.org/r%E9sum%E9.xml#r%C3%A9sum%C3%A9", where the document name is encoded in iso-8859-1 based on server settings, but the fragment identifier is encoded in UTF-8 according to [XPointer] 🌈. The IRI corresponding to that URI would be (in XML notation) "http://www.example.org/r%E9sum%E9.xml#résumé"; 💅.
Similar considerations apply to query parts 🧚. The IRI feature of including non-ASCII characters can only be used if the query part is encoded in UTF-8 ✨.
6.5. Relative IRI References 🎀
Okay bestie, good news: resolving relative IRI references against a base is lowkey straightforward ✨ You can apply the algorithms from [RFC3986] directly. The only twist is that you treat the extra characters IRIs allow in IRI references the same way unreserved characters get treated in URI references 💅
7. URI/IRI Processing Guidelines (Informative) 🧚
This section is informative, so it's giving "here are some tips" and not strict rules 🫶 It's all about supporting IRIs in the same software that already handles URIs: software interfaces that handle URIs, software where users type URIs in, software that creates or generates URIs, software that displays URIs, formats and protocols that carry URIs around, and software that interprets URIs. Lots of these may need modification before they work properly with IRIs 😭 Everything here also applies to URI references and IRI references, periodt 💖
7.1. URI/IRI Software Interfaces 🔌
Interfaces that handle URIs, like URI-handling APIs and protocols that transfer URIs, need interfaces and protocol elements designed to carry IRIs. Main character energy 👑
If the current handling in an API or protocol is based on US-ASCII, UTF-8 is recommended as the character encoding for IRIs. It's compatible with US-ASCII, it follows the recommendations of [RFC2277], and it makes converting to URIs easy 🍓 Either way, the API or protocol definition must clearly define the character encoding to be used.
Moving from URI-only to IRI-capable components requires no mapping, although the conversion described in section 3.2 above may be performed. It is preferable not to perform this inverse conversion when there is a chance it can't be done correctly 🙈
7.2. URI/IRI Entry ⌨️
Some components let users put URIs into the system by typing or dictation, for example. This software must be updated to allow for IRI entry 🌸
A person looking at a visual representation of an IRI (a sequence of glyphs, in some order, on some visual display) or listening to one will use an entry method for the characters in their language to input the IRI. Depending on the script and the input method, this may be a more or less complicated process 😅
The process of IRI entry must ensure, as much as possible, that the restrictions defined in section 2.2 are met. This may be done by choosing appropriate input methods or variants/settings thereof, by appropriately converting the characters being input, by eliminating characters that cannot be converted, and/or by issuing a warning or error message to the user.
As an example of variant settings, input method editors for East Asian Languages usually allow Latin letters and related characters to be input in full-width or half-width versions. For IRI input, the input method editor should be set so that it produces half-width Latin letters and punctuation and full-width Katakana 💖
An input field used primarily or solely for URIs/IRIs may let the user view an IRI as it is mapped to a URI. Places where IRI input is frequent may provide the option to view an IRI as mapped to a URI. This will help users when some of the software they use doesn't accept IRIs yet 🫶
An IRI input component that interfaces with components handling URIs, but not IRIs, must map the IRI to a URI before passing it to those components.
For the input of IRIs with right-to-left characters, please see section 4.3.
7.3. URI/IRI Transfer between Applications 📨
Lots of apps, especially mail user agents, try to detect URIs in plain text. They use heuristics based on URI syntax to do it. Then they let the user click on those URIs to get the resource in an appropriate (usually scheme-dependent) application 💌
These apps have to be upgraded to use IRI syntax as the base for their heuristics. In particular, a non-ASCII character should not be taken as the indication that an IRI has ended 🙅 They also have to make sure they correctly convert the detected IRI from the character encoding of the document or application where it appears to the character encoding used by the system-wide IRI invocation mechanism. If that mechanism only accepts URIs, they have to convert it to a URI instead (according to section 3.1).
The clipboard is another popular way to move URIs and IRIs between applications. On most platforms, it can store and transfer text in many languages and scripts. Used correctly, it transfers characters, not bytes, which will do the right thing with IRIs ✨
7.4. URI/IRI Generation 🏭
Systems that offer resources over the Internet, where those resources have logical names, sometimes generate URIs for the resources they offer automatically. For example, some HTTP servers can generate a directory listing for a file directory and then respond to the generated URIs with the files 📂
Lots of legacy character encodings are in use across various file systems. Many currently deployed systems don't transform the local character representation of the underlying system before generating URIs 😬
For maximum interoperability, systems that generate resource identifiers should make the appropriate transformations. For example, if a file system contains a file named résumé.html, a server should expose this as r%C3%A9sum%C3%A9.html in a URI. That allows use of résumé.html in an IRI, even if the file name is kept locally in a character encoding other than UTF-8 🌈
This recommendation particularly applies to HTTP servers. For FTP servers, similar considerations apply; see [RFC2640].
7.5. URI/IRI Selection 🎨
Sometimes resource owners and publishers control the IRIs used to identify their resources. This control is mostly exercised by controlling the resource names, such as file names, directly.
In these cases, it is recommended to avoid choosing IRIs that are easily confused. For example, in US-ASCII, the lower-case ell ("l") is easily confused with the digit one ("1"), and the upper-case oh ("O") is easily confused with the digit zero ("0"). Publishers should avoid confusing users with "br0ken" or "1ame" identifiers 🙃
Outside the US-ASCII repertoire, there are way more chances for confusion, and a complete set of guidelines is too long to include here. As long as names are limited to characters from a single script, native writers of a given script or language will know best when ambiguities can appear and how to avoid them. What may look ambiguous to a stranger may be totally obvious to the average native user 🦋 On the other hand, in some cases the UCS contains variants for compatibility reasons, for example for typographic purposes. These should be avoided wherever possible. Although there may be exceptions, newly created resource names should generally be in NFKC [UTR15] (which means that they are also in NFC).
As an example, the UCS contains the "fi" ligature at U+FB01 for compatibility reasons. Wherever possible, IRIs should use the two letters "f" and "i" rather than the "fi" ligature. An example where the latter may be used is in the query part of an IRI for an explicit search for a word written containing the "fi" ligature.
In certain cases, characters from different scripts might look the same. The best-known example is the similarity of the Latin "A", the Greek "Alpha", and the Cyrillic "A". To avoid such cases, only IRIs should be created where all the characters in a single component are used together in a given language. This usually means all of these characters will be from the same script, but some languages mix characters from different scripts (such as Japanese). This is similar to the heuristics used to tell letters and numbers apart in the examples above. Also, for Latin, Greek, and Cyrillic, using lowercase letters results in fewer ambiguities than using uppercase letters would 💜
7.6. Display of URIs/IRIs 🖥️
In situations where the rendering software isn't expected to display the non-ASCII parts of the IRI correctly using the available layout and font resources, those parts should be percent-encoded before being displayed.
For display of Bidi IRIs, please see section 4.1.
7.7. Interpretation of URIs and IRIs 🔍
Software that interprets IRIs as the names of local resources should accept IRIs in multiple forms and convert and match them with the appropriate local resource names.
First, multiple representations include both IRIs in the native character encoding of the protocol and their URI counterparts.
Second, it may include URIs constructed based on character encodings other than UTF-8. These URIs may be produced by user agents that don't conform to this specification and use legacy character encodings to convert non-ASCII characters to URIs. Whether this is necessary, and which character encodings to cover, depends on a number of factors, such as the legacy character encodings used locally and the distribution of various user agent versions. For example, software for Japanese may accept URIs in Shift_JIS and/or EUC-JP in addition to UTF-8 🎌
Third, it may include additional mappings to be more user-friendly and robust against transmission errors. These would be similar to how some servers currently treat URIs as case insensitive or do extra matching to account for spelling errors. For characters beyond the US-ASCII repertoire, this may, for example, include ignoring the accents on received IRIs or resource names. Please note that such mappings, including case mappings, are language dependent 🌍
It can be difficult to identify a resource unambiguously if too many mappings are taken into consideration. However, percent-encoded and not percent-encoded parts of IRIs can always be clearly distinguished. Also, the regularity of UTF-8 (see [Duerst97]) makes the potential for collisions lower than it may seem at first 🌸
7.8. Upgrading Strategy 🎀
okay so bestie, this part is all about the rollout order 🧠. Where this recommendation puts extra constraints on software that's already deployed everywhere, it's super important to roll out upgrades carefully and keep all the interdependencies in mind 💅.
If IRIs can't be interpreted correctly, they should not be created, generated, or transported 🙅♀️. So upgrading URI-interpreting software to accept IRIs is the top priority 👑.
Here's the twist 🦋: a single IRI only gets interpreted by one or a very few interpreters that are known in advance, even though it can be entered and transported super widely.
So IRIs benefit most from a broad upgrade of the software that lets you enter and transport them 🫶. Still, before any individual IRI gets published, the matching interpreting software should be upgraded too, so it covers the forms that the various versions of entry and transport software will send 🔥.
Upgrading generating software to make IRIs (instead of using a local character encoding) should only happen after the service has been upgraded to accept IRIs 😤. Same energy here: IRIs should only be generated when the service accepts them, and when the intervening infrastructure and protocol is known to transport them safely 🔒.
Software that converts URIs to IRIs for display should only get upgraded after upgraded entry software has been widely deployed to the people who'll see the result 👀.
Where there's a free choice of character encodings, using UTF-8 instead of another encoding can seriously cut the effort and dependencies for moving to IRIs 🌈. For example, when a new file-based web server is set up, using UTF-8 for file names makes the switch to IRIs way easier 🍓. And when a new web form uses UTF-8 as the character encoding of the form page, the returned query URIs will use UTF-8 too (unless the user, for whatever reason, changes the character encoding), so they'll be compatible with IRIs 🧚.
Taken together, these recommendations let you extend URIs to IRIs so you can handle characters other than US-ASCII while keeping interoperability problems to a minimum 💖. For considerations on upgrading URI scheme definitions, see section 6.4 ⭐.
8. Security Considerations 🚨
The security considerations discussed in [RFC3986] also apply to IRIs 🫢. On top of that, the following issues need particular care for IRIs:
Incorrect encoding or decoding can lead to security problems 💀. In particular, some UTF-8 decoders don't check against overlong byte sequences. As an example, a "/" is encoded with the byte 0x2F in both UTF-8 and US-ASCII, but some UTF-8 decoders also wrongly interpret the sequence 0xC0 0xAF as a "/". A sequence such as "%C0%AF.." may pass some security tests and then be interpreted as "/.." in a path if UTF-8 decoders are fault-tolerant, if conversion and checking aren't done in the right order, and/or if reserved characters and unreserved characters aren't clearly distinguished 😱.
There are various ways "spoofing" can happen with IRIs 🎭. "Spoofing" means that somebody may add a resource name that looks the same or similar to the user, but points to a different resource. The added resource may pretend to be the real one by looking very similar, but it may contain all kinds of changes that are hard to spot and can cause all kinds of problems. Most spoofing possibilities for IRIs are extensions of the ones for URIs 🙃.
Spoofing can happen for various reasons 🤔. First, a user's normalization expectations, or the actual normalization when entering an IRI or transcoding an IRI from a legacy character encoding, may not match the normalization used on the server side. Conceptually, this is no different from the problems surrounding case-insensitive web servers. For example, a popular web page with a mixed-case name ("http://big.example.com/PopularPage.html") might be "spoofed" by someone who is able to create "http://big.example.com/popularpage.html". However, the use of unnormalized character sequences, and of additional mappings for user convenience, may increase the chance of spoofing. Protocols and servers that allow the creation of resources with names that are not normalized are particularly vulnerable to these attacks. This is an inherent security problem of the relevant protocol, server, or resource and isn't specific to IRIs, but it's mentioned here for completeness 🧐.
Spoofing can happen in various IRI components, such as the domain name part or a path part 🌐. For considerations specific to the domain name part, see [RFC3491]. For the path part, administrators of sites that allow independent users to create resources in the same sub area may have to be careful to check for spoofing 👩💻.
Spoofing can happen because in the UCS many characters look very similar 👯. Details are discussed in Section 7.5. Again, this is very similar to spoofing possibilities on US-ASCII, e.g., using "br0ken" or "1ame" URIs 😂.
Spoofing can happen when URIs with percent-encodings based on various character encodings are accepted to deal with older user agents 🕰️. In some cases, particularly for Latin-based resource names, this is usually easy to detect, because UTF-8-encoded names, when interpreted and viewed as legacy character encodings, produce mostly garbage 🗑️.
When concurrently used character encodings have a similar structure but no characters have exactly the same encoding, detection is more difficult 😵💫.
Spoofing can happen with bidirectional IRIs if the restrictions in section 4.2 aren't followed. The same visual representation may be interpreted as different logical representations, and vice versa 🔄. It's also very important that a correct Unicode bidirectional implementation be used 🛠️.
9. Acknowledgements 🫶
We would like to thank Larry Masinter for his work as coauthor of many earlier versions of this document (draft-masinter-url-i18n-xx) 💜.
The discussion on the issue addressed here started a long time ago 🕰️. There was a thread in the HTML working group in August 1995 (under the topic of "Globalizing URIs") and in the www-international mailing list in July 1996 (under the topic of "Internationalization and URLs"), and there were ad-hoc meetings at the Unicode conferences in September 1995 and September 1997 🌸.
Many thanks go to Francois Yergeau, Matitiahu Allouche, Roy Fielding, Tim Berners-Lee, Mark Davis, M.T. Carrasco Benitez, James Clark, Tim Bray, Chris Wendt, Yaron Goland, Andrea Vine, Misha Wolf, Leslie Daigle, Ted Hardie, Bill Fenner, Margaret Wasserman, Russ Housley, Makoto MURATA, Steven Atkin, Ryan Stansifer, Tex Texin, Graham Klyne, Bjoern Hoehrmann, Chris Lilley, Ian Jacobs, Adam Costello, Dan Oscarson, Elliotte Rusty Harold, Mike J. Brown, Roy Badami, Jonathan Rosenne, Asmus Freytag, Simon Josefsson, Carlos Viegas Damasio, Chris Haynes, Walter Underwood, and many others for help with understanding the issues and possible solutions, and with getting the details right 👑✨.
This document is a product of the Internationalization Working Group (I18N WG) of the World Wide Web Consortium (W3C). Thanks to the members of the W3C I18N Working Group and Interest Group for their contributions and their work on [CharMod]. Thanks also go to the members of many other W3C Working Groups for adopting IRIs, and to the members of the Montreal IAB Workshop on Internationalization and Localization for their review 🌈.
10. References 📚
10.1. Normative References
[ASCII]American National Standards Institute, "Coded Character Set -- 7-bit American Standard Code for Information Interchange", ANSI X3.4, 1986.[ISO10646]International Organization for Standardization, "ISO/IEC 10646:2003: Information Technology - Universal Multiple-Octet Coded Character Set (UCS)", ISO Standard 10646, December 2003.[RFC2119]Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.[RFC2234]Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997.[RFC3490]Faltstrom, P., Hoffman, P., and A. Costello, "Internationalizing Domain Names in Applications (IDNA)", RFC 3490, March 2003.[RFC3491]Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)", RFC 3491, March 2003.[RFC3629]Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.[RFC3986]Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, January 2005.[UNI9]Davis, M., "The Bidirectional Algorithm", Unicode Standard Annex #9, March 2004,<http://www.unicode.org/reports/tr9/tr9-13.html>.[UNIV4]The Unicode Consortium, "The Unicode Standard, Version 4.0.1, defined by: The Unicode Standard, Version 4.0 (Reading, MA, Addison-Wesley, 2003. ISBN 0-321-18578-1), as amended by Unicode 4.0.1 (http://www.unicode.org/versions/Unicode4.0.1/)", March 2004.[UTR15]Davis, M. and M. Duerst, "Unicode Normalization Forms", Unicode Standard Annex #15, April 2003,<http://www.unicode.org/unicode/reports/tr15/tr15-23.html>.
10.2. Informative References ✨
okay bestie, these are the extra-credit reads 📚 Nothing in here is strictly required to implement the spec, but they're the receipts for where all this came from. Here they are, complete:
[BidiEx]"Examples of bidirectional IRIs",http://www.w3.org/International/iri-edit/BidiExamples.[CharMod]Duerst, M., Yergeau, F., Ishida, R., Wolf, M., and T. Texin, "Character Model for the World Wide Web: Resource Identifiers", World Wide Web Consortium Candidate Recommendation, November 2004,http://www.w3.org/TR/charmod-resid.[Duerst97]Duerst, M., "The Properties and Promises of UTF-8", Proc. 11th International Unicode Conference, San Jose, September 1997,http://www.ifi.unizh.ch/mml/mduerst/papers/PDF/IUC11-UTF-8.pdf.[Gettys]Gettys, J., "URI Model Consequences",http://www.w3.org/DesignIssues/ModelConsequences.[HTML4]Raggett, D., Le Hors, A., and I. Jacobs, "HTML 4.01 Specification", World Wide Web Consortium Recommendation, December 1999,http://www.w3.org/TR/html401/appendix/notes.html#h-B.2.[RFC2045]Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.[RFC2130]Weider, C., Preston, C., Simonsen, K., Alvestrand, H., Atkinson, R., Crispin, M., and P. Svanberg, "The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996", RFC 2130, April 1997.[RFC2141]Moats, R., "URN Syntax", RFC 2141, May 1997.[RFC2192]Newman, C., "IMAP URL Scheme", RFC 2192, September 1997.[RFC2277]Alvestrand, H., "IETF Policy on Character Sets and Languages", BCP 18, RFC 2277, January 1998.[RFC2368]Hoffman, P., Masinter, L., and J. Zawinski, "The mailto URL scheme", RFC 2368, July 1998.[RFC2384]Gellens, R., "POP URL Scheme", RFC 2384, August 1998.[RFC2396]Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifiers (URI): Generic Syntax", RFC 2396, August 1998.[RFC2397]Masinter, L., "The "data" URL scheme", RFC 2397, August 1998.[RFC2616]Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.[RFC2640]Curtin, B., "Internationalization of the File Transfer Protocol", RFC 2640, July 1999.[RFC2718]Masinter, L., Alvestrand, H., Zigmond, D., and R. Petke, "Guidelines for new URL Schemes", RFC 2718, November 1999.[UNIXML]Duerst, M. and A. Freytag, "Unicode in XML and other Markup Languages", Unicode Technical Report #20, World Wide Web Consortium Note, June 2003,http://www.w3.org/TR/unicode-xml/.[XLink]DeRose, S., Maler, E., and D. Orchard, "XML Linking Language (XLink) Version 1.0", World Wide Web Consortium Recommendation, June 2001,http://www.w3.org/TR/xlink/#link-locators.[XML1]Bray, T., Paoli, J., Sperberg-McQueen, C., Maler, E., and F. Yergeau, "Extensible Markup Language (XML) 1.0 (Third Edition)", World Wide Web Consortium Recommendation, February 2004,http://www.w3.org/TR/REC-xml#sec-external-ent.[XMLNamespace]Bray, T., Hollander, D., and A. Layman, "Namespaces in XML", World Wide Web Consortium Recommendation, January 1999,http://www.w3.org/TR/REC-xml-names.[XMLSchema]Biron, P. and A. Malhotra, "XML Schema Part 2: Datatypes", World Wide Web Consortium Recommendation, May 2001,http://www.w3.org/TR/xmlschema-2/#anyURI.[XPointer]Grosso, P., Maler, E., Marsh, J. and N. Walsh, "XPointer Framework", World Wide Web Consortium Recommendation, March 2003,http://www.w3.org/TR/xptr-framework/#escaping.
Appendix A. Design Alternatives 🎀
this part is the "what we almost did instead" tea ☕ it summarizes the big design options that got considered and why they got benched. spoiler: a lot of drama, very few regrets 💅
Appendix A.1. New Scheme(s)
so the idea was to introduce new schemes, like httpi: and ftpi:, or a whole new metascheme, i:, which would give you prefixes like i:http: and i:ftp:. the goal was to make IRI-to-URI conversion depend on the scheme, or to tell apart percent-encodings that came from IRI-to-URI conversion and ones that came from old legacy character encodings 🤔
but new schemes aren't needed to tell URIs apart from true IRIs (aka IRIs with non-ASCII characters) 🌸 the payoff of spotting where a percent-encoding came from is pretty small, since UTF-8 can be detected with very high reliability. and deploying new schemes is extremely hard, so skipping them makes IRIs way easier to actually roll out 🙌 making conversion depend on the scheme is highly inadvisable, and separate schemes for IRIs would basically encourage that. using one uniform convention for IRI-to-URI conversion keeps IRI implementation separate from whether any new schemes ever show up ✨
Appendix A.2. Character Encodings Other Than UTF-8
UTF-7 was in the running early on as an alternative to UTF-8 for converting IRIs to URIs 👀 it wouldn't have needed percent-encoding, and in most cases it would've been shorter than percent-encoded UTF-8.
but using UTF-8 avoids a double layering and overloading of the "+" character, which is a whole mess we don't need 🫶 UTF-8 is fully compatible with US-ASCII, so the IETF has recommended it, and it's widely used everywhere now. UTF-7 never really caught on and is now clearly being discouraged 💀 plus, making implementations convert from UTF-8 to UTF-7 and back would've been an extra burden. no thanks 🙅♀️
Appendix A.3. New Encoding Convention
instead of using the existing percent-encoding convention for URIs, which works on octets, someone floated a brand new encoding convention. for example, using %u to introduce UCS code points 🧚
but the existing octet-based percent-encoding mechanism doesn't need a URI syntax upgrade and doesn't need corresponding server upgrades either. so why reinvent the wheel? 🛞✨
Appendix A.4. Indicating Character Encodings in the URI/IRI
some proposals wanted to indicate the character encodings used in a URI or IRI with a new syntactic convention inside the URI itself, kinda like the "charset" parameter for emails and web pages 📬 as an example, the label in square brackets in "http://www.example.org/ros[iso-8859-1]é" said that the "é" after it had to be interpreted as iso-8859-1.
if UTF-8 is used exclusively, you don't need an upgrade to the URI syntax at all 🎉 it also avoids having potentially multiple labels that have to be copied correctly in every case, even on the side of a bus or on a napkin, which would cause usability problems (and be prohibitively annoying 😭). using only UTF-8 also cuts down on transcoding errors and confusion. iconic behavior honestly 👑
Authors' Addresses 🌍
these are the two lovely humans behind the doc 🫶 note from the authors: please write "Duerst" with a u-umlaut wherever possible, for example as "Dürst" in XML and HTML.
Martin Duerst (Note: Please write "Duerst" with u-umlaut wherever
possible, for example as "Dürst" in XML and
HTML.)
World Wide Web Consortium
5322 Endo
Fujisawa, Kanagawa 252-8520
Japan
Phone: +81 466 49 1170
Fax: +81 466 49 1171
EMail: duerst@w3.org
URI: http://www.w3.org/People/D%C3%BCrst/
(Note: This is the percent-encoded form of an IRI.)
Michel Suignard
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
U.S.A.
Phone: +1 425 882-8080
EMail: michelsu@microsoft.com
URI: http://www.suignard.comFull Copyright Statement ⚖️
Copyright (C) The Internet Society (2005).
this document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
this document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🔐
the IETF doesn't take a position on whether any Intellectual Property Rights or other rights that someone might claim over the technology in this document are valid or in scope, or on whether any license under those rights might be available. it also doesn't claim to have made any independent effort to find any such rights 🕵️♀️ the IETF's procedures for rights in IETF Documents are in BCP 78 and BCP 79.
copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses that will be made available, and the results of any attempt to get a general license or permission for implementers or users of this specification can be found in the IETF's online IPR repository at http://www.ietf.org/ipr.
the IETF invites anyone interested to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology needed to implement this standard. please send that info to the IETF at ietf-ipr@ietf.org 💌
Acknowledgement 💖
funding for the RFC Editor function is currently provided by the Internet Society. thank you to them, and to everyone who made this doc possible 🌈✨
📦 where this page comes from
- ietf: rfc3987 2005-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms