RFC 5137

ASCII Escaping of Unicode Characters

📅 published February 2008, from ietf

RFC 5137: ASCII Escaping of Unicode Characters ✨

Okay bestie, this one is a BCP 💅 (Best Current Practice, BCP 137, to be exact). It came out in February 2008 and was written by J. Klensin for the Network Working Group. It's in the Best Current Practice category, and the header doesn't list any RFCs it obsoletes or updates. So it's a standalone, no-drama doc 🧚

Status of This Memo 📋

This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Distribution of this memo is unlimited.

Abstract 💌

Escapes are needed a lot 🙈 Whenever a protocol has to encode characters that can't be represented or sent directly, it needs some kind of escape mechanism. With ASCII, the old-school escape was the decimal or hexadecimal numeric value of the character, and people wrote it in a bunch of different ways. Then Unicode came along, where characters take up two or more octets and can be coded in several different forms, and suddenly escapes got way more complicated 😭

This document talks through some of the options that are already in use. It also covers what to think about when picking one for new IETF protocols and for protocols that are being internationalized now 🦋

Table of Contents 🗂️

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.1.  Context and Background . . . . . . . . . . . . . . . . . .  3
     1.2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  4
     1.3.  Discussion List  . . . . . . . . . . . . . . . . . . . . .  4
   2.  Encodings that Represent Unicode Code Points: Code
       Position versus UTF-8 or UTF-16 Octets . . . . . . . . . . . .  4
   3.  Referring to Unicode Characters  . . . . . . . . . . . . . . .  5
   4.  Syntax for Code Point Escapes  . . . . . . . . . . . . . . . .  6
   5.  Recommended Presentation Variants for Unicode Code Point
       Escapes  . . . . . . . . . . . . . . . . . . . . . . . . . . .  7
     5.1.  Backslash-U with Delimiters  . . . . . . . . . . . . . . .  7
     5.2.  XML and HTML . . . . . . . . . . . . . . . . . . . . . . .  7
   6.  Forms that Are Normally Not Recommended  . . . . . . . . . . .  8
     6.1.  The C Programming Language: Backslash-U  . . . . . . . . .  8
     6.2.  Perl: A Hexadecimal String . . . . . . . . . . . . . . . .  8
     6.3.  Java: Escaped UTF-16 . . . . . . . . . . . . . . . . . . .  9
   7.  Security Considerations  . . . . . . . . . . . . . . . . . . .  9
   8.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . .  9
   9.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 10
     9.1.  Normative References . . . . . . . . . . . . . . . . . . . 10
     9.2.  Informative References . . . . . . . . . . . . . . . . . . 10
   Appendix A.  Formal Syntax for Forms Not Recommended . . . . . . . 12
     A.1.  The C Programming Language Form  . . . . . . . . . . . . . 12
     A.2.  Perl Form  . . . . . . . . . . . . . . . . . . . . . . . . 12
     A.3.  Java Form  . . . . . . . . . . . . . . . . . . . . . . . . 12

1. Introduction 🎀

1.1. Context and Background 🌸

Okay so the escape problem is old 💀 With ASCII [ASCII] coding, the traditional escape was either the decimal or hexadecimal numeric value of the character, written in lots of different ways. In different contexts you'll see %dNN or %NN for decimal, and %NN, %xNN, X'nn', and %X'NN' for hexadecimal. "%NN" got really popular in recent years to mean a hexadecimal value without any extra labels, probably because URLs are everywhere 🌍 Some applications even use octal forms, and while those don't generalize well, the MIME Quoted-Printable and Encoded-word forms count as yet another set of escapes. So even for the basic cases of ASCII and the standards built on top of it, like the ISO 8859 family, we've been juggling several escaping forms, each one the result of some history 🙃

When you move to Unicode [Unicode] [ISO10646], it gets even more complicated, because characters take up two or more octets and can be coded in several different forms. Unicode represents characters as code points, which are numeric values from 0 to hex 10FFFF. When code points show up in flowing text, they're written in the so-called "U+" notation, as values from U+0000 to U+10FFFF. Once they're serialized into octets, the same code point can be represented in different forms:

  • in UTF-8 with one to four octets [RFC3629]
  • in UTF-16 with two or four octets (or one or two seizets -- 16-bit units)
  • in UTF-32 with exactly four octets (or one 32-bit unit)

When people escape characters, they've used a lot of hexadecimal representations of the serialized forms, plus variations on the U+ notation, which are known as code point escapes ✨

Per the existing best-practices recommendations [RFC2277], new protocols that have to carry textual content for humans SHOULD be designed so that the full repertoire of Unicode characters can be represented in that text 🌈

This document proposes that existing protocols being internationalized, and those that need an escape mechanism, SHOULD use some contextually appropriate variation on references to code points as described in Section 2, unless other considerations outweigh the ones described here 🫶

This recommendation doesn't apply to protocols that already accept native UTF-8 or some other encoding of Unicode. In general, when protocols get internationalized, it's better to accept those forms than to use escapes. This recommendation covers cases, including transition arrangements, where that isn't practical 🍓

Escapes for Unicode characters also show up in presentations to users, meaning user interfaces (UI). The formats specified here, and the reasoning behind them, may apply in UI contexts too. But this is not a proposal to standardize UI or presentation forms, so don't read it that way 🙅‍♀️

This document doesn't make general recommendations for processing Unicode strings or for their contents. It assumes that the strings you might want to escape are valid and reasonable, and it leaves the definition of "valid and reasonable" to other documents. You can find recommendations about general treatment of Unicode strings in lots of places, including the Unicode Standard itself and the W3C Character Model [W3C-CharMod], as well as specific rules in individual protocols 📚

1.2. Terminology 💅

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

Additional Unicode-specific terminology shows up in [UnicodeGlossary], but you don't need it to understand this spec 🧠

1.3. Discussion List 📬

Discussion of this document should be addressed to the discuss@apps.ietf.org mailing list 💜

2. Encodings that Represent Unicode Code Points: Code Position versus UTF-8 or UTF-16 Octets 🧚

Here's the big choice 👑 There are two major families of ways to escape Unicode characters. One family uses the code point in some representation (more on that in the next section). The other family encodes the octets of the UTF-8 encoding, or some other encoding, in some representation. A few other options are possible, but they've been rare in practice. This specification recommends that, unless there are compelling reasons to do otherwise, Unicode code points SHOULD be used rather than a representation of UTF-8 (or UTF-16) octets. There are several reasons for this, including:

  • One reason so many IETF protocols have worked well is that they use human-readable text forms to communicate, instead of encodings that usually need a computer program (or hand simulation of an algorithm) to decode. That suggests the presentation form should reference the Unicode tables for characters, and do so as simply as possible.
  • Because of how UTF-8 works, a human looking at a decimal or hexadecimal numeral that represents UTF-8 octets needs one or more decoding steps to figure out the Unicode code point that can be used to look up the character in a table. That might be fine when the goal really is to represent the UTF-8 form. But in general it just hides the info you want, makes errors more likely, and makes debugging harder 😭
  • Except for characters in the ASCII subset of Unicode (U+0000 through U+007F), the code point form is generally more compact than forms based on coding UTF-8 octets, and sometimes it's way more compact 🦋

The same considerations that apply to representing the octets of the UTF-8 encoding also apply to more compact ACE encodings, such as the "bootstring" encoding [RFC3492], with or without its "Punycode" profile.

Similar considerations apply to UTF-16 encoding, such as the \uNNNN form used in Java (see Section 6.3). Those forms are equivalent to code point references for the Basic Multilingual Plane (BMP, Plane 0). But to reach higher planes, you need a two-stage decoding process to handle surrogates 🌸

Quick heads-up: the BMP-only equivalence is the whole reason the surrogate thing matters. Higher planes aren't a simple one-to-one match with the \uNNNN form.

3. Referring to Unicode Characters 🦋

Whatever gets decided about escapes for Unicode characters in protocol or similar contexts, text that refers to a Unicode code point SHOULD use the U+NNNN[N[N]] syntax, as specified in the Unicode Standard, where the NNNN... string consists of hexadecimal numbers. Text that actually contains a Unicode character SHOULD use a syntax that's more suitable for automated processing 💖

4. Syntax for Code Point Escapes ✨

Okay so there are a bunch of ways to write code point escapes, and they're all the same thing in content and meaning 🧚 The only real difference is how they look. The best one depends on the app you're building, because one form might just "fit" better in your context than the others 💅 That said, hexadecimal is clearly the way to go. Decimal or octal offsets SHOULD NOT be used 🙈

Since this spec doesn't pick one syntax for everyone, any protocol that uses escapes MUST define the syntax it's using, including any escapes needed so the escape sequence can be used literally 🎀

If you're the one choosing a format, think about at least these things:

  • If similar or related protocols already use one form, it's probably best to pick that same form. Consistency and predictability are everything 💖
  • A Unicode code point can land anywhere from U+0000 to U+10FFFF. Different escape systems use four, five, six, or eight hex digits. To avoid clever syntax tricks and the confusion and errors that come with them, forms with explicit string delimiters are generally preferred. Symmetric paired delimiters are way easier to read than delimiters that look totally unrelated to each other 🫶
  • Syntax forms that start with "\u" and have no explicit delimiters have shown up in a bunch of different escape systems. That includes the four or eight digit syntax of C [ISO-C] (see Section 6.1), the UTF-16 encoding of Java [Java] (see Section 6.3), and some setups that follow the "\u" with four, five, or six digits. Because it's not clear which option is actually being used, that confusion alone might be a reason to avoid all of these forms 😭
  • Forms that need you to decode surrogate pairs have a lot of the same problems as encoding UTF-8 octets. Internet protocols SHOULD NOT use surrogate pairs 💀

There are lots of ways to represent a Unicode code point position, and none of them is "best" for every context 🦋 Also, if the escape mechanism itself needs an escape, the best option for that might be different depending on where you are.

Some popular forms that you might reasonably consider for a given protocol are described below, and each one is tagged with a current-use context when that's possible. The two in this section are recommended for use in Internet Protocols. The other popular ones show up in Section 6, along with a look at their downsides ✨

5.1. Backslash-U with Delimiters

One of the recommended forms is a variation on all those forms that start with "\u" (see, e.g., Section 6.1, below>), but it adds explicit delimiters for the reasons we already covered 💜

Here's how it looks in ABNF [RFC5234]:

EmbeddedUnicodeChar =  %x5C.75.27 4*6HEXDIG %x27
   ; starting with lowercase "\u" and "'" and ending with "'".
   ; Note that the encodings are considered to be abstractions
   ; for the relevant characters, not designations of specific
   ; octets.

HEXDIG =  "0" / "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9" /
   "A" / "B" / "C" / "D" / "E" / "F"
   ; effectively identical with definition in RFC 5234.

If you're building an app with this form, you should spell out how to escape the introducing backslash ("") when you need to. "\" is one obvious option, but it's not the only one 🎀

5.2. XML and HTML

The other recommended form is the one used in XML, and it looks like "&#xNNNN;". Like the Perl form (Section 6.2), this one has a clear ending delimiter, which cuts down on ambiguity 👑 HTML uses something similar, but in some cases the semicolon can be left off. When that happens, the delimiter's benefit disappears, so the HTML form without the semicolon SHOULD NOT be used. Honestly, this format does get called ugly and awkward outside of HTML, XML, and similar contexts 😂

In ABNF:

EmbeddedUnicodeChar =   %x26.23.78 2*6HEXDIG %x3B
   ; starts with "&#x" and ends with ";"

Also, a literal "&" can be written as & when you're using this style 🍓

6.1. The C Programming Language: Backslash-U

These forms are used in the C Programming Language [ISO-C] when you need an ASCII escape for embedded Unicode characters:

\UNNNNNNNN (for any Unicode character) and

\uNNNN (for Unicode characters in plane 0)

This form has some downsides that could matter a lot. First, using a case difference between "u" (for the four-digit form) and "U" (for the eight-digit form) might not feel natural in places where uppercase and lowercase are usually treated as the same, and it could be confusing to people who aren't very familiar with Latin-based alphabets. (Though those people might have even more trouble reading the relevant English text and explanations, lol.) Second, as discussed in Section 4, having a bunch of different conventions that start with \u or \U can cause confusion, because people end up making wrong assumptions about what they're looking at 😵‍💫

6.2. Perl: A Hexadecimal String

Perl uses the form \x{NNNN...}. The nice thing about this one is the explicit delimiters, which solve the problem of variable-length strings and mean you don't have to rely on the case-change trick from the form above to tell Plane 0 apart from the more general forms ✨ Other programming languages would probably go for X'NNNN...' forms for hexadecimal strings and maybe U'NNNN...' for Unicode-specific strings, but those don't seem to be in use around the IETF.

There's a possible ambiguity with two-character or low-numbered sequences in this notation. Octets in the range \x(00) through \x(FF) could be read as being in the local character set instead of as Unicode code points. Because of that ambiguity, and because IETF documents don't have a way to include pragmas (see [PERLUniIntro] for more on the "encoding" pragma in Perl and other details), the Perl forms should be used with extreme caution, if at all 🫣

6.3. Java: Escaped UTF-16

Java [Java] uses the form \uNNNN, but it refers to UTF-16 values, not Unicode code points 🧃 Even though the syntax looks like the one in Section 6.1, the tie to UTF-16 makes it in many ways more similar to the UTF-8 encodings we talked about above than to an escape that designates Unicode code points. The UTF-16 form, and so the Java escape notation, can only represent characters outside Plane 0 (that is, above U+FFFF) by using surrogate pairs, which brings up some of the same issues as using UTF-8 octets. For characters in Plane 0, the Java form is indistinguishable from the Plane 0-only form described in Section 6.1. Just for that reason alone, it SHOULD NOT be used as an escape except in those Java contexts where it's natural 💅

7. Security Considerations 🔐

This document proposes a set of rules for encoding Unicode characters when other considerations don't apply. Since all of the recommended encodings are unambiguous and normalization issues aren't involved, it shouldn't introduce any security issues that aren't already there from simply using non-ASCII characters, no matter how they're encoded. The mechanisms suggested should slightly lower the risk of confusing users with encoded characters, because they make the identity of the characters being used a little more obvious than some of the alternatives 👀

An escape mechanism like the one specified in this document can let characters be represented in more than one way. If software interprets the escaped form, there's a real risk that security checks, along with any necessary checks for things like minimal or normalized forms, happen at the wrong point. This is serious, bestie, so please don't skip it 🚨

8. Acknowledgments 🫶

This document came out of a bunch of discussions within the IETF Applications Area, and as part of work on email internationalization and internationalized domain name updates. It's a synthesis of a huge number of discussions, and the participants' comments are gratefully acknowledged. The help of Mark Davis in building a list of alternative presentations and picking among them was especially important 🌸

Tim Bray, Peter Constable, Stephane Bortzmeyer, Chris Newman, Frank Ellermann, Clive D.W. Feather, Philip Guenther, Bjoern Hoehrmann, Simon Josefsson, Bill McQuillan, der Mouse, Phil Pennock, and Julian Reschke gave careful reading and some corrections and suggestions on the various working drafts that came before this document. Taken together, their suggestions drove the big revision of this document and its recommendations between version -00 and version -01, plus more improvements in the versions after that 🎉

9. References 📚

9.1. Normative References

  • [ISO10646] International Organization for Standardization, "Information Technology -- Universal Multiple-Octet Coded Character Set (UCS)", ISO/IEC 10646:2003, December 2003.
  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.
  • [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, January 2008.
  • [Unicode] The Unicode Consortium, "The Unicode Standard, Version 5.0", 2006. (Addison-Wesley, 2006. ISBN 0-321-48091-0).

9.2. Informative References ✨

okay so the informative refs are the bonus reading list 📚 nothing here is required, but they're the receipts for everything we talked about

[ASCII] American National Standards Institute (formerly United States of America Standards Institute), "USA Code for Information Interchange", ANSI X3.4-1968, 1968.

ANSI X3.4-1968 has been replaced by newer versions with slight modifications, but the 1968 version remains definitive for the Internet. 💅

[ISO-C] International Organization for Standardization, "Information technology -- Programming languages -- C", ISO/IEC 9899:1999, 1999. 🌸

[Java] Sun Microsystems, Inc., "Java Language Specification, Third Edition", 2005, <http://java.sun.com/docs/books/jls/third_edition/html/lexical.html#95413p>. 🎀

[PERLUniIntro] Hietaniemi, J., "perluniintro", Perl documentation 5.8.8, 2002, <http://perldoc.perl.org/perluniintro.html>. 🦋

[RFC2277] Alvestrand, H., "IETF Policy on Character Sets and Languages", BCP 18, RFC 2277, January 1998. 💜

[RFC3492] Costello, A., "Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)", RFC 3492, March 2003. 🧚

[UnicodeGlossary] The Unicode Consortium, "Glossary of Unicode Terms", June 2007, <http://www.unicode.org/glossary>. 🌈

[W3C-CharMod] Duerst, M., "Character Model for the World Wide Web 1.0", W3C Recommendation, February 2005, <http://www.w3.org/TR/charmod/>. ⭐

bestie, this is the appendix for the forms we said we'd rather you didn't use 🙈 the syntax isn't written out inline on purpose, to discourage it, but if people are gonna use them anyway, they should at least do it consistently. ngl, some of these forms were never pinned down precisely in their original specs, and others drifted over time in ways that don't quite match up. so these definitions are not normative and might not even line up with every reasonable reading of their sources 😭 the "HEXDIG" used below is defined back in Section 5.1.

A.1. The C Programming Language Form 💖

Here's the ABNF [RFC5234] for the C form:

EmbeddedUnicodeChar =  BMP-form / Full-form

BMP-form =  %x5C.75 4HEXDIG ; starting with lowercase "\u"
      ; The encodings are considered to be abstractions for the
      ; relevant characters, not designations of specific octets.

Full-form =  %x5C.55 8HEXDIG ; starting with uppercase "\U"

A.2. Perl Form 🌙

The Perl version, which starts with "\x":

EmbeddedUnicodeChar =   %x5C.78 "{" 2*6HEXDIG "}" ; starts with "\x"

A.3. Java Form ☕

And the Java one, also starting with "\u":

EmbeddedUnicodeChar =   %x5C.7A 4HEXDIG ; starts with "\u"

Author's Address 🫶

John C Klensin 1770 Massachusetts Ave, #322 Cambridge, MA 02140 USA

Phone: +1 617 245 1457 EMail: john-ietf@jck.com

Copyright (C) The IETF Trust (2008).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🦋

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. 💜

📦 where this page comes from