RFC 2279

UTF-8, a transformation format of ISO 10646

📅 published January 1998, from ietf

UTF-8, a transformation format of ISO 10646 🎀

okay bestie, so this is RFC 2279, a standards track doc from the Network Working Group 💅 It was written by F. Yergeau at Alis Technologies and dated January 1998. It obsoletes RFC 2044, which means this one officially takes over from it. ✨

Status of This Memo 📜

This document specifies an Internet standards track protocol for the Internet community, and it asks for discussion and suggestions for improvements 🫶 If you want the current standardization state and status of this protocol, check the "Internet Official Protocol Standards" (STD 1) doc. Distribution of this memo is unlimited 🌈

Copyright (C) The Internet Society (1998). All Rights Reserved. 💜

Abstract 🦋

ISO/IEC 10646-1 defines a multi-octet character set called the Universal Character Set (UCS), and it covers most of the world's writing systems 🌸 The thing is, multi-octet characters aren't compatible with lots of current apps and protocols 😭 so people came up with a few UCS transformation formats (UTF), each with its own vibe and characteristics.

UTF-8 is the main character of this memo 👑 It preserves the full US-ASCII range, which keeps it compatible with file systems, parsers and other software that rely on US-ASCII values but stay transparent to every other value. This memo updates and replaces RFC 2044, and it especially sorts out the question of versions of the relevant standards 🔥

1. Introduction ✨

ISO/IEC 10646-1 [ISO-10646] defines a multi-octet character set called the Universal Character Set (UCS), which covers most of the world's writing systems 🌍 Two multi-octet encodings are defined. One is UCS-4, a four-octet-per-character encoding. The other is UCS-2, a two-octet-per-character encoding that can only address the first 64K characters of the UCS (the Basic Multilingual Plane, BMP). Outside of that there are currently no assignments 🍓

The Unicode standard [UNICODE] defines the same set of characters, and it's honestly iconic because it adds extra character properties and application details that implementors would love 💖 It just doesn't have the UCS-4 encoding. Up until now, changes in Unicode and amendments to ISO/IEC 10646 have tracked each other, so the character repertoires and code point assignments have stayed in sync 🫶 The standardization committees have committed to keeping that very useful synchronism going.

UCS-2 and UCS-4 are kind of hard to use with lots of current apps and protocols that assume 8 or even 7 bit characters 🙈 Even newer systems that can handle 16 bit characters can't process UCS-4 data. So that's why so-called UCS transformation formats (UTF) got developed, each with different characteristics 🧚

UTF-1 is only of historical interest now, since it was removed from ISO/IEC 10646 💀 UTF-7 can encode the full BMP repertoire using only octets with the high-order bit clear (7 bit US-ASCII values, [US-ASCII]), so it's deemed a mail-safe encoding ([RFC2152]) 📬 UTF-8, the object of this memo, uses all bits of an octet, but it preserves the full US-ASCII range. US-ASCII characters are encoded in one octet with the normal US-ASCII value, and any octet with such a value can only stand for a US-ASCII character, and nothing else. Iconic consistency 💅

UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire into pairs of UCS-2 values from a reserved range. UTF-16 affects UTF-8 in that UCS-2 values from the reserved range must be treated specially in the UTF-8 transformation 👀

UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of octets. The number of octets, and the value of each one, depend on the integer value assigned to the character in ISO/IEC 10646 🎀 This transformation format has the following characteristics (all values are in hexadecimal):

  • Character values from 0000 0000 to 0000 007F (US-ASCII repertoire) correspond to octets 00 to 7F (7 bit US-ASCII values). The direct result is that a plain ASCII string is also a valid UTF-8 string 🌈
  • US-ASCII values don't show up otherwise in a UTF-8 encoded character stream. This keeps it compatible with file systems or other software (e.g. the printf() function in C libraries) that parse based on US-ASCII values but stay transparent to other values 🧚
  • Round-trip conversion is easy between UTF-8 and either of UCS-4 or UCS-2 🔁
  • The first octet of a multi-octet sequence tells you how many octets are in the sequence 🔢
  • The octet values FE and FF never appear 🙅‍♀️
  • Character boundaries are easy to find from anywhere in an octet stream 🗺️
  • The lexicographic sorting order of UCS-4 strings is preserved. Of course, this is of limited interest, since the sort order isn't culturally valid in either case 🤷‍♀️
  • The Boyer-Moore fast search algorithm can be used with UTF-8 data ⚡
  • UTF-8 strings can be fairly reliably recognized as such by a simple algorithm. The probability that a string of characters in any other encoding appears as valid UTF-8 is low, and it keeps dropping as the string gets longer 🔍

UTF-8 was originally a project of the X/Open Joint Internationalization Group (XOJIG), with the goal of specifying a File System Safe UCS Transformation Format [FSS-UTF] that's compatible with UNIX systems and supports multilingual text in a single encoding 🌸 The original authors were Gary Miller, Greger Leijonhufvud and John Entenmann. Later, Ken Thompson and Rob Pike did significant work on the formal UTF-8 👑

You can also find a description in Unicode Technical Report #4 and in the Unicode Standard, version 2.0 [UNICODE]. The definitive reference, including provisions for UTF-16 data within UTF-8, is Annex R of ISO/IEC 10646-1 [ISO-10646] 📖

2. UTF-8 definition 💖

In UTF-8, characters are encoded using sequences of 1 to 6 octets. The only octet of a "sequence" of one has the higher-order bit set to 0, and the remaining 7 bits are used to encode the character value. In a sequence of n octets, n>1, the initial octet has the n higher-order bits set to 1, followed by a bit set to 0. The remaining bit(s) of that octet contain bits from the value of the character being encoded. The following octet(s) all have the higher-order bit set to 1 and the following bit set to 0, which leaves 6 bits in each to hold bits from the character being encoded 🫧

The table below sums up the format of these different octet types. The letter x marks the bits available for encoding bits of the UCS-4 character value 📊

UCS-4 range (hex.)           UTF-8 octet sequence (binary)
0000 0000-0000 007F   0xxxxxxx
0000 0080-0000 07FF   110xxxxx 10xxxxxx
0000 0800-0000 FFFF   1110xxxx 10xxxxxx 10xxxxxx

0001 0000-001F FFFF   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03FF FFFF   111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7FFF FFFF   1111110x 10xxxxxx ... 10xxxxxx

Here's how encoding from UCS-4 to UTF-8 goes 🪄

  1. Figure out how many octets you need from the character value and the first column of the table above. It's important to note that the rows of the table are mutually exclusive, which means there's only one valid way to encode a given UCS-4 character 💅
  2. Prepare the high-order bits of the octets as per the second column of the table 🎀
  3. Fill in the bits marked x from the bits of the character value. Start from the lower-order bits of the character value and put them first in the last octet of the sequence, then the next-to-last, and so on until all the x bits are filled in 🌈

The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can, in principle, be worked out from the above by simply extending each UCS-2 character with two zero-valued octets 🧩 However, pairs of UCS-2 values between D800 and DFFF (surrogate pairs in Unicode parlance) are actually UCS-4 characters transformed through UTF-16, so they need special treatment. The UTF-16 transformation has to be undone first, which yields a UCS-4 character that then gets transformed as above 🔄

Here's how decoding from UTF-8 to UCS-4 goes 🔍

  1. Initialize the 4 octets of the UCS-4 character with all bits set to 0 🫧
  2. Determine which bits encode the character value from the number of octets in the sequence and the second column of the table above (the bits marked x) 🗝️
  3. Distribute the bits from the sequence to the UCS-4 character. Start with the lower-order bits from the last octet of the sequence and keep going to the left until no x bits are left 🧁

If the UTF-8 sequence is no more than three octets long, decoding can go directly to UCS-2 ✨

NOTE -- actual implementations of the decoding algorithm above should protect against decoding invalid sequences. For instance, a naive implementation may (wrongly) decode the invalid UTF-8 sequence C0 80 into the character U+0000, which may have security consequences and/or cause other problems. See the Security Considerations section below. 🚨

A more detailed algorithm and formulae can be found in [FSS_UTF], [UNICODE] or Annex R to [ISO-10646] 📚

5. MIME registration 💖

Okay so this section is the registration bit, where the doc officially asks for UTF-8 to get a MIME charset label ✨ The label is [CHARSET-REG] approved, and the proposed charset parameter value is "UTF-8" 🎀 It's for labelling media types with text made from characters in the ISO/IEC 10646 repertoire, including all amendments up to amendment 5 (the Korean block), encoded into octets using the scheme we just walked through 🦋 UTF-8 fits MIME content types under the "text" top-level type, so it's giving main character energy 💅

Now, the label "UTF-8" doesn't include a version number. It just refers to ISO/IEC 10646 generically 🧚 This is on purpose, and the reasoning is so good, I'm obsessed 😭

A MIME charset label has one job: give just enough info to turn the bytes on the wire into characters, and that's it (see RFC 2045, section 2.2, in [MIME]) 🍓 As long as a character set standard doesn't change in a breaking way, version numbers don't help anyone. Learning from the tag that new characters might show up doesn't teach you anything about them, and they're arriving anyway 🙈

So as long as the standards keep evolving compatibly, labels with version numbers only look useful. The apparent advantage is only apparent, bestie 💀 Version-specific labels also have a real downside. When an older app gets data with a newer label it doesn't recognize, it might not recognize the label at all and be totally unable to deal with the data. A generic, known label would have gotten mostly correct processing, and the data probably doesn't even contain any new characters 🌸

Now, the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible change 😬 That kind of goes against the whole idea of a version-independent MIME charset label I just described. BUT the compatibility problem only shows up with data containing Korean Hangul characters encoded per Unicode 1.1 (or equivalently ISO/IEC 10646 before amendment 5), and arguably there's no such data to worry about. That's literally why the incompatible change was deemed okay 👑

So in practice, a version-independent label is warranted, as long as it's understood to cover all versions after Amendment 5, and as long as no incompatible change actually happens. If a later version of ISO/IEC 10646 does make incompatible changes, the MIME charset label defined here stays aligned with the previous version, unless and until the IETF specifically decides otherwise 🫶

The doc also proposes registering a second charset parameter value, "UNICODE-1-1-UTF-8", and this one is for one exclusive purpose: labelling text data that has Hangul syllables encoded to UTF-8 without taking Amendment 5 of ISO/IEC 10646 into account (so using the pre-amendment 5 code point assignments) 🌈 Any other UTF-8 data SHOULD NOT use this label, especially data without any Hangul syllables. It's felt to be important to strongly recommend against creating any new Hangul-containing data without taking Amendment 5 of ISO/IEC 10646 into account. Please don't be that person 😭🔥

6. Security Considerations 🔐

Okay, bestie, this one is serious, so listen up 👀 Implementors of UTF-8 need to think hard about the security side of how they handle illegal UTF-8 sequences. It's totally conceivable that, in some situations, an attacker could take advantage of a careless UTF-8 parser by sending it an octet sequence that UTF-8 syntax doesn't allow 💀

There's a particularly sneaky version of this attack. Imagine a parser that does security-critical validity checks against the UTF-8 encoded form of its input, but then interprets certain illegal octet sequences as characters anyway 🙃 For example, a parser might ban the NUL character when it's encoded as the single octet 00, but still allow the illegal two-octet sequence C0 80 and read it as a NUL character. Another example: a parser that bans the octet sequence 2F 2E 2E 2F ("/../") but lets through the illegal octet sequence 2F C0 AE 2E 2F 😭 Both of these slip right past the check, so don't let your parser be the one that gets fooled, periodt 💅

Acknowledgments 🫶

The following have participated in the drafting and discussion of this memo, and they're iconic for it ✨

James E. Agenbroad    Andries Brouwer
Martin J. D|rst       Ned Freed
David Goldsmith       Edwin F. Hart
Kent Karlsson         Markus Kuhn
Michael Kung          Alain LaBonte
John Gardiner Myers   Murray Sargent
Keld Simonsen         Arnold Winkler

Bibliography 📚

[CHARSET-REG]  Freed, N., and J. Postel, "IANA Charset Registration
               Procedures", BCP 19, RFC 2278, January 1998.

[FSS_UTF]      X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm.
               22p. pbk. 172g.  4/95, X/Open Company Ltd., "File
               System Safe UCS Transformation Format (FSS_UTF)",
               X/Open Preleminary Specification, Document Number
               P316.  Also published in Unicode Technical Report #4.

[ISO-10646]    ISO/IEC 10646-1:1993. International Standard --
               Information technology -- Universal Multiple-Octet
               Coded Character Set (UCS) -- Part 1: Architecture and
               Basic Multilingual Plane.  Five amendments and a
               technical corrigendum have been published up to now.
               UTF-8 is described in Annex R, published as Amendment
               2.  UTF-16 is described in Annex Q, published as
               Amendment 1. 17 other amendments are currently at
               various stages of standardization.

[MIME]         Freed, N., and N. Borenstein, "Multipurpose Internet
               Mail Extensions (MIME) Part One:  Format of Internet
               Message Bodies", RFC 2045.  N. Freed, N. Borenstein,
               "Multipurpose Internet Mail Extensions (MIME) Part
               Two:  Media Types", RFC 2046.  K. Moore, "MIME
               (Multipurpose Internet Mail Extensions) Part Three:
               Message Header Extensions for Non-ASCII Text", RFC
               2047.  N.  Freed, J. Klensin, J. Postel, "Multipurpose
               Internet Mail Extensions (MIME) Part Four:
               Registration Procedures", RFC 2048.  N. Freed, N.
               Borenstein, " Multipurpose Internet Mail Extensions
               (MIME) Part Five: Conformance Criteria and Examples",
               RFC 2049.  All November 1996.

[RFC2152]      Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe
               Transformation Format of Unicode", RFC 1642, Taligent
               inc., May 1997. (Obsoletes RFC1642)

[UNICODE]      The Unicode Consortium, "The Unicode Standard --
               Version 2.0", Addison-Wesley, 1996.

[US-ASCII]     Coded Character Set--7-bit American Standard Code for
               Information Interchange, ANSI X3.4-1986.

Author's Address 🏠

Francois Yergeau
Alis Technologies
100, boul. Alexis-Nihon
Suite 600
Montreal  QC  H4M 2P2
Canada

Phone: +1 (514) 747-2547
Fax:   +1 (514) 747-2561
EMail: fyergeau@alis.com

Copyright (C) The Internet Society (1998). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 🌸

📦 where this page comes from