RFC 2044

UTF-8, a transformation format of Unicode and ISO 10646

📅 published October 1996, from ietf

RFC 2044 🎀 UTF-8, a transformation format of Unicode and ISO 10646

Okay bestie, the paperwork first 📋✨ This is Request for Comments 2044 from the Network Working Group. The author is F. Yergeau of Alis Technologies. It's Category: Informational (not a standard!), dated October 1996. It doesn't obsolete or update anything, since its header doesn't list anything.

Status of This Memo 🌸

This memo gives information to the Internet community. It doesn't specify an Internet standard of any kind, and distribution is totally unlimited 🦋

Abstract 🔥

Okay so the Unicode Standard (version 1.1) and ISO/IEC 10646-1:1993 teamed up to define a 16-bit character set that covers most of the world's writing systems, which is iconic 👑 BUT 16-bit characters don't play nicely with a lot of current apps and protocols 😭 That's why people made a few so-called UCS transformation formats (UTF), each with its own vibe. UTF-8 (the main character of this memo) keeps the full US-ASCII range intact 💖 US-ASCII characters get encoded as one octet with their usual US-ASCII value, and any octet with that value can only be a US-ASCII character. So software that relies on US-ASCII values keeps working, and it's transparent to everything else ✨

1. Introduction 💅

So the Unicode Standard, version 1.1 [UNICODE], and ISO/IEC 10646-1:1993 [ISO-10646] together define a 16-bit character set called UCS-2, which covers most of the world's writing systems 🌍 ISO 10646 also defines a 31-bit set called UCS-4, but nothing outside the UCS-2 region (the Basic Multilingual Plane, BMP) is assigned yet. Honestly, UCS-2 and UCS-4 are a pain in many apps and protocols that assume 8-bit or even 7-bit characters 🙈 Even newer systems that handle 16-bit characters can't process UCS-4 data. So people came up with UCS transformation formats (UTF), each with different characteristics 🧚

UTF-1 is only of historical interest now, since it was removed from ISO 10646. UTF-7 encodes the full Unicode repertoire using only octets with the high-order bit clear (7-bit US-ASCII values, [US-ASCII]), which makes it a mail-safe encoding ([RFC1642]) 💌 UTF-8 (our girl) uses all the bits of an octet, but it's iconic because it preserves the full US-ASCII range:

US-ASCII characters are encoded in one octet with the normal US-ASCII value, and any octet with that value can only stand for a US-ASCII character, and nothing else.

UTF-16 is a scheme that transforms a subset of the UCS-4 repertoire into a pair of UCS-2 values from a reserved range 🫶 This affects UTF-8 because UCS-2 values from that reserved range have to be handled specially in the UTF-8 transformation.

UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of octets. The count and the value of each depend on the integer value the character has in ISO 10646 🔢 This format has these characteristics (all values are in hexadecimal):

Character values from 0000 0000 to 0000 007F (the US-ASCII repertoire) map to octets 00 to 7F (7-bit US-ASCII values).
US-ASCII values don't show up anywhere else in a UTF-8 encoded character stream. This keeps it compatible with file systems and other software (like the printf() function in C libraries) that parse based on US-ASCII values but are transparent to other values 🧠
Round-trip conversion is easy between UTF-8 and any of UCS-4, UCS-2, or Unicode 🔁
The first octet of a multi-octet sequence tells you how many octets are in the sequence 🎯
Character boundaries are easy to find from anywhere in an octet stream 🗺️
The lexicographic sorting order of UCS-4 strings is preserved. Of course that's of limited interest, since the sort order isn't culturally valid in either case 🙃
The octet values FE and FF never appear 🚫

UTF-8 started as a project of the X/Open Joint Internationalization Group (XOJIG). The goal was a File System Safe UCS Transformation Format [FSS-UTF] that works with UNIX systems and supports multilingual text in a single encoding 🌈 The original authors were Gary Miller, Greger Leijonhufvud and John Entenmann. Later on, Ken Thompson and Rob Pike did a ton of significant work on the formal UTF-8 💪

You can also find a description in Unicode Technical Report #4 [UNICODE]. The definitive reference, including provisions for UTF-16 data within UTF-8, is Annex R of ISO/IEC 10646-1 [ISO-10646] 📖

2. UTF-8 definition 🎀

In UTF-8, characters are encoded using sequences of 1 to 6 octets. A "sequence" of one octet has its higher-order bit set to 0, and the other 7 bits encode the character value. In a sequence of n octets (n>1), the first octet has its n higher-order bits set to 1, followed by a bit set to 0. The remaining bit(s) of that octet hold bits from the character value. The following octets all have the higher-order bit set to 1 and the next bit set to 0, which leaves 6 bits in each for the character value 🧩

The table below sums up the format of these octet types. The letter x marks bits available for encoding bits of the UCS-4 character value:

UCS-4 range (hex.)           UTF-8 octet sequence (binary)
0000 0000-0000 007F   0xxxxxxx
0000 0080-0000 07FF   110xxxxx 10xxxxxx
0000 0800-0000 FFFF   1110xxxx 10xxxxxx 10xxxxxx

0001 0000-001F FFFF   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03FF FFFF   111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7FFF FFFF   1111110x 10xxxxxx ... 10xxxxxx

Here's how encoding from UCS-4 to UTF-8 goes:

Work out how many octets you need from the character value and the first column of the table above.
Set up the high-order bits of the octets according to the second column of the table.
Fill in the bits marked x with bits from the character value. Start with the lower-order bits and put them first in the last octet of the sequence, then the next-to-last, and so on, until every x bit is filled in 🫧

The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can, in principle, be worked out from the above by just extending each UCS-2 character with two zero-valued octets. But UCS-2 values between D800 and DFFF are really UCS-4 characters that went through UTF-16, so they need special treatment: you have to undo the UTF-16 transformation, which gives you a UCS-4 character that then gets transformed as above 🔄

Here's how decoding from UTF-8 to UCS-4 goes:

Set all bits of the 4 octets of the UCS-4 character to 0.
Work out which bits encode the character value from the number of octets in the sequence and the second column of the table above (the bits marked x).
Distribute the bits from the sequence into the UCS-4 character. Start with the lower-order bits from the last octet of the sequence and keep moving left until no x bits are left 🎯

If the UTF-8 sequence is no more than three octets long, decoding can go straight to UCS-2 (or equivalently Unicode) ✨

A more detailed algorithm and formulae can be found in [FSS_UTF], [UNICODE] or Annex R to [ISO-10646] 📚

3. Examples 🍓

The Unicode sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391, 002E) encodes like this:

41 E2 89 A2 CE 91 2E

The Unicode sequence "Hi Mom <WHITE SMILING FACE>!" (0048, 0069, 0020, 004D, 006F, 006D, 0020, 263A, 0021) encodes like this:

48 69 20 4D 6F 6D 20 E2 98 BA 21

The Unicode sequence for the Han characters of the Japanese word "nihongo" (65E5, 672C, 8A9E) encodes like this:

E6 97 A5 E6 9C AC E8 AA 9E

MIME registrations 🦋

This memo is meant to be the basis for registering a MIME character encoding (charset) under [RFC1521]. The proposed charset parameter value is "UTF-8". That label would go on media types containing text made of characters from the ISO 10646-1 repertoire, encoded into a sequence of octets using the scheme outlined above 🫶

Security Considerations 💀

Security issues are not discussed in this memo. 😱 Read that again: nothing about security is covered here, so keep that in mind as you go! 🙈

Acknowledgments 🌟

These lovely people helped with drafting and discussing this memo 💖

   James E. Agenbroad   Andries Brouwer
   Martin J. D|rst      David Goldsmith
   Edwin F. Hart        Kent Karlsson
   Markus Kuhn          Michael Kung
   Alain LaBonte        Murray Sargent
   Keld Simonsen        Arnold Winkler

Bibliography 📚

[FSS_UTF]      X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm.
               22p. pbk. 172g.  4/95, X/Open Company Ltd., "File Sys-
               tem Safe UCS Transformation Format (FSS_UTF)", X/Open
               Preleminary Specification, Document Number P316.  Also
               published in Unicode Technical Report #4.

[ISO-10646]    ISO/IEC 10646-1:1993. International Standard -- Infor-
               mation technology -- Universal Multiple-Octet Coded
               Character Set (UCS) -- Part 1: Architecture and Basic
               Multilingual Plane.  UTF-8 is described in Annex R,
               adopted but not yet published.  UTF-16 is described in
               Annex Q, adopted but not yet published.

[RFC1521]      Borenstein, N., and N. Freed, "MIME (Multipurpose
               Internet Mail Extensions) Part One: Mechanisms for
               Specifying and Describing the Format of Internet Mes-
               sage Bodies", RFC 1521, Bellcore, Innosoft, September
               1993.

[RFC1641]      Goldsmith, D., and M. Davis, "Using Unicode with
               MIME", RFC 1641, Taligent inc., July 1994.

[RFC1642]      Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe
               Transformation Format of Unicode", RFC 1642,
               Taligent, Inc., July 1994.

[UNICODE]      The Unicode Consortium, "The Unicode Standard --
               Worldwide Character Encoding -- Version 1.0", Addison-
               Wesley, Volume 1, 1991, Volume 2, 1992.  UTF-8 is
               described in Unicode Technical Report #4.

[US-ASCII]     Coded Character Set--7-bit American Standard Code for
               Information Interchange, ANSI X3.4-1986.

Author's Address 🎀

   Francois Yergeau
   Alis Technologies
   100, boul. Alexis-Nihon
   Suite 600
   Montreal  QC  H4M 2P2
   Canada

   Tel: +1 (514) 747-2547
   Fax: +1 (514) 747-2561
   EMail: fyergeau@alis.com

📦 where this page comes from

ietf: rfc2044 1996-10
💖 girlified by claude-haiku-5-5 on 2026-10-15: a language model's rewrite of the original, under its terms