RFC 2152
UTF-7 A Mail-Safe Transformation Format of Unicode
RFC 2152: UTF-7 🦋
Network Working Group 💖 Request for Comments: 2152. Bestie, this doc is by D. Goldsmith (Apple Computer, Inc.) and M. Davis (Taligent, Inc.) 👑. It's Informational (not a standard, see below), dated May 1997, and it obsoletes RFC 1642. It doesn't list any updates.
A Mail-Safe Transformation Format of Unicode ✨
Status of this Memo 🌸
Okay so this memo is just here to inform the Internet community. It is not a standard of any kind, so don't go treating it like one 🙈. Distribution is unlimited, so share it with everyone 💅.
Abstract 🌈
So the Unicode Standard, version 2.0, and ISO/IEC 10646-1:1993(E) (as amended) team up to define a character set we'll call Unicode from here on, and it covers most of the world's writing systems 🌍. The problem is that Internet mail (STD 11, RFC 822) only supports 7-bit US ASCII as its character set 😭. MIME (RFC 2045 through 2049) made mail way more flexible with different media types and character sets, so in theory it could carry Unicode. But MIME doesn't name Unicode as a permitted character set or say how it should be encoded, even though it does let new character sets get registered over time 🫶.
This document describes a transformation format of Unicode that uses only 7-bit ASCII octets 🎀. In the best case, when the document is made up only of US-ASCII characters, it stays readable by humans 🧚. It also explains how this format gets used with MIME and RFC 1641, "Using Unicode with MIME".
Motivation 💭
Other Unicode transformation formats exist and could be used here, like UTF-8 (also called UTF-2 or UTF-FSS) 🙄. The issue is that they use octets in the range decimal 128 through 255 to encode Unicode characters outside US-ASCII. In mail, those octets then have to be encoded again, so you're running text through two encoding passes 🤯. That makes non-US-ASCII characters way bigger, which is unfair to non-English speakers 💔. For example, UTF-8 combined with the Quoted-Printable content transfer encoding of MIME represents US-ASCII characters in one octet, but other characters may need up to nine octets 😬.
Overview 🍓
UTF-7 encodes Unicode characters as US-ASCII octets, and uses shift sequences for anything outside that range ✨. To make that work, one character from the US-ASCII repertoire gets reserved as the shift character 🔑.
Some mail gateways and systems can't handle the whole US-ASCII set (EBCDIC-based ones, for example) 😵. So UTF-7 has special provisions for encoding US-ASCII characters in a way that every mail system can handle 💜.
UTF-7 should normally be used only with 7-bit transports, like mail 📬. Everywhere else, plain Unicode or UTF-8 is the better pick 💅.
See RFC 1641, "Using Unicode with MIME", for the full overall spec on using Unicode transformation formats with MIME 📖.
Definitions 🎀
First up, the definition of Unicode 💖:
The 16 bit character set Unicode is defined by "The Unicode Standard, Version 2.0". This character set is identical with the character repertoire and coding of the international standard ISO/IEC 10646-1:1993(E); Coded Representation Form=UCS-2; Subset=300; Implementation Level=3, including the first 7 amendments to 10646 plus editorial corrections.
Note. Unicode 2.0 further specifies the use and interaction of these character codes beyond the ISO standard. However, any valid 10646 sequence is a valid Unicode sequence, and vice versa; Unicode supplies interpretations of sequences on which the ISO standard is silent as to interpretation.
Next, some handy definitions of the US-ASCII character subsets 🧁.
Set D (directly encoded characters) is made up of the following characters (taken from RFC 1521, Appendix B, which isn't in RFC 2045 anymore 🙃): the upper and lower case letters A through Z and a through z, the 10 digits 0-9, and the following nine special characters. Note that "+" and "=" are left out 🙅♀️:
Character ASCII & Unicode Value (decimal)
' 39
( 40
) 41
, 44
- 45
. 46
/ 47
: 58
? 63Set O (optional direct characters) is made up of the following characters. Note that "" and "~" are left out 🤏:
Character ASCII & Unicode Value (decimal)
! 33
" 34
# 35
$ 36
% 37
& 38
* 42
; 59
< 60
= 61
> 62
@ 64
[ 91
] 93
^ 94
_ 95
' 96
{ 123
| 124
} 125Rationale. "" and "~" are left out because lots of ASCII variants redefine them 🙈.
Set B (Modified Base 64) is the set of characters in the Base64 alphabet defined in RFC 2045, minus the pad character "=" (decimal value 61) 🫧.
Rationale. The pad character = is excluded because UTF-7 is designed for use inside header fields, as set out in RFC 2047 🎟️. The only readable encoding in RFC 2047 is "Q" (based on the Quoted-Printable from RFC 2045), so the "=" character isn't available without a ton of escape sequences 😩. This was super unfortunate but unavoidable 💀. The "=" character could otherwise have been used as the UTF-7 escape character too, instead of "+" 🥲.
Note that all characters in US-ASCII have the same value in Unicode when zero-extended to 16 bits ✨.
Use of Character Set UTF-7 Within MIME ✨
okay so UTF-7 is mail-safe 💌, which means it can be used with any content transfer encoding in MIME 🎀 (except where line length and line break restrictions get violated 🙈). Specifically, the 7 bit encoding for bodies and the Q encoding for headers are both acceptable 💖. The MIME character set tag is UTF-7, and it signifies any version of Unicode equal to or greater than 2.0 🌈.
Example 🧚. Here is a text portion of a MIME message containing the Unicode sequence "Hi Mom <WHITE SMILING FACE>!" (hexadecimal 0048, 0069, 0020, 004D, 006F, 006D, 0020, 263A, 0021).
Content-Type: text/plain; charset=UTF-7
Hi Mom +Jjo-!Example 🌸. Here is a text portion of a MIME message containing the Unicode sequence representing the Han characters for the Japanese word "nihongo" (hexadecimal 65E5,672C,8A9E).
Content-Type: text/plain; charset=UTF-7
+ZeVnLIqe-Example 🦋. Here is a text portion of a MIME message containing the Unicode sequence "A<NOT IDENTICAL TO><ALPHA>." (hexadecimal 0041,2262,0391,002E).
Content-Type: text/plain; charset=utf-7
A+ImIDkQ.Example 🍓. Here is a text portion of a MIME message containing the Unicode sequence "Item 3 is <POUND SIGN>1." (hexadecimal 0049, 0074, 0065, 006D, 0020, 0033, 0020, 0069, 0073, 0020, 00A3, 0031, 002E).
Content-Type: text/plain; charset=UTF-7
Item 3 is +AKM-1.Line breaks matter for best interoperability 🫶. Systems that may not support Unicode or MIME will have an easier time if you prep text for mail the Internet way: lines should be short and terminated with the proper SMTP CRLF sequence 📬. Unicode LINE SEPARATOR (hexadecimal 2028) and PARAGRAPH SEPARATOR (hexadecimal 2029) should be converted to SMTP line breaks. Ideally a Unicode-aware user agent handles this transparently, which would be so iconic 👑.
This prep isn't absolutely necessary, since UTF-7 and the appropriate MIME content transfer encoding can handle text that doesn't follow Internet conventions. The catch is that readability by systems without Unicode or MIME will be impaired 😭. See RFC 2045 for a discussion of mail interoperability issues.
Lines should never be broken in the middle of a UTF-7 shifted sequence, since such sequences may not cross line breaks 💅. Therefore, UTF-7 encoding should take place after line breaking. If a line containing a shifted sequence is too long after encoding, a MIME content transfer encoding such as Quoted Printable can be used to encode the text 🌷. Another possibility is to perform line breaking and UTF-7 encoding at the same time, so that lines containing shifted sequences already conform to length restrictions ✨.
Discussion 🎀
okay so this part is where we explain why we introduced UTF-7 instead of just using the existing Unicode transformation formats (like UTF-8) with MIME's content transfer encodings 🤓. First, let's list some assumptions about character frequency in typical natural language text, which we use to estimate typical storage requirements 💜:
- Most Western European languages use roughly 7/8 of their letters from US-ASCII and 1/8 from Latin 1 (ISO-8859-1).
- Most non-Roman alphabet-based languages (e.g., Greek) use about 1/6 of their letters from ASCII (since white space is in the 7-bit area) and the rest from their alphabets.
- East Asian ideographic-based languages (including Japanese) use essentially all of their characters from the Han or CJK syllabary area.
- Non-directly encoded punctuation characters do not occur frequently enough to affect the results.
Notice that current 8 bit standards, such as ISO-8859-x, require use of a content transfer encoding 👀. For comparison with what comes next, the costs break down like this (note that many of these figures are approximate since they depend on the exact composition of the text 🙏):
8859-x in Base64
Text type Average octets/character
All 1.338859-x in Quoted Printable
Text type Average octets/character
US-ASCII 1
Western European 1.25
Other 2.67Note also that Unicode encoded in Base64 takes a constant 2.67 octets per character. For purposes of comparison, we will look at UTF-8 in Base64 and Quoted Printable, and UTF-7. Also note that fixed overhead for long strings is relative to 1/n, where n is the encoded string length in octets 🧮.
UTF-8 in Base64
Text type Average octets/character
US-ASCII 1.33
Western European 1.5
Some Alphabetics 2.44
All others 4UTF-8 in Quoted Printable
Text type Average octets/character
US-ASCII 1
Western European 1.63
Some Alphabetics 5.17
All others 7-9UTF-7
Text type Average octets/character
Most US-ASCII 1
Western European 1.5
All others 2.67+2/nWe feel that UTF-8 in Quoted Printable is not viable 🙅♀️, due to the very large expansion of all text except Western European. It would only be viable in texts made of large expanses of US-ASCII or Latin characters with occasional other characters sprinkled in. We would prefer to introduce one encoding that works reasonably well for all users 🫶.
We also feel that UTF-8 in Base64 has high expansion for non-Western-European users, and is less desirable because it cannot be read directly, even when the content is largely US-ASCII 😤. The base encoding of UTF-7 gives competitive results and is readable for ASCII text 🦋.
UTF-7 gives results competitive with ISO-8859-x, with access to all of the Unicode character set 🌈. We believe this justifies the introduction of a new transformation format of Unicode 👑.
As an alternative to UTF-7, it might be possible to intermix Unicode characters with other character sets using an existing MIME mechanism, the multipart/mixed content type. We're ignoring the issues with line breaks for the moment (thanks to Nathaniel Borenstein for suggesting this 💕). For instance (repeating an earlier example):
Content-type: multipart/mixed; boundary=foo
Content-Disposition: inline
--foo
Content-type: text/plain; charset=us-ascii
Hi Mom
--foo
Content-type: text/plain; charset=UNICODE-2-0
Content-transfer-encoding: base64
Jjo=
--foo
Content-type: text/plain; charset=us-ascii
!
--foo--Theoretically, this removes the need for UTF-7 in message bodies (multipart may not be used in header fields). However, we feel that as use of the Unicode character set becomes more widespread, intermittent use of specialized Unicode characters (such as dingbats and mathematical symbols) will occur. Text will also typically include small snippets from other scripts, such as Cyrillic, Greek, or East Asian languages. Anything in the Roman script is already handled adequately by existing MIME character sets 💅. Although the multipart technique works well for large chunks of text in alternating character sets, we feel it does not adequately support the kinds of uses just discussed, so we still believe the introduction of UTF-7 is justified 🎀.
Summary ✨
The UTF-7 encoding allows Unicode characters to be encoded within the US-ASCII 7 bit character set 🦋. It is most effective for Unicode sequences that contain relatively long strings of US-ASCII characters interspersed with either single Unicode characters or strings of Unicode characters 🌸. This matters because it lets the US-ASCII portions be read on systems without direct Unicode support 🫶.
UTF-7 should only be used with 7 bit transports such as mail 📬. In other contexts, use of straight Unicode or UTF-8 is preferred 💜.
Acknowledgements 💖
Many thanks to the following people for their contributions, comments, and suggestions 🙌. If we have omitted anyone, it was through oversight and not intentionally 🥺.
- Glenn Adams
- Harald T. Alvestrand
- Nathaniel Borenstein
- Lee Collins
- Jim Conklin
- Dave Crocker
- Steve Dorner
- Dana S. Emery
- Ned Freed
- Kari E. Hurtta
- John H. Jenkins
- John C. Klensin
- Valdis Kletnieks
- Keith Moore
- Masataka Ohta
- Einar Stefferud
- Erik M. van der Poel
Appendix A -- Examples 🎀
okay so this is the longer example, and it's giving real-world 💅. it was originally in Big5 code and got condensed for brevity. there are two versions: the first uses optional characters from set O (so it may not pass through some mail gateways), and the second doesn't. 🦋
Content-type: text/plain; charset=utf-7
Below is the full Chinese text of the Analects (+itaKng-).
The sources for the text are:
"The sayings of Confucius," James R. Ware, trans. +U/BTFw-:
+ZYeB9FH6ckh5Pg-, 1980. (Chinese text with English translation)
+Vttm+E6UfZM-, +W4tRQ066bOg-, +UxdOrA-: +Ti1XC2b4Xpc-, 1990.
"The Chinese Classics with a Translation, Critical and Exegetical
Notes, Prolegomena, and Copius Indexes," James Legge, trans., Taipei:
Southern Materials Center Publishing, Inc., 1991. (Chinese text with
English translation)
Big Five and GB versions of the text are being made available
separately.
Neither the Big Five nor GB contain all the characters used in this
text. Missing characters have been indicated using their Unicode/ISO
10646 code points. "U+-" followed by four hexadecimal digits
indicates a Unicode/10646 code (e.g., U+-9F08). There is no good
solution to the problem of the small size of the Big Five/GB
character sets; this represents the solution I find personally most
satisfactory.
(omitted...)
I have tried to minimize this problem by using variant characters
where they were available and the character actually in the text was
not. Only variants listed as such in the +XrdxmVtXUXg- were used.
(omitted...)
John H. Jenkins +TpVPXGBG- jenkins@apple.com 5 January 1993
(omitted...)the big thing to clock in that first version is that the characters are all encoded in UTF-7 using the +...- shifted sequences, so it's a little messy to read, but the Chinese text is all in there 🍓 ✨
now here's the second version, same text, but this one doesn't use any of those optional set O characters, so it's the one that's more likely to make it through mail gateways 🧚 🫶
Content-type: text/plain; charset=utf-7
Below is the full Chinese text of the Analects (+itaKng-).
The sources for the text are:
+ACI-The sayings of Confucius,+ACI- James R. Ware, trans. +U/BTFw-:
+ZYeB9FH6ckh5Pg-, 1980. (Chinese text with English translation)
+Vttm+E6UfZM-, +W4tRQ066bOg-, +UxdOrA-: +Ti1XC2b4Xpc-, 1990.
+ACI-The Chinese Classics with a Translation, Critical and Exegetical
Notes, Prolegomena, and Copius Indexes,+ACI- James Legge, trans.,
Taipei: Southern Materials Center Publishing, Inc., 1991. (Chinese
text with English translation)
Big Five and GB versions of the text are being made available
separately.
Neither the Big Five nor GB contain all the characters used in this
text. Missing characters have been indicated using their Unicode/ISO
10646 code points. +ACI-U+-+ACI- followed by four hexadecimal digits
indicates a Unicode/10646 code (e.g., U+-9F08). There is no good
solution to the problem of the small size of the Big Five/GB
character sets+ADs- this represents the solution I find personally
most satisfactory.
(omitted...)
I have tried to minimize this problem by using variant characters
where they were available and the character actually in the text was
not. Only variants listed as such in the +XrdxmVtXUXg- were used.
(omitted...)
John H. Jenkins +TpVPXGBG- jenkins+AEA-apple.com 5 January 1993
(omitted...)Security Considerations 🔐
okay bestie, this one's short and sweet: security issues are not discussed in this memo 💀 so there's nothing more to say on that front here.
References 📚
[UNICODE 2.0] "The Unicode Standard, Version 2.0", The Unicode
Consortium, Addison-Wesley, 1996. ISBN 0-201-48345-9.
[ISO 10646] ISO/IEC 10646-1:1993(E) Information Technology--Universal
Multiple-octet Coded Character Set (UCS). See also
amendments 1 through 7, plus editorial corrections.
[RFC-1641] Goldsmith, D., and M. Davis, "Using Unicode with MIME",
RFC 1641, Taligent, Inc., July 1994.
[US-ASCII] Coded Character Set--7-bit American Standard Code for
Information Interchange, ANSI X3.4-1986.
[ISO-8859] Information Processing -- 8-bit Single-Byte Coded Graphic
Character Sets -- Part 1: Latin Alphabet No. 1, ISO
8859-1:1987. Part 2: Latin alphabet No. 2, ISO 8859-2,
1987. Part 3: Latin alphabet No. 3, ISO 8859-3, 1988.
Part 4: Latin alphabet No. 4, ISO 8859-4, 1988. Part 5:
Latin/Cyrillic alphabet, ISO 8859-5, 1988. Part 6:
Latin/Arabic alphabet, ISO 8859-6, 1987. Part 7:
Latin/Greek alphabet, ISO 8859-7, 1987. Part 8:
Latin/Hebrew alphabet, ISO 8859-8, 1988. Part 9: Latin
alphabet No. 5, ISO 8859-9, 1990.
[RFC822] Crocker, D., "Standard for the Format of ARPA Internet
Text Messages", STD 11, RFC 822, UDEL, August 1982.
[MIME] Borenstein N., N. Freed, K. Moore, J. Klensin, and J.
Postel, "MIME (Multipurpose Internet Mail Extensions)
Parts One through Five", RFC 2045, 2046, 2047, 2048, and
2049, November 1996.Authors' Addresses 🌈
David Goldsmith
Apple Computer, Inc.
2 Infinite Loop, MS: 302-2IS
Cupertino, CA 95014
Phone: 408-974-1957
Fax: 408-862-4566
EMail: goldsmith@apple.com
Mark Davis
Taligent, Inc.
10201 N. DeAnza Blvd.
Cupertino, CA 95014-2233
Phone: 408-777-5116
Fax: 408-777-5081
EMail: mark_davis@taligent.com📦 where this page comes from
- ietf: rfc2152 1997-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms