RFC 2130

The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996

📅 published April 1997, from ietf

Network Working Group RFC 2130 ✨

RFC 2130: The Report of the IAB Character Set Workshop, held 29 February - 1 March, 1996 🎀

Okay bestie, let's get the vibe check done first. This is an Informational RFC from the Network Working Group, dated April 1997. It's a report, so it's not a standard. 💅

The authors are:

  • C. Weider, Microsoft
  • C. Preston, Preston & Lynch
  • K. Simonsen, DKUUG
  • H. Alvestrand, UNINETT
  • R. Atkinson, Cisco Systems
  • M. Crispin, University of Washington
  • P. Svanberg, KTH

The title block doesn't list any Obsoletes or Updates entries. 🦋

Status of this Memo 📜

So this memo gives information to the Internet community. It is not an Internet standard of any kind, and distribution of it is unlimited. 🌸

Acknowledgments 🫶

The authors are so grateful to the Information Sciences Institute (ISI), especially Joyce K. Reynolds, for graciously hosting the event. Joe Kemp and Jeanine Yamazaki at ISI made sure the facilities were perfect. The Internet Society also deserves a shoutout because they paid travel for participants who might not have been able to come otherwise. 💖

The authors also thanked the many experts who joined the workshop and the mailing list. You can find the full list in Appendix D. And Bunyip Information Systems provided the mailing list facilities, which was super kind of them. 🌈

Table of Contents 🗂️

Abstract
0:    Executive summary..........................................   2
1:    Introduction...............................................   3
2:    Character sets on the Internet -- the problem..............   3
2.1:  Character set handling in existing protocols...............   4
3:    Architectural model........................................   6
3.1:  Segments defined...........................................   7
3.2:  On the wire................................................   8
3.3:  Determining which values of CCS, CES, and TES are used.....   9
3.4:  Recommended Defaults.......................................  10
3.5:  Guidelines for conversions between coded character sets....  13
4:    Presentation issues........................................  14
5:    Open issues................................................  14
5.1:  Language tags..............................................  15
5.2:  Public identifiers.........................................  16
5.3:  Bi-directionality..........................................  16
6:    Security Considerations....................................  16
7:    Conclusions................................................  16
8:    Recommendations............................................  17
8.1:  To the IAB.................................................  17
8.2:  For new Internet protocols.................................  18
8.3:  For registration of new character sets.....................  18
Appendix A: List of protocols affected by character set issues...  20
Appendix B: Acronyms.............................................  23
Appendix C: Glossary.............................................  24
Appendix D: References...........................................  25
Appendix E: Recommended reading..................................  27
Appendix F: Workshop attendee list...............................  29
Appendix G: Authors' Addresses...................................  30

Abstract 🦋

This report shares what the IAB-sponsored invitational workshop, held 29 February - 1 March, 1996, figured out about using character sets on the Internet. It explains why Internet protocols that send text need character set handling. It also gives a conceptual framework for specifying character sets. The report recommends MIME tagging for transmitted text, and it recommends a default character set without saying that other character sets aren't needed. Finally, it makes a list of recommendations to the IAB, IANA, and the IESG for getting the character set framework built into text transmission protocols. 💜

0. Executive summary ✨

"Character Set" means a lot of different things to different people. Even the MIME registry of character sets lists things that are super different in their semantics and where they're used. 🙈 This workshop gives the IAB and IETF guidance on using character sets on the Internet. It also offers a shared framework so the many characters in use out there can interoperate. 🌸

The framework has four parts. First is an architecture model, which names the pieces needed to send text on the wire. Second are recommendations for tagging transmitted (and stored) text. Third are recommended defaults for each level of the model. Last is a set of recommendations to the IAB, IANA, and the IESG for getting this framework into text transmission protocols. 🔥

The architectural model has 7 layers, but only three are required for on-the-wire transmission. ngl, that's a big deal. 👑 A Coded Character Set is a mapping from a set of abstract characters to a set of integers. A Character Encoding Scheme is a mapping from a Coded Character Set (or several) to a set of octets. A Transfer Encoding Syntax is a transformation applied to data that has already been encoded with a Character Encoding Scheme, so it can be transmitted. These layers should be specified in a transmitted text stream using the MIME encoding mechanisms. 🧚

Here's the main recommendation. For new protocols, or new versions of old protocols that send text, use ISO 10646 as the default Coded Character Set and UTF-8 as the default Character Encoding Scheme. These defaults don't deprecate other character sets when they're needed. They're just guidance and a spec that helps everyone interoperate. 💅

1. Introduction 🎀

This is the report of an IAB-sponsored invitational workshop on using character sets on the Internet. It was held 29 February - 1 March 1996 at the Information Sciences Institute (ISI) in Marina del Rey, California. The report also covers the mailing list discussion up to and a little past the workshop itself. 🦋

The goals were to give the IAB and the IETF guidance on using character sets on the Internet and, if possible, a common framework for interoperability between the many character sets in use. Both goals were achieved. 🎉

2. Character sets on the Internet - the problem 🌈

The term 'character set' is usually used for the contents of lots of text transmission and display protocols on the Internet. Because it means different things to different people, things got confusing. For example, the MIME registry of character sets [MIME] has items that can be really different in how useful and what they mean across various Internet protocols. 🙃

There's also a huge pile of different text encoding schemes in use on the Internet. Honestly, that's not a problem on its own, because each scheme grew up to meet real needs. 🌷 The issue is that information applications like mail, directories, and the World Wide Web each came up with their own techniques for handling the growing number of schemes. A robust information architecture for the Internet needs as much interoperability between those techniques as possible. 🫶

Showing plain text sent over the Internet correctly takes a lot of info about the text, like the underlying character set, the language, and so on. 🧠 If the receiving app wants to use local (cultural) conventions when it shows the data, it needs even more formatting info. That formatting info is what's needed to turn things like dates, times, numbers, and monetary notation into a form the user recognises. The POSIX [POSIX] notion of locale covers language, coded character set, and cultural conventions. 💖

To avoid pointless debate and make the most of the workshop's time, the group declared these issues out of scope:

  • glyphs
  • sorting
  • culture (e.g. do we present the American or British spelling?)
  • user interface issues
  • internal representation of textual data
  • included characters (why aren't certain characters available in any character set?)
  • locale (in the POSIX sense)
  • font registration
  • semantics
  • user input/output issues
  • Han unification issues

There were some related issues that did make it onto the discussion list. The most important one was the 'locale' components needed for transporting and identifying multilingual texts. 🌍

2.2. Character Set handling in existing protocols 🔍

One of the group's biggest worries was that the character set handling framework must not break existing protocols. 😤 Keeping that in mind, the group looked at how character sets are already being used in existing protocols. Appendix A has a list of those protocols, plus some recommendations for change. 📋

2.2.1. General comments 💭

The problem areas fall into three main categories: protocols, identifiers, and data. 🗂️

2.2.1.1. Protocols 🔧

The protocol machinery SHOULD NOT be changed. For example, letting SMTP [SMTP] use both MAIL FROM and POST FRA would be dangerous for the stability of the protocol. 😬 However, lots of protocols carry error messages and other info meant for humans to read. It MIGHT be a good idea to let those be localised into a specific language and character set, instead of staying in English and US-ASCII [ASCII]. If that happens, new extensions should follow the framework outlined below. 🌸

Part 1 of 6 done, bestie! Part 2 is coming up next with the architectural model. 🦋

2.2.1.2: Identifiers ✨

Okay bestie, the IAB gave a really strong direction in RFC 1958 [RFC 1958] 💅, and it goes like this:

4.3 Public (i.e. widely visible) names should be in case
    independent ASCII.  Specifically, this refers to DNS names,
    and to protocol elements that are transmitted in text format.
    ...
5.4 Designs should be fully international, with support for
    localization (adaptation to local character sets). In
    particular, there should be a uniform approach to character
    set tagging for information content.

So for protocols that have only ever used US-ASCII, UTF-8 [UTF-8] is a super simple upgrade path 🦋. BUT the switch has to be negotiated, either by negotiating a protocol version or by negotiating charset usage. And a fallback to a US-ASCII compatible representation, like UTF-7 [UTF-7], MUST be available 🫶. No skipping that part!

The need to pass application data like language along with individual identifiers depends on the application 🌸. Protocols SHOULD check how much they actually need that when they design their mechanisms. Also, forcing the ASCII requirement onto identifiers that only live in a local context (like private mailbox folder names) is both unrealistic and unreasonable 🙈. For those, you should instead think about how to keep character set handling consistent.

2.2.1.3: Data

Data that needs character set handling includes text, databases, and HTML [HTML] pages, for example 📄. For all of these, supporting multiple character sets and proper application information is absolutely vital 😭, and it MUST be supported. Non-negotiable, periodt 💖.

2.3. Architectural requirements

Okay so to tackle the issues on this list, the team first made an architectural model 🧚 that sets out the components needed to fully specify how textual data gets transmitted. A lot of these will look familiar if you've used encoding protocols like MIME. We're not going to dig into all of them here, though. We mostly focus on the parts needed to specify the 'on-the-wire' phase of text transmission ✨.

Mandating one single, all-encompassing character set would not fit well with the IETF philosophy of planning for architectural diversity 🌈. So the best we can do is provide a common framework for identifying and using the huge pile of character sets on the Internet. It would be a major win if the total number of Coded Character Sets could be kept to a minimum 👑. This framework should meet these requirements:

  • it should not break existing protocols (because then the likelihood of deployment is very small),
  • it should allow the use of character sets currently used on the Internet, and
  • it should be relatively easy to build into new protocols.

3. Architectural model 🎀

The basic architectural model that guided our discussions is shown below. The big distinction was between the segments needed to successfully transmit character set data on-the-wire and the segments needed to present that data to a user in a way they can actually understand 👀. Our discussions were mainly limited to the segments that specify the 'on-the-wire' transmission of textual data.

User interface issues: these are briefly discussed in Section 3.1.1.
     Layout
     Culture
     Locale
     Language
On-the-wire: see section 3.2 for detailed discussion.
     Transfer Syntax
     Character Encoding Scheme
     Coded Character Set

3.1: Segments defined 🧁

3.1:1: User interface

3.1.1.1: Layout

Layout covers the stuff needed for displaying text to the user, like font selection, word-wrapping, etc. 💅 It's kinda like the 'presentation' layer in the 7-layer ISO telecommunications model [ISO-7498].

3.1.1.2: Culture

Culture is all about cultural preferences, which affect spelling, word choice, and so forth 🌷.

3.1.1.3: Locale

The locale component has the info needed to make choices about text manipulation so the text shows up in a format people expect 🗓️. This can include how dates, times, and monetary symbols are displayed. Locale changes are usually applied to a text stream before it's shown to the user, though they're also used to specify input formats ⌨️.

3.1.1.4: Language

This component specifies the language of the transmitted text 🌍. Sometimes, in specific cases, language info is needed to hit a certain quality level when displaying a text stream. For example, UTF-8 encoded Han may need a language tag so the right glyphs get picked for display at a particular quality level ✨.

Other info besides language can also get you the required quality in the display process. In particular, a font tag is enough to produce identical results 🎨. But tying a language to a block of text is useful way beyond display. As the amount of info in multiple languages on the World Wide Web keeps growing, it becomes critical to specify which language a document is in, so automatic indexing and retrieval can find relevant documents 🔍.

The term 'language tag' should be reserved for the short identifier from RFC 1766 [RFC-1766] that only serves to identify the language 🏷️. Other text attributes that are closely tied to the document's language, like desired font or text direction, should be specified with other identifiers instead of overloading the language tag 🙅‍♀️.

3.2: On the wire 🦋

There are three segments of the model that are required for completely specifying the content of a transmitted text stream (with the occasional exception of the Language component, mentioned above) 💖. They are:

  1. Coded Character Set,
  2. Character Encoding Scheme, and
  3. Transfer Encoding Syntax.

Each of these abstract components must be explicitly specified by the transmitter when the data is sent 📤. Sometimes it's implicit because of the protocol or standard being used (like ANSI/NISO Z39.50). Also, in MIME, the Coded Character Set and Character Encoding Scheme are specified by the Charset parameter to the Content-Type header field, and the Transfer Encoding Syntax is specified by the Content-Transfer-Encoding header field 🧠.

3.2.1: Coded Character Set

A Coded Character Set (CCS) is a mapping from a set of abstract characters to a set of integers 🔢. Examples of coded character sets are ISO 10646 [ISO-10646], US-ASCII [ASCII], and the ISO-8859 series [ISO-8859].

3.2.2: Character Encoding Scheme

A Character Encoding Scheme (CES) is a mapping from a Coded Character Set, or several coded character sets, to a set of octets 🧩. Examples are ISO 2022 [ISO-2022] and UTF-8 [UTF-8]. A given CES is typically associated with a single CCS; for example, UTF-8 applies only to ISO 10646 💅.

3.2.3: Transfer Encoding Syntax

It's really common to need to transform encoded text into a format that specific protocols can transmit 🚚. The Transfer Encoding Syntax (TES) is a transformation applied to character data encoded using a CCS and possibly a CES so it can be transmitted. Examples are Base64 Encoding [Base64], gzip encoding, and so forth 🎁.

3.3: Determining which values of CCS, CES, and TES are used 🔎

To completely specify which CCS, CES, and TES are used in a specific text transmission, there needs to be a consistent set of labels for saying which CCS, CES, and TES are in use 🏷️. Once the right mechanisms have been picked, there are six techniques for attaching these labels to the data 🌟.

The labels themselves are named and registered, either with IANA [IANA] or with some other registry 📋. Ideally, their definitions can be retrieved from some registration authority 🗂️.

Labels may be determined in one of the following ways:

  • Determined by guessing, where the receiver of the text has to guess the values of the CCS, CES, and TES. For example: "I got this from Sweden so it's probably ISO-8859-1." This is obviously not a very foolproof way to decode text 🙃.
  • Determined by the standard, where the protocol used to transmit the data has made documented choices of CCS, CES, and TES in the standard. So the encodings are known through the access protocol. For example, HTTP [HTTP] uses (but is not limited to) ISO-8859-1, and SMTP uses US-ASCII 📬.
  • Attached to the transfer envelope, where the descriptive labels are attached to the wrapper placed around the text for transport. MIME headers are a good example of this technique 💌.
  • Included in the data stream, where the data stream itself has been encoded in a way that signals the character set used. For example, ISO-2022 encodes the data with escape sequences to give information on the character subset currently being used 🪄.
  • Agreed by prior bilateral agreement, where some out-of-band negotiation has let the text transmitter and receiver determine the CCS, CES, and TES for the transmitted text 🤝.
  • Agreed to by negotiation during some phase, typically initialization of the protocol 🔄.

3.3.1: Recommendations for value specification mechanisms

Okay so each of these techniques (except guessing) is useful in particular situations, BUT interoperability needs a more consistent set of techniques 🫶. So we recommend that MIME registered values be used for all tagging of character sets and languages UNLESS there is an existing mechanism for determining the required information using one of the other techniques (except guessing). This recommendation is going to take a fair bit of work from protocol designers, implementors, the IETF, the IESG, and the IAB 😅.

However, it's important to point out that the MIME concept of 'charset' in some cases cuts across several layers of components in our model 🌀. While that can be accepted in existing registrations, we also recommend that the MIME registration procedure for character sets be modified to show how a proposed character set deals with the CCS and the CES. Most 'charsets' have a well defined CCS and CES, so they should merely be teased apart for the registration ✂️💜.

There are a number of other recommendations, but we'll cover those in the next sections 👀.

okay so here's the tea 🫖: you can't make ONE mandatory set of defaults for every Internet protocol, bestie 😭. There's a whole mountain of existing practice, future protocols will have different jobs, and those jobs decide how they handle text. Like, mail is basically all text, so coded character sets are a huge deal there. And since everyone expects to email everyone, mail protocols have to be super good at handling different character set encodings 💅. But if a protocol barely uses strings, it doesn't need a fancy multi-charset setup, as long as the one charset it picked can handle everything the protocol needs 🧚. This also goes for the way character set parameters get specified. If only one character set encoding is needed, just make it explicit in the protocol spec. Protocols that need more character set support will need a more elaborate specification technique 🎀.

3.4.1. Clarity of specification

we recommend that each protocol clearly specify what it is using for each layer of the transmission model 📋. Users (or clients) should never have to guess what the parameter is for a given layer 🙈.

3.4.2. Default Coded Character Set:

the default Coded Character Set is the repertoire of ISO-10646 🌈.

3.4.3. Default Character Encoding Scheme

for text-oriented protocols, new protocols should use UTF-8 💜, and protocols with a backwards compatibility requirement should use the default of the existing protocol. So US-ASCII for mail, and ISO-8859-1 for HTTP 🍓. The recommended specification scheme is the MIME "charset" specification, using the IANA "charset" specifications. The MIME specifications will need to be clarified to meet this model in the future 🔮.

for other protocols, the default should be UTF-8, because it lets US-ASCII get entered as-is and unlocks the full repertoire of ISO 10646 ✨.

some protocols, like the ones descended from SGML [SGML], have other natural notations for characters outside their "natural" repertoire. For example, HTML [HTML] lets you use &#nnnn to refer to any ISO 10646 character 🦋. Note that this, like all other encodings that depend on "escape characters", redefines at least one character from the base character set to act as an indicator of "foreign" characters. Use of this approach must be weighed very carefully ⚠️.

⚠️ escape characters redefine at least one character from the base set, so weigh that very carefully!

3.4.4. Default Transport Encoding Scheme

there is no recommended default for this level 🤷‍♀️. For plain text oriented protocols, the bytestream transport format should be 8-bit clean, possibly with normalization of end-of-line indicators. Some special cases could be made for protocols that are not 8-bit clean, such as encoding it for transport over 7-bit connections. For binary, the same recommendation holds as above. The specification technique should either be defined in the protocol, if only one way is permitted, or use MIME content-transfer-encoding (CTE) techniques, with IANA registered values 🧃.

3.4.5. Default Language

there is no recommended default for the language level 🌍. For human readable text, there should always be a way to specify the natural language 🗣️. The specification technique should be a MIME identifier with IANA registered values for languages. If headers are used, the header should be Content-Language 💖.

3.4.6. Default Locale

the default should be the POSIX locale 💻. The specification technique should use the Cultural register of CEN ENV 12005 [CEN] for the values. If headers are used, the header should be Content-Locale 🌸.

3.4.7. Default Culture

there is no recommended default for the Culture level 🎭. The specification technique should be a MIME or MIME-like identifier (e.g. Content-Culture) and should use the Cultural register of CEN ENV 12005 for its values 🪩.

3.4.8. Default Presentation

there is no recommended default for the Presentation level 🖼️. The specification technique should be a MIME or MIME-like identifier (e.g. Content-Layout) and use the glyph register of ISO 10036 and other registers for its values ✏️.

3.4.9. Multiplexing

in some cases, text transmission needs a bunch of different values for one parameter. For example, English annotation of Japanese text might totally require shifting the Content-Language parameter 🎌. How you switch a parameter's value inside one body of text depends on the application. For instance, the HTML I18N [I18N] work gives most of its elements a language attribute, including <SPAN>, <HTML>, and <BODY>, so you can switch between languages 🔀. When only one value is needed, it should be as general as possible and specified in the protocol standard with a reference to the IANA or other registry value 🫶. All levels should be specified explicitly 📣.

3.4.10. Storage

okay this one's important 🚨. Because stored text may very well be stored without any of the extra info needed for decoding, stored text SHOULD be tagged in a MIME compliant fashion 💾. This fixes the problem of not being able to interpret text that's been stored for a long time, or text whose provenance isn't available 😭.

3.5. Guidelines for conversions between coded character sets 🔁

this section covers various algorithms for converting a source text S, encoded in the coded character set CCS(S), into a target text T, encoded in the coded character set CCS(T) 💫.

Rep(X) is the character repertoire of coded character set X, meaning the set of characters that can be represented with X 📚.

3.5.1. Exact conversion

when Rep(CCS(S)) and Rep(CCS(T)) are equal, or Rep(CCS(S)) is a subset of Rep(CCS(T)), exact conversion is possible. That means T is equal to S 🦄. The octets just need to be remapped. The algorithm for doing this remapping is simple, as long as the IANA-registered definition tables for CCS(S) and CCS(T) are available 🧩.

3.5.2. Approximate conversion

in all other cases, any conversion creates a text T that differs from S 🙃. There are different principles for how this inevitable difference should be handled, and you pick one based on the purpose and requirements of the conversion 🎯. Where possible, the client application should get mechanisms to figure out what has been done to the text 🕵️‍♀️.

3.5.2.1. Length-modifying conversion for human display

when the length of the target text T is allowed to differ from the length of the source text S, use a conversion method where each source character gets converted to one or several target character(s), picking by best resemblance 🎨.

Examples:

LATIN CAPITAL LETTER [*] ->  AE
COPYRIGHT SIGN       [*] -> (c)

3.5.2.2. Length-preserving conversion for human display

where the text T must be presented and the length of T cannot differ from the length of S, use a conversion method where each source character gets converted to one target character, using some kind of best resemblance criteria to choose it 👀.

Examples:

LATIN CAPITAL LETTER  [*] -> A
COPYRIGHT SIGN        [*] -> C

3.5.2.3. Conversion without data loss

where the conversion of the text S into T must be completely reversible, apply a Character Encoding Syntax or another reversible transformation method 🔒. This case shows up most often in data storage requirements 💅.

Examples:

LATIN CAPITAL LETTER [*] -> &AE
COPYRIGHT SIGN       [*] -> &(C

An alternate method can be used if the size of Rep(CCS(T)) >= Rep(CCS(S)). Then, for each character in Rep(CCS(S)) that isn't present in Rep(CCS(T)), define a mapping into a character in Rep(CCS(T)) that isn't present in Rep(CCS(S)) 🪄.

Examples:

LATIN CAPITAL LETTER  [*] -> CYRILLIC CAPITAL LETTER [*]
COPYRIGHT SIGN  [*] -> PARTIAL DIFFERENTIAL SIGN [*]
🫣 heads up: conversion without data loss requires redefining some member of T to mean "the introduction of character data outside T". That effectively adds another level of CES on top of CES(T).

4. Presentation issues 🌷

there are a bunch of things to think about when picking the base character set 🧐. One big one is how convenient the protocol is for users with limited equipment, like someone who only has ISO 8859-1 or a keyboard that can't enter all the characters in ISO 10646 📱. Alternative representations should be considered for these users, for both input and output 🫂. Possible options for showing characters that can't be displayed include transliteration (a la CEN/TC304 or ISO TC46/SC2), RFC 1345 [RFC-1345] representative icons, or the WG2 short name (u+xxxx) 🔤.

5. Open issues 🚧

in addition to the issues declared out of scope and listed in section 2.1, the following issues are still open and need to be addressed in other forums 🕰️. These issues are language tags, public identifiers such as URL names, and bi-directionality. They're briefly discussed below because they kept creeping into the discussion over and over 🙄💜.

5.1: Language tags 🌸

okay so the workshop didn't explicitly tackle the so-called "CJK issue," but a few members felt strongly that we needed some mechanism for correct Han character display in ISO-10646. Saying "it's just a font issue" wasn't gonna cut it, bestie 💅

The "CJK issue" is the long-running drama about "Han unification," which means using one ISO-10646 codepoint to stand in for several national variants of the same Chinese (Han) character. ISO-10646 can map uniquely to any single CJK national character set, but without extra info, an app literally can't tell which national variant to display for a given piece of text 😭

We agreed that language tags would be enough to sort out the unified characters 🎀. In our opinion there wasn't a big technical difference between using different coded character sets with overlapping codepoints and using one coded character set plus language tags. Either way, the app has enough info to display the text properly ✨

People also noticed that in current MIME charset usage, the language is kind of implied along with the coded character set and the character encoding syntax. We agreed that's way too much overloading for MIME charsets 🙈

To specify the language of a block of text, we recommend the MIME tag Content-Language. There are still a few questions to work out though:

  • Is Content-Language: actually suitable?
  • Does it overlap with the other intended functions of Content-Language: described in RFC 1766?
  • What does Content-Language: zh-tw, ja, ko, zh-cn even mean here? We think it means that, when drawing a Han character, the Taiwanese variant (probably traditional Han) is preferred, followed by the Japanese, Korean, and mainland Chinese (probably simplified Han) variants. It does NOT mean "mixed text containing Taiwanese, Japanese, Korean, and mainland Chinese text with all the national variants in each of these." 🤯

Mixed CJK text, where different variants sharing the same codepoint show up at the same time, needs language tags embedded in the data 🧚 Ohta and Handa propose in RFC 1554 [RFC-1554] a MIME charset that uses ISO-2022 shifts between multiple coded character sets. Basically, it's an encoding that uses coded character sets to display the right glyphs 💜

Some folks speculate that mixed CJK text is pretty rare, so it'd be okay to require that such text be represented in a rich text format that supports language tags. Put another way, you could make a simplifying assumption for TEXT/PLAIN in email using ISO-10646, one that doesn't need multiple display representations for the same codepoint. A mechanism like RFC 1554 could cover this need if it turned out to matter, though honestly RFC 1554 should really be labelled TEXT/ISO-2022 🦋

Also, we're saying it again: support for language tagging SHOULD be built into new protocols, because it's gonna be a critical piece of automated indexing and retrieval in the information apps of the future 🔥

5.2: Public identifiers 🫶

There's a huge demand from the user community to use non-ASCII characters in URL names, IMAP mailbox names, file names, and other public identifiers. This is still an open problem, so we're not done with it yet 👑

5.3: Bi-directionality 🌈

We realized a consistent framework for bi-directional text was needed, but nobody tried to work on it at this workshop 🙃

6: Security Considerations 💀

There are no security considerations associated with character sets. Not a single one. Iconic, honestly 💅✨

7: Conclusions 🎀

This paper gives you a conceptual framework and a set of recommendations which, if adopted, should give us a solid foundation for interoperability on the Internet 🌸 There are still a bunch of open issues that need to be addressed for even better use of text on the Internet, so this isn't the end of the story, bestie 🧚

8: Recommendations 👑

8.1: To the IAB 🍓

There were a number of recommendations to the IAB about making the standards process more aware of the need for character set interoperability, and about the framework itself 💖

A: The IAB should trigger the examination of all RFCs to determine the way they handle character sets, and obsolete or annotate the RFCs where necessary.

B: The IESG should trigger the recommendation of procedures to the RFC editor to encourage RFCs to specify character set handling if they specify the transmission of text.

C: The IAB should trigger the production of a perspectives document on the character set work that has gone on in the past and relate it to the current framework.

D: Full ISO 10646 has a sufficiently broad repertoire, and scope for further extension, that it is sufficient for use in Internet Protocols (without excluding the use of existing alternatives). There is no need for specific development of character set standards for the Internet. ✨

E: The IAB should encourage the IRTF to create a research group to explore the open issues of character sets on the Internet. This group should set its sights much higher than this workshop did. 🚀

F: The IANA (perhaps with the help of an IETF or IRTF group) should develop procedures for the registration of new character sets for use in the Internet.

G: Register UTF-8 as a Character Encoding Scheme for MIME. 🦋

H: The current use of the x-* format for distinguishing experimental tags should be continued for private use among consenting parties. All other namespaces should be allocated by IANA. 🔒

I: Application protocol RFCs SHOULD include a section on "multilingual Considerations". 🌍

J: Application Protocol RFCs SHOULD indicate how to transfer 'on the wire' all characters in the character sets they use. They SHOULD also specify how to transfer other information that applications may need to know about the data. 📡

K: The IESG should trigger a set of extensions to RFC 1522 to allow language tagging of the free text parts of message headers. 🏷️

8.2: For new Internet protocols 🧚

New protocols don't have to worry about being compatible with old 7-bit pipes, so lucky them 😌 New protocol specifications SHOULD use ISO 10646 as the base charset unless there's an overriding need to use a different base character set.

New protocols SHOULD use values from the IANA registries when referring to parameter values. How these values are carried in the protocols is protocol dependent. If the protocol uses RFC-822-like headers, the header names already in use SHOULD be used. 💜

For protocols with only a single choice for each component, the protocol should use the most general specification and should be specified with reference to the registered value in the protocol standard.

Protocols SHOULD tag text streams with the language of the text. Yes, every time 🫶

8.3: For the registration of new character sets 🌷

Ned Freed is releasing a new MIME registration document alongside this paper 💌

8.3.1: A definition table for a coded character set 📋

A definition table for a coded character set A must, for each character C that is in the repertoire of A, give:

a) if C is present in ISO 10646, the code value (in hexadecimal form) for that character.

b) if C is not present in ISO 10646, but may be constructed using ISO 10646 combining characters, the series of code values (in hexadecimal form) used to construct that character.

c) if C is not present in ISO 10646, a textual description of the character, and a reference to its origin.

8.3.2: A definition of a character encoding scheme 🔍

A definition of a character encoding scheme consists of:

  • A description of an algorithm which transforms every possible sequence of octets to either a sequence of pairs <CCS, code value> or to the error state illegal octet sequence
  • Specifications, either by reference to CCS's registered by IANA or in text, of each CCS upon which this CES is based.

Appendix A: ✨

A-1: IETF Protocols ✨

okay so this section is the tea 🫖 on how a bunch of existing protocols deal with juggling multiple character sets. bestie, it's giving "every protocol has its own drama" 💅

Email

SMTP

See 8.2. ESMTP makes it easy to negotiate the use of alternate language and encoding if it's needed. ✨

Headers

RFC 1522 forms an adequate framework for supporting text 💖 UTF-8 alone is not a possible solution, because the mail pathways are assumed to be 7-bit 'forever' 💀 However, RFC 1522 should be extended to allow language tagging of the free text parts of message headers.

Bodies

Choosing charset parameters for Email text bodies is reasonably well covered by the charset= parameter on Text/* MIME types. Language is defined by the Content-language header of RFC 1766. Other information will have to be added using body part headers. Due to the way MIME differentiates between body part headers and message headers, these will all have to have names starting with Content- 🎀

NetNews

NNTP

See 8.2. No strong tradition for negotiating encoding in NNTP exists 🙈

NetNews Messages

These should be able to leverage off the mechanisms defined for Email 🫶 One difference is that nearly all NNTP channels are 8-bit clean, and some NNTP newsgroups have a tradition of using 8-bit charsets in both headers and bodies. Defining a character set default on a per newsgroup basis might be a suitable approach 🌸

RTCP

The identifiers carried as information about parties are already defined to be in UTF-8 🦋 literally already sorted ✨

FTP

Protocol

See 8.2. The common use of welcome banners in the login response means there might be strong reason to let client and server negotiate a language that's different from the default for greetings and error messages. This should be a simple protocol extension 🧚

Filenames

Many fileservers now have the capability of using non-ASCII characters in filenames, but the "dir" and "get" commands of are defined in terms of US-ASCII only 😭 One possible solution would be to define a "UTF-8" mode for the transfer of filenames and directory information. This would need to be a negotiated facility, with fallback to US-ASCII if it's not negotiated. The important point here is consistency between all implementations 🔥 A single charset is better here than the ability to handle multiple charsets, no cap.

World Wide Web

HTTP

See 8.2. The single-shot style of HTTP makes negotiation more complex than it would otherwise be 💅

HTML

Internationalization of HTML [I18N] seems fairly well covered in the current "I18N" document 🌈 It needs review to see if it needs more specific details in order to carry application information apart from the language.

URLs

URLs are "input identifiers", and powerful arguments should be made if they are ever to be anything but US-ASCII 🍓

IMAP

IMAP's information objects are MIME Email objects, so they're able to use that standard's methods 💜 However, IMAP folder names are local identifiers, and there is strong reason to allow non-ASCII characters in these. A UTF-8 negotiation might be the most appropriate thing, however, UTF-8 is awkward to use 🙈 Unfortunately, UTF-7 isn't suitable because it conflicts with popular hierarchy delimiters. The most recent IMAP work in progress specification describes a modified UTF-7 which avoids this problem ✨

DNS

DNS names are the prime example of identifiers that need to stay in US-ASCII for global interoperability 🌸 However, some DNS information, in particular TXT records, may represent information (such as names) that is outside the ASCII range. A single solution is the best, and problems resulting from UTF-8 should be investigated 🔍

WHOIS++

WHOIS++ version 1 is defined to use ISO 8859-1. The next version will use UTF-8 🦋 The currently designed changes will also allow the specification of individual attributes on attribute names. These will make passing application information about the values (such as language) easier. No immediate action seems necessary 💖

WHOIS

This has been a stable protocol for so many years now that it seems unwise to suggest modifying it 👑 Furthermore, compatible extensions exist in RWHOIS and WHOIS++, so modification should rather be made to these protocols than to the WHOIS protocol itself 🫶

Telnet

This is a prime example of a protocol where character set support is necessary and nonexistent 😭 The current work in progress on character set negotiation in Telnet seems adequate to the task. The question of passing other application data that might be useful is still open 💀

A-2: Non-IETF protocols

For these protocols, the IETF does not have any power to change them 🥲 However, the guidelines developed by the workshop may still be useful as input to the further development of the protocols ✨

Gopher: Gopher, Gopher+ 🌿

Prospero (Archie) 🧚

NFS: Filesystem 🗂️

CORBA, Finger, GEDI, IRC, ISO 10160/1, Kerberos, LPR, RSTAT, RWhois, SGML, TFTP, X11, X.500, Z39.50 💅

Appendix B: Acronyms 🎀

Here's the whole acronym lineup, kept exactly as it is 🫶

ASCII       American National Standard Code for Information Character
              Sets
CCS         Coded Character Sets
CEN ENV     European Committee for Standardisation (CEN) European
              pre-standard (ENV)
CES         Character Encoding Scheme
CJK         Chinese Japanese Korean
CORBA       Common Object Request Broker Architecture
CTE         Content Transfer Encoding
DNS         Domain Name Service
ESMTP       Extended SMTP
FTP         File Transfer Protocol
HTML        Hypertext Transfer Protocol
I18N        Internationalization (or 18 characters between the first
              (I) and last (n)character)
IAB         Internet Activities Board
IANA        Internet Assigned Numbers Authority
IESG        Internet Engineering Steering Group
IETF        Internet Engineering Task Force
IMAP        Internet Message Access Protocol
IRC         Internet Relay Chat
IRTF        Internet Research Task Force
ISI         Information Sciences Institute
ISO         International Standards Organization
MIME        Multipurpose Internet Mail Extensions
NFS         Networked File Server
NNTP        Net News Transfer Protocol
POSIX       Portable Operating System Interface
RFC         Request for Comments (Internet standards documents)
RPC         Remote Procedure Call
RSTAT       Remote Statistics
RTCP        Real-Time Transport Control Protocol
Rwhois      Referral Whois
SGML        Standard Generalized Mark-up Language
SMTP        Simple Mail Transfer Protocol
TES         Transfer Encoding Syntax
TFTP        Trivial File Transfer Protocol
URL         Uniform Resource Locator
UTF         Universal Text/Translation Format

Appendix C: Glossary 📖

Time for the vocab check 🧠 These are the definitions, in my own words, so you know exactly what everyone means.

Bi-directionality 🔁 A property of some text where text written right-to-left (Arabic or Hebrew) and text written left-to-right (e.g. Latin) are intermixed in one and the same line.

Character ✨ A single graphic symbol represented by a sequence of one or more bytes.

Character Encoding Scheme 🎀 The mapping from a coded character set to an encoding which may be more suitable for a specific purpose. For example, UTF-8 is a character encoding scheme for ISO 10646.

Character Set 🌸 An enumerated group of symbols (e.g., letters, numbers or glyphs).

Coded Character Set 🔢 The mapping from a set of integers to the characters of a character set.

Culture 🌍 Preferences in the display of text based on cultural norms, such as spelling and word choice.

Language 🗣️ The words and combinations of words that constitute a system of expression and communication among people with a shared history or set of traditions.

Layout 📐 Information needed to display text to the user, similar to the presentation layer in the ISO telecommunications model.

Locale 💜 The attributes of communication, such as language, character set and cultural conventions.

On-the-wire 📦 The data that actually gets put into packets for transmission to other computers.

Transfer Encoding Syntax 🚚 The mapping from a coded character set which has been encoded in a Character Encoding Scheme to an encoding which may be more suitable for transmission using specific protocols. For example, Base64 is a transfer encoding syntax.

Appendix D: References 📚

Here's the full reference list, exactly as written, because citations are sacred 🫶

[*]  Non-ASCII character

[ASCII]  ANSI X3.4:1986  "Coded  Character Sets - 7 Bit American
     National Standard Code for Information Interchange (7-bit ASCII)"

[Base64]  Freed, N., and N. Borenstein, "Multipurpose Internet
     Mail Extensions (MIME) Part One: Format of Internet Message
     Bodies", RFC 2045, November 1996.

[CEN]  see http://tobbi.iti.is/TC304/welcome.html for current status.

[HTML]  Berners-Lee, T., and D. Connolly, "Hypertext Markup Language -
     2.0", RFC 1866, November 1995.

[HTTP]  Berners-Lee, T., Fielding, R., and H. Nielsen, "Hypertext
     Transfer Protocol -- HTTP/1.0", RFC 1945, May 1996.

[I18N]  Yergeau, F., et.al.,  "Internationalization of the Hypertext
     Markup Language", RFC 2070, January 1997.

[IANA] Reynolds, J., and J. Postel, "Assigned Numbers", STD 2, RFC
     1700, ISI, October 1994.

[ISO-2022]  ISO/IEC 2022:1994,  "Information technology -- Character
     Code Structure and Extension Techniques",  JTC1/SC2.

[ISO-7498]  ISO/IEC 7498-1:1994,  "Information technology - Open Systems
     Interconnection - Basic Reference Model:  The Basic Model".

[ISO-8859]  Information Processing -- 8-bit Single-Byte Coded Graphic
     Character Sets -- Part 1: Latin Alphabet no. 1,
     ISO 8859-1:1987(E). Part 2: Latin Alphabet no. 2, ISO 8859-2
     1987(E). Part 3: Latin Alphabet no. 3, ISO 8859-3:1988(E).
     Part 4: Latin Alphabet no. 4, ISO 8859-4, 1988(E). Part 5:
     Latin/Cyrillic Alphabet ISO 8859-5, 1988(E). Part 6:
     Latin/Arabic Alphabet, ISO 8859-6, 1987(E). Part 7: Latin/Greek
     Alphabet, ISO 8859-7, 1987(E). Part 8: Latin/Hebrew Alphabet, ISO
     8859-8-1988(E).Part 9: Latin Alphabet no. 5, ISO 8859-9, 1990(E).
     Part 10: Latin Alphabet no. 6, ISO 8859-10:1992(E).

[ISO-10646]  ISO/IEC 10646-1:1993(E ),  "Information technology --
     Universal Multiple-Octet Coded Character Set (UCS) -- Part 1:
     Architecture and Basic Multilingual Plane".  JTC1/SC2, 1993

[MIME]  See [Base64]

[POSIX]  Institute of Electrical and Electronics Engineers.  "IEEE
     standard interpretations for IEEE standard portable operating
     systems interface for computer environments". IEEE Std 1003.1
     -1988/Int, 1992 edition.  Sponsor, Technical Committee on Operating
     Systems of the IEEE Computer Society.  New York, NY: Institute of
     Electrical and Electronic Engineers, 1992.

RFC 1340  See [IANA]

[RFC-1345]  Simonsen, K., "Character Mnemonics & Character Sets",
     RFC 1345, Rationel Alim Planlaegning, June 1992.

[RFC-1554]  Ohta, M., and K. Handa,  "ISO-2022-JP-2: Multilingual
     Extension of ISO-2022-JP",  Tokyo Institute of Technology, ETL,
     December 1993.

RFC 1642  See [UTF-7]

[RFC-1766]  Alvestrad, H., "Tags for the Identification of Languages",
     RFC 1766, UNINETT, March 1995.

[RFC 1958]  Carpenter, B. (ed.) "Architectural Principles of the
     Internet", RFC 1958, IAB, June 1996.
  • [SGML] ISO 8879:1986 "Information Processing - Text and Office Systems - Standard Generalized Markup Language (SGML)" 📜
  • [SMTP] Postel, J., "Simple Mail Transfer Protocol", STD 10, RFC 821, August, 1982. 📬
  • [Unicode] "The Unicode standard, version 2.0. Unicode Consortium. Reading, Mass.: Addison-Wesley Developers Press, 1996" 🌍
  • [UTF-7] Goldsmith, D., and M. Davis, "UTF-7: A Mail Safe Transformation Format of Unicode", RFC 1642, Taligent, Inc., July 1994. 💌
  • [UTF-8] International Standards Organization, Joint Technical Committee 1 (ISO/JTC1), "Amendment 2:1993, UCS Transformation Format 8 (UTF-8)", in ISO/IEC 10646-1:1993 Information technology - Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane. JTC1/SC2, 1993. 🔤

okay bestie, this is your reading list 🫶 if you want to go deeper, these are the vibes:

  • Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, UNINETT, March 1995.
  • Alvestrand, H., "X.400 Use of Extended Character Sets", RFC 1502, SINTEF DELAB, August 1993.
  • Borenstein, N., "Implications of MIME for Internet Mail Gateways", RFC 1344, Bellcore, June 1992.
  • Freed, N., and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.
  • Chernov, A., "Registration of a Cyrillic Character Set", RFC 1489, RELCOM Development Team, July 1993.
  • Choi, U., and K. Chan, "Korean Character Encoding for Internet Messages", RFC 1557, KAIST, December 1993.
  • Freed, N., and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, November 1996.
  • Goldsmith, D., and M. Davis, "Transformation Format for Unicode", RFC 1642, Taligent, Inc., July 1994.
  • Goldsmith, D., and M. Davis, "Using Unicode with MIME", RFC 1641, Taligent, Inc., July 1994.
  • Jerman-Blazic, B. "Character handling in computer communication" in "user needs in information technology standards", Computer Weekly Professional service, eds. C.D. Evans, B.L. Meed & R.S. Walker, P.C. Butterworth Heineman, 1993, Oxford, Boston, p. 102-129.
  • Jerman-Blazic, B. "Tool supporting the internationalization of the generic network services", Computer Networks and ISDN Systems, No. 27 (1994), p. 429-435.
  • Jerman-Blazic, B., A. Gogala and D. Gabrijelcic, "Transparent language processing: A solution for internationalization of Internet services", The LISA Forum Newsletter, 5 (1996) p. 12-21
  • Lee, F., "HZ - A Data Format for Exchanging Files of Arbitrarily Mixed Chinese and ASCII Characters", RFC 1843, Stanford University, August 1995.
  • McCarthy, J., "Arbitrary Character Sets", RFC 373, Stanford University, July 1972.
  • Moore, K., "MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text", RFC 1522, September 1993. (Obsoleted by RFC 2047.)
  • Moore, K., "MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text", RFC 2047, University of Tennessee, November 1996.
  • Murai, J., Crispin, M., and E. von der Poel. "Japanese Character Encoding for Internet Messages", RFC 1468, Keio University & Panda Programming, June 1993.
  • Nussbacher, H., "Handling of Bi-directional Texts in MIME", Israeli Inter-University, December 1993.
  • Nussbacher, H., and Y. Bourvine, "Hebrew Character Encoding for Internet Messages", RFC 1555, Israeli Inter-University and Hebrew University, December 1993.
  • Ohta, M., "Character Sets ISO-10646 and ISO-10646-J-1", RFC 1815, Tokyo Institute of Technology, July 1995.
  • Postel, J., and J. Reynolds, "File Transfer Protocol (FTP)", STD 9, RFC 959, ISI, October 1985.
  • Postel, J., and J. Reynolds, "Telnet Protocol Specification", STD 8, RFC 854, ISI, May 1983.
  • Reynolds, J., and J. Postel, "Assigned Numbers", STD 2, RFC 1700, ISI, October 1994. p.100-117.
  • Rose, M., "The Internet Message", Prentice Hall, 1992.
  • Simonsen, K., "Character Mnemonics & Character Sets", RFC 1345, Rationel Almen Planlaegning, June 1992.
  • Unicode Consortium. "The Unicode standard, version 2.0. Reading, Mass.: Addison-Wesley Developers Press, 1996
  • Wei, U., et.al. "ASCII Printable Characters-Based Chinese Character Encoding for Internet Messages", RFC 1842, AsiInfo Services, Inc., et.al. August 1995.
  • Yergeau, F. "UTF-8, a transformation format of Unicode and ISO 10646", RFC 2044, ALIS Technologies, October 1996.
  • Zhu, H., et.al., "Chinese Character Encoding for Internet Messages", RFC 1922, Tsinghua University, et.al., March 1996.

Appendix F: Workshop attendee list 🎀

these people were participants on the workshop mailing list 💜 a * means they came to the workshop in person ✨

  Glenn Adams <glenn@spyglass.com>
* Joan Aliprand <joan@unicode.org>
* Harald Alvestrand <Harald.T.Alvestrand@uninett.no>
* Ran Atkinson <ran@cisco.com>
* Bert Bos <bert@w3.org>
* Brian Carpenter <brian@dxcoms.cern.ch>
* Mark Crispin <mrc@panda.com>
  Makx Dekkers <dekkers@pica.nl>
  Robert Elz <kre@munnari.oz.au>
  Patrik Faltstrom <paf@paf.se>
* Zhu Haifeng <zhf@net.tsinghua.edu.cn>
  Keniichi Handa<handa@etl.go.jp>
  Olle Jarnefors <ojarnef@admin.kth.se>
  Borka Jerman-Blazic <borka@e5.ijs.si>
  John Klensin <klensin@mail1.reston.mci.net>
* Larry Masinter <masinter@parc.xerox.com>
* Rick McGowan <Rick_McGowan@next.com>
* Keith Moore <moore+charsets@cs.utk.edu>
* Lisa Moore <lisam@vnet.ibm.com>
  Ruth Moulton <ruth@muswell.demon.co.uk>
* Cecilia Preston <cecilia@well.com>
* Joyce K. Reynolds <jkrey@isi.edu>
* Keld Simonsen <keld@dkuug.dk>
* Gary Smith <Gary_Smith@oclc.org>
* Peter Svanberg <psv@nada.kth.se>
* Chris Weider <cweider@microsoft.com >

Appendix G: Authors' Addresses 🦋

the crew behind this doc, where to find them 💅

Chris Weider
Microsoft Corp.
1 Microsoft Way
Redmond, WA 98052
USA

EMail: cweider@microsoft.com


Cecilia Preston
Preston & Lynch
PO Box 8310
Emeryville, CA 94662
USA

EMail: cecilia@well.com


Keld Simonsen
DKUUG
Freubjergvey 3
DK-2100 Kxbenhavn X
Danmark

EMail: Keld@dkuug.dk


Harald T. Alvestrand
UNINETT
P.O.Box 6883 Elgeseter
N-7002 TRONDHEIM
NORWAY

EMail: Harald.T.Alvestrand@uninett.no


Randall Atkinson
cisco Systems
170 West Tasman Drive
San Jose, CA 95134-1706
USA

EMail: rja@cisco.com

Mark Crispin
Networks & Distributed Computing
University of Washington
4545 15th Avenue NE
Seattle, WA  98105-4527
USA

EMail: mrc@cac.washington.edu


Peter Svanberg
Dept. of Numberical Analysis and Computing Science (Nada)
Royal Institute of Technology
SE-100 44 STOCKHOLM
SWEDEN

EMail: psv@nada.kth.se
📦 where this page comes from