RFC 2277

IETF Policy on Character Sets and Languages 🎀

okay bestie, so this is RFC 2277 💖 by H. Alvestrand from UNINETT, dated January 1998. It's a Best Current Practice (BCP 18), and its header doesn't list any Obsoletes or Updates, so it isn't replacing or changing anything else. 🦋

Status of This Memo ✨

This doc specifies an Internet Best Current Practice for the Internet Community, and it asks for discussion and suggestions for improvements. Distribution of this memo is unlimited.

Copyright Notice 📜

Copyright (C) The Internet Society (1998). All Rights Reserved.

1. Introduction ✨

The Internet is international. 🌍

With the international Internet comes a total must-have: swapping data in a whole bunch of languages, which use a bewildering number of characters. 😭

This document lays out the policies the Internet Engineering Steering Group (IESG) is applying to standardization work in the Internet Engineering Task Force (IETF), so Internet protocols can actually meet those needs. 💅

It's very much based on the recommendations from the IAB Character Set Workshop of February 29-March 1, 1996, which is written up in RFC 2130 [WR]. This doc tries to be concise, explicit and clear, so if you want more background, go read RFC 2130 🧚

The document uses the terms 'MUST', 'SHOULD' and 'MAY', and their negatives, the way [RFC 2119] describes. Here, 'the specification' (as RFC 2119 uses it) means the processing of protocols being submitted to the IETF standards process. 📝

2. Where to do internationalization 🌸

Internationalization is for humans. So protocols themselves are not internationalized, but text strings are. 💖 When protocol elements look like text tokens (like in lots of IETF application layer protocols), protocols MUST specify which parts are protocol and which parts are text. [WR 2.2.1.1]

Names are a whole thing, because people feel strongly about them, lots of them are mostly for local use, and they tend to leak out of their local context sometimes. 🙈 RFC 1958 [RFC 1958] recommends US-ASCII for all globally visible names.

This document doesn't mandate a policy on name internationalization. It does require that all protocols describe whether names are internationalized or US-ASCII. 🫶

NOTE: In the protocol stack for any given application, there is usually one or a few layers that need to address these problems. 👑

It would, for instance, not make sense to define language tags for Ethernet frames 💀 But it's the responsibility of the working groups (WGs) to make sure that whenever internationalization is left to "another layer", the people responsible for that layer actually know they HAVE that responsibility. (Yes, that's the emphasis from the original, and it's iconic 😤)

3. Definition of Terms 📖

This document uses the term "charset" to mean a set of rules for mapping from a sequence of octets to a sequence of characters. It's basically a coded character set combined with a character encoding scheme. This is also what shows up as an identifier in MIME "charset=" parameters, and it's registered in the IANA charset registry [REG]. (Note that this is NOT a term used by other standards bodies, like ISO.) 🧁

For a definition of "coded character set", check the workshop report. 📚

A "name" is an identifier such as a person's name, a hostname, a domainname, a filename or an E-mail address. It's often treated as an identifier rather than as a piece of text, and protocols often use it to identify entities without any surrounding text. 🦋

3.1. What charset to use 🎀

All protocols MUST identify, for all character data, which charset is in use. ✨

Protocols MUST be able to use the UTF-8 charset. That's the ISO 10646 coded character set combined with the UTF-8 character encoding scheme, as defined in [10646] Annex R (published in Amendment 2), and it's for all text. 💖

Protocols MAY specify, in addition, how to use other charsets or other character encoding schemes for ISO 10646, such as UTF-16. But lacking the ability to use UTF-8 is a violation of this policy. A violation like that would need a variance procedure ([BCP9] section 9) with clear and solid justification in the protocol specification document before it could be entered into or advanced upon the standards track. 🚨

For existing protocols, or protocols that move data out of existing datastores, supporting other charsets (or even using a default other than UTF-8) may be a requirement. That's acceptable, but UTF-8 support MUST be possible. 🫶

When using charsets other than UTF-8, those charsets MUST be registered in the IANA charset registry. If necessary, they can be registered when the protocol is published. 🌈

(Note: ISO 10646 calls the UTF-8 CES a "Transformation Format" rather than a "character encoding scheme", but it fits the charset workshop report definition of a character encoding scheme.) 🤓

3.2. How to decide a charset 🧠

When a protocol allows a choice of multiple charsets, someone has to decide which charset to use. 👀

In some cases, like HTTP, there's direct or semi-direct communication between the producer and consumer of text-containing data. In those cases, it may make sense to negotiate a charset before sending data. 🤝

In other cases, like E-mail or stored data, there's no such communication. The best you can do is make sure the charset is clearly identified with the stored data, and pick a charset that's as widely known as possible. 💌

A charset is an absolute, bestie. Text encoded in a charset cannot be rendered comprehensibly without support for that charset. ⚠️

(This also applies to English texts. Charsets like EBCDIC do NOT have ASCII as a proper subset.) 💀

Negotiating a charset may be regarded as an interim mechanism, to be supported until support for interchange of UTF-8 is prevalent. But the timeframe of "interim" may be at least 50 years, so there's every reason to think of it as permanent in practice. 🕰️✨

4. Languages 🌈

4.1. The need for language information 💜

All human-readable text has a language. 📝

Lots of operations (high quality formatting, text-to-speech synthesis, searching, hyphenation, spellchecking and so on) benefit hugely from info about the language of a piece of text. [WC 3.1.1.4] 🔥

Humans have some tolerance for foreign languages, but they're generally very unhappy being shown text in a language they don't understand. That's why language negotiation is needed. 😭

In most cases, machines can't deduce the language of a transmitted text by themselves. So the protocol has to specify how to transfer the language info, if it's going to be available at all. 🤖

Language and processing are complicated together. For example, if I compare name-of-thing(lang=en) to name-of-thing(lang=no) for equality, I'd generally expect a match. Meanwhile, the word ask(no) is a kind of tree, and it's hardly useful as a command verb. 🌳💀

4.2. Requirement for language tagging 🏷️

Protocols that transfer text MUST provide for carrying information about the language of that text. ✨

Protocols SHOULD also provide for carrying information about the language of names, where that's appropriate. 💖

Note that this does NOT mean such information must always be present. The requirement is that if the sender wants to send info about the language of a text, the protocol provides a well-defined way to carry it. 🫶

4.3. How to identify a language 🗣️

The RFC 1766 language tag is, at the moment, the most flexible tool available for identifying a language. Protocols SHOULD use it, or provide clear and solid justification in the document for doing otherwise. 👑

Also, a language is distinct from a POSIX locale. A POSIX locale identifies a set of cultural conventions, which may imply a language (the POSIX or "C" locale of course do not). An RFC 1766 language tag identifies only a language. 🧁

4.4. Considerations for language negotiation 🤝

Protocols where users get text presented to them in response to their actions MUST provide for support of multiple languages. 🌸

How this gets done varies between protocols. For instance, sometimes a negotiation where the client proposes a set of languages and the server replies with one is appropriate. Other times, a server might send multiple variants of a text and let the client pick which one to display. 🦋

Negotiation is useful when one side of the exchange can present text in multiple languages, and the other side has a preference for one of them. The most common example is the text part of error responses, or Web pages that are available in multiple languages. 🌐

Negotiating a language should be regarded as a permanent requirement of the protocol, one that won't go away at any point in the future. 💅

In many cases, it should be possible to include it as part of connection establishment, together with authentication and other preference negotiation. 🔐

4.5. Default Language 🆘

When human-readable text has to be presented in a context where the sender doesn't know the recipient's language preferences (like login failures, E-mailed warnings, or before any language negotiation), the text SHOULD be presented in Default Language. 🙏

Default Language is assigned the tag i-default according to the procedures of RFC 1766. It isn't a specific language. It identifies the condition where the user's language preferences can't be established. 🤷‍♀️

Messages in Default Language MUST be understandable by an English-speaking person. English is the language that the greatest number of people worldwide can get adequate help in when working with computers, so that's the reasoning. 🌍💜

Note that negotiating English is NOT the same as Default Language. Default Language is an emergency measure for otherwise unmanageable situations. 🚨

In many cases, using only English text is reasonable. In some cases, the English text may be augmented by text in other languages. ✨

5. Locale 🍓

The POSIX standard [POSIX] defines a concept called a "locale", which includes a lot of information about collating order for sorting, date format, currency format and so on. 📅

In some cases (especially with text the user is expected to process), locale information can be usefully attached to the text. This identifies the sender's opinion about the appropriate rules to follow when processing the document, and the recipient may choose to agree with it or ignore it. 🧠

This document doesn't require communicating locale information on all text, but it encourages including it when appropriate. 🫶

Language and character set information will often be present as parts of a locale tag, like no_NO.iso-8859-1, where the language comes before the underscore and the character set comes after the dot. So care must be taken to define precisely which specification of character set and language applies to any one text item. 👀

The default locale is the "POSIX" locale. 💖

---

part 2 is coming next, so stay tuned 🧚🌈

6. Documenting internationalization decisions 🌍

okay so bestie, if a document deals with internationalization issues at all, the approaches you picked SHOULD get collected into a section called "Internationalization considerations" 💅 and placed right next to the Security Considerations section 🫶

this is giving easy reference energy 🎀 anyone who's implementing the protocol and wants advice on these issues can just flip straight to it 🦋

7. Security Considerations 🔒

ngl this one is short but it's serious 😭 security warnings in a foreign language might make the user do inappropriate stuff 💀 and multilingual systems usually have problems with consistency between language variants 🙈 other than that, no relevant security considerations have been identified 💖

8. References 📚

• [10646] ISO/IEC, Information Technology - Universal Multiple-Octet Coded Character Set (UCS) - Part 1: Architecture and Basic Multilingual Plane, May 1993, with amendments
• [RFC 2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
• [WR] Weider, C., Preston, C., Simonsen, K., Alvestrand, H, Atkinson, R., Crispin, M., and P. Svanberg, "The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996", RFC 2130, April 1997.
• [RFC 1958] Carpenter, B., "Architectural Principles of the Internet", RFC 1958, June 1996.
• [POSIX] ISO/IEC 9945-2:1993 Information technology -- Portable Operating System Interface (POSIX) -- Part 2: Shell and Utilities
• [REG] Freed, N., and J. Postel, "IANA Charset Registration Procedures", BCP 19, RFC 2278, January 1998.
• [UTF-8] Yergeau, F., "UTF-8, a transformation format of ISO 10646", RFC 2279, January 1998.
• [BCP9] Bradner, S., "The Internet Standards Process -- Revision 3," BCP 9, RFC 2026, October 1996.

9. Author's Address 🧚

Harald Tveit Alvestrand
UNINETT
P.O.Box 6883 Elgeseter
N-7002 TRONDHEIM
NORWAY

Phone: +47 73 59 70 94
EMail: Harald.T.Alvestrand@uninett.no

10. Full Copyright Statement 💜

this part is the legal bit, so I'm keeping it word for word 🫶 no sneaky edits here

Copyright (C) The Internet Society (1998). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.