RFC 1502

X.400 Use of Extended Character Sets

📅 published August 1993, from ietf

RFC 1502: X.400 Use of Extended Character Sets 💖

(part 1 of 3)

  • Working group: Network Working Group 🌸
  • Author: H. Alvestrand, SINTEF DELAB ✨
  • Date: August 1993 📅
  • Category: IAB standards track protocol (more on that in the Status section below 👀)
  • Obsoletes / Updates: nothing is listed in its title block 🙈

Status of this Memo ✨

Okay so this RFC is an IAB standards track protocol for the whole Internet community, and it's literally asking for discussion and suggestions 💬. For the current standardization state and status of this protocol, check the current edition of the "IAB Official Protocol Standards" 📚. And distribution of this memo is unlimited 🌈.

1. Introduction 🦋

Bestie, since 1988, X.400 has been able to carry a bunch of different character sets in one message, using the body part "GeneralText" defined by ISO/IEC 10021-7 💅.

Since 1992, the Internet has been able to do the same thing with messages containing multiple character sets, through the mechanism defined in RFC-MIME 💌.

So this RFC suggests a way to use "GeneralText" so everyone uses this body part in a more harmonized way 🫶.

2. General principles 🌸

2.1. Goals 👑

The target of this memo is to define a way of using existing standards to achieve:

  1. in the short term: a standard for sending E-mail in the European languages (Latin letters with European accents, Greek and Cyrillic) 💌
  2. in the medium term: extending this to cover the Hebrew and Arabic character sets 🌍
  3. in the long term: opening up true international E-mail by allowing the full character set specified in ISO-10646 to be used 🌏

The author thinks this document is a spec that can easily handle any character set in the ISO registry. By giving guidance on choosing character sets, it'll help different systems interwork 🔥.

2.2. Families of character sets 🎀

2.2.1. ISO 6937/T.61 💅

ISO 6937 is a coding technique that's used and recommended in T.51 and T.101 (Teletex and Videotex service) and in X.500 🌈. It gives you a repertoire of 333 Latin-script characters by using non-spacing diacritical marks, and it lines up closely with CCITT recommendation T.61.

The catch is that the character stream has two modes. Some characters are one byte and some are two (composite characters) 😭. That makes systems like an E-mail UA or GW way more complicated.

It also doesn't stretch to other languages like Korean or Chinese, or even Greek, unless you bring in the character set switching techniques of ISO 2022 🙃.

2.2.2. ISO 8859 🧚

ISO 8859 is a family of character sets, each one fitted to a group of languages 🍓. Every character in ISO 8859 is coded in a single byte.

There are 11 parts of ISO 8859 right now, plus a "supplementary" set registered as ISO IR 154 📚. Most single-byte languages fit into one of these sets. Yes, there are sets for Greek, Hebrew and Arabic, but the rendering direction for Hebrew and Arabic is still a whole debate 🙈.

Every ISO 8859 set includes US-ASCII as a subset, and they all use 8 bits 💖.

A lot of people think ISO 8859 is the answer 💁. X windows, for example, ships with ISO-8859-1 as the "standard" character set, and you can specify others. But the same apps often can't switch character sets within text, so it's a mess for a truly multilingual setup 😩. (Also, most fonts claiming to be "ISO-8859-1" in X11R5 are actually 7-bit fonts. The implied lie is very unfortunate 💀.) It does work fine if your second language is English, though, since English can be written in every ISO 8859 set 🌸.

Parts 3 and 4 never really caught on, and they're expected to get discarded, so they should therefore not be used 🚫.

Fun fact: an ISO 8859 set is actually 2 sets in the ISO sense. US-ASCII is in the G0 set and another character set is in the G1 set 🤓. Calling both of them "character set" is confusing, but that's just tradition 🙃.

2.2.3. ISO 10646 🌈

Right now ISO 10646 has just been accepted as an International Standard ✨. It's basically a 32-bit character set. All of the characters in current use are numbered within the first 16 bits, which leaves room to grow 🌱.

You can't use ISO 10646 as a normal character set 🙅, because it doesn't follow the byte-value usage rules in ISO 2022 and other places. It uses the "control space" for (parts of) graphic character codes 😭.

There are a bunch of ways to encode ISO 10646 characters "on the wire" 🦋. ISO 2022 has methods to switch to them, either as "other coding system without return" or as "other coding system with return". The second one means you can hop back to where you came from with an ISO 2022 escape sequence 🔁.

These registrations are already done 💅:

ISO 10646 UCS-2 Level 1 has been registered with ESC 2/5 2/15 4/0,
ISO 10646 UCS-4 Level 1 has been registered with ESC 2/5 2/15 4/1,

These ones have been applied for 📝:

Reg# Escape sequence  Standard/Sponsor   Description
174  ESC 2/5 2/15 F   ISO/IEC 10646      UCS-2, Level 2
175  ESC 2/5 2/15 F   ISO/IEC 10646      UCS-4, Level 2
176  ESC 2/5 2/15 F   ISO/IEC 10646      UCS-2, Level 3
177  ESC 2/5 2/15 F   ISO/IEC 10646      UCS-4, Level 3
178  ESC 2/5 F        ISO/IEC 10646      UTF-1
<< NOTE: The registration numbers for UCS-2 level 1 and UCS-4 level 1 are not known. Neither are the assigned final characters for the other sets. Information requested!>>

This character set is going to be super important later on 🔮, but right now few systems can support it directly 😔.

And the GeneralText body part can carry any of these character sets 💌.

2.3. Body parts that can be used in X.400 📬

Right now there's no established way to move a full character set around in X.400-based E-mail 😤. Later on, a new body part based on ISO 10646 will probably show up, or GeneralText might be able to use ISO 10646. That hasn't been sorted out yet 🤷.

For now, the body parts that are deployed and available are 🧾:

  1. IA5Text
  2. For X.400/84: ISO6937Text and Teletex
  3. For X.400/88: GeneralText

IA5Text is the go-to for E-mail that only has IA5 characters (basically US-ASCII) 💖.

ISO6937Text is defined in the ISO DIS documents for X.400(84) [10] 📄. Those never became a standard, so they're now pretty hard to find 🔍. In principle it's limited to text that can be presented in ISO 6937. But ISO 6937 points to the ISO 2022 method for switching character sets, so in theory you could use any ISO-registered character set. The catch is there's no way to announce which character sets you used 😬, which makes interworking with equipment that doesn't support the same sets really complicated.

Still though, it's the only body part suitable for carrying non-paginated text in non-basic character sets in X.400(84) 🌟.

Teletex is identical in every version of X.400, and it has the same implicit ISO6937 problem 😵. Plus it specifies a page format, like a left margin of 5 character positions. That's often not what you want 🙄.

All the Teletex details live in recommendation T.51 and its relatives 📚.

GeneralText is defined in ISO 10021-8, which is the part of [9] that matches CCITT recommendation [11] 🎀. It's an Extended body part, so CCITT implementations don't need any modification to carry it ✨.

GeneralText is great for interchange because it has proper announcement facilities 📣. It can use any number of character sets, and it announces them in two places: the Encoded Information Types of the X.400 envelope, and the parameters of the body part 🗂️.

So we recommend this body part for carrying unformatted text in X.400/88 💅.

3. GUIDELINES FOR THE GENERATION OF GENERALTEXT ✨

3.1. Formal definition of GeneralText 🌸

A GeneralText message is basically a byte stream holding characters and character switching sequences, following [12] 🧵.

The X.400 ASN.1 definition of the GeneralText body part is:

general-text-body-part EXTENDED-BODY-PART-TYPE
    PARAMETERS GeneralTextParameters IDENTIFIED BY id-ep-general-text
    DATA       GeneralTextData
    ::= id-et-general-text

GeneralTextParameters ::= SET OF CharacterSetRegistration

CharacterSetRegistration ::= INTEGER (1..32767)

GeneralTextData ::= GeneralString

This definition comes from ISO/IEC 10021-7 [9], Annex I 📖, with changes made in the MHS Implementor' Guide, version 8, chapter 3.6.3, bullet F130. Fun twist: it doesn't show up in the CCITT version of the standards 🙈.

3.2. Brief description of ISO 2022 character set switching 🦋

At any moment there are 4 graphic character sets active in a GeneralText message: G0, G1, G2 and G3 💅. There are also 2 control character sets, called C0 and C1 🎀.

At any moment, one of G0-G3 is active in code positions 2/1 to 7/14, and another one is active in code positions 10/0 to 15/15. You set these up with so-called "locking shift" sequences 🔐.

(Formally, code positions 2/0 and 7/15 are reserved for "space" and "DEL" respectively, and only 94-character character sets can be used in G0. In practice though, people sometimes ignore this restriction 🙃)

You can also grab a single character from a non-active set using "single shift" sequences ⚡.

The control character sets always sit in code positions 0/0 to 1/15 (C0) and 8/0 to 9/15 (C1) 🧭.

The character sets active as G0-G3 and C0-C1 can be changed with "character set designating sequences" 🔄.

At the very start of a GeneralText message, you must always assume that set 2 (IA5) is active as G0, shifted into the lower half, that set 1 (standard) is active as C0, and that no G1-G3 or C1 set is invoked 🌟. This is specified in the definition of "GeneralString" in [5], the definition of ASN.1 encoding (section 23.5.2).

If that's not a suitable starting state, a message must always start with the necessary announcers and escape sequences to designate and invoke the character sets actually used 📣. The character sets in use can be changed later in the message with escape sequences 🔁.

The parameters of a GeneralText message always list every character set used, by quoting their ISO reference numbers 🏷️.

You simply can't use a character set that isn't registered with ISO in a GeneralText message 🚫.

It's also impossible to figure out what a byte really means in a GeneralText message unless you scan the whole thing for shift and escape sequences 🕵️.

more to come in part 2 💜

3.3. How to use the character sets ✨

okay bestie, this is the part where the RFC tells you exactly what to do 💅

🎀 RECOMMENDATION:

When the text you're rendering can be represented in one of the ISO-8859 character sets, the G0 set should be set to ISO 646 International Reference Version (1991), also called US-ASCII, ISO-IR-6. Keep it simple, babe. 🌸

The older character set ISO-IR-2, ISO 646 IRV(1983), should NOT be used. This means the escape sequence ESC 2/8 4/2 (designating US-ASCII as G0) should always occur at the beginning of the message. 🦋

The G1 set should be set to the character set identified by the relevant ISO-8859 part. G2 and G3 are not used. ✨

This corresponds to the first level of ISO 4873 usage. 💖

For the currently defined parts of ISO 8859, the character set designations for the G1 set are (relative to ISO 8859:1987):

Part    ISO IR name             Escape sequence Remarks
                                for G1 use

1       ISO-IR-100              Esc 2D 41    West Europe (Latin-1)
2       ISO-IR-101              Esc 2D 42    East European (Latin-2)
3       ISO-IR-109              Esc 2D 43    (Latin-3)
4       ISO-IR-110              Esc 2D 44    (Latin-4)
5       ISO-IR-144              Esc 2D 4C    Cyrillic
6       ISO-IR-127              Esc 2D 47    Arabic
7       ISO-IR-126              Esc 2D 46    Greek
8       ISO-IR-138              Esc 2D 48    Hebrew
9       ISO-IR-148              Esc 2D 4D    Turkish (Latin-5)
10      ISO-IR-157              Not listed   Sami (Latin-6)

The escape sequence for 8859-10 (Latin-6) is not listed in RFC 1345. 🙈

NOTE: The use of ISO 8859-3 and ISO 8859-4 is NOT recommended if other possibilities exist. 🚩
NOTE: There is a debate about the Arabic and Hebrew character sets. These languages are normally read right to left, but encodings have been done in both "visual" (left to right) and "phonetic" (right to left) ordering, there is significant disagreement about what the "right" way to do it is, and the character sets mentioned do not specify it. So, one should be careful not to use these character sets until a standard is agreed upon, or the result will probably be unreadable (siht ekil). 💀

(Note that there is some confusion as to what parts are actually standardized; the Norwegian standards institute reports that only part 1, 2, 3, 4, 6, 7 and 8 are currently standards. Other reports claim that both 8859-10 and 8859-11 are standards, and I definitely think that 8859-9 is.) 🤓

NOTE: ISO has not ruled out the possibility of changing the ISO 8859 standard. This would involve changing the registry information in this table, so this should be assumed valid for ISO 8859 versions that are current in 1993. 🕰️

The G1 set should be permanently shifted into the upper half of the code page. 🌈

When the text is not representable in one of the ISO-8859 character sets, the following rules may be applied:

  1. If any Latin characters are used, keep IA5 as the G0 set. 🍓
  2. If a mainstream character set is used (Greek, Cyrillic, Hebrew, Arabic), designate this as the G1 character set, and permanently shift this into the upper half of the code page (LS1R).

EXCEPTION: The Japanese community has a long tradition of switching between the Japanese 16-bit character set ISO-IR-87 and US-ASCII as the G0 set. See [7] for details. If ISO-IR-87 is used, that technique should be used instead of the one recommended here. 🎌

  1. If occasional extensions to a character set that is basically Latin occur (like accents, national variants and so on), and these are available in a single character set, designate the relevant set as G2 and use single shift (SS2) to invoke characters from this character set.

The ISO 8859 supplementary set, ISO-IR-154, is recommended for this purpose.

This corresponds to the ISO 4873 "second level" application. ✨

  1. If two non-Latin character sets are used, the second should be designated as G3, and shifted into the upper half of the code page by the use of Locking Shift 3 Right (LS3R).

This corresponds to the ISO 4873 "third level" application. 👑

  1. If avoidable, use of character sets with floating accents, like ISO 6937, should be avoided. 🙅‍♀️
  2. The shifts changing the lower half of the code table (SI/SO, LS2 and LS3) should NOT be used. 🚫

RATIONALE: Keeping the G0 set reserved for US-ASCII will ensure that text in US-ASCII has the same bit representation always. It's giving consistency, no cap. 💜

The use of the upper code page for other scripts ensures that both text in these languages and text of this type mixed with English can be represented without the use of shift sequences. Literally no shift sequences needed, which is so iconic. 🧚

If the language and/or content of a text is completely unknown, chapter 5 gives an algorithm that may be used to decide upon the character sets. This might, for instance, be suitable for use at automatic mail gateways. 📬

NOTE: At the time of this writing, few applications that use ISO 4873 level 2 and level 3 encoding exist. It has been estimated that implementing them in an application that already uses a rich repertoire of characters is a matter of programmer-days, not programmer-months, but this has not been proven. ⏳

4. GUIDELINES FOR THE RENDERING OF GENERALTEXT ✨

As a basic rule, one should NOT assume that any of the rules above are followed. 🙃

An user agent capable of rendering GeneralText should:

  1. ALWAYS be able to identify and render characters in IA5, no matter how they are designated and invoked. 💅
  2. ALWAYS be able to identify and render characters in the "native" character sets, no matter how they are designated and invoked. 🫶
  3. ALWAYS indicate the presence of characters that cannot be adequately represented on the current output device. 👀
  4. NEVER render a character in an unknown or unrepresentable character set by displaying the character in the same bit position in the native character set. 🙅‍♀️
  5. PREFERABLY be able to identify and render characters that are the same as characters in the "native" character sets, even though they are designated and invoked as part of other character sets. This applies in particular to the "invariant" part of ISO 8859, parts 1 through 6. 🌸
  6. PREFERABLY be able to combine the floating accents of ISO 6937 with their base characters for suitable rendering using the capabilities of the current output device. ✨
  7. PREFERABLY be able to display text both in a mode using fallbacks for nonrenderable characters and in a mode designating nonrenderable characters as such. 🦋
  8. PREFERABLY be able to save the content of a GeneralText message to a file or other suitable media, saving all character set information, for later processing by other means. It is not illegal to render the character set information into a different format; however, it should be noted that it is easy to lose vital information if the format chosen for representing character sets does not offer the possibility of referencing all character sets in the ISO registry of character sets. 📁

These requirements also apply to gateways that transform the message into some other format, for example a gateway that transforms a message into MIME using [7] for the purpose. 🔁

5. RECOMMENDATION FOR SELECTION OF CHARACTER SETS 💅

5.1. Algorithm for selection of character sets ✨

When one has text in which characters from several character sets occur, and wants to process this into a GeneralText document, it is often hard to guess right at the character sets to select. Honestly, same. 😭

The following paragraphs give an algorithm that can be started at the beginning of a message, and at the end of it, return a set of character sets that can be used as G0..G3 character sets, OR an indication that the task is impossible. 🧩

VARIABLES:

  • UsedSets: The set of character sets that MUST be used for this message.
  • UsableSets: The set of character sets that MAY be used for this message. Each set also contains a counter for each character position.
  • PossibleSets: The set of all the character sets known to be usable in the destination format.

ALGORITHM:

  1. Add IA5 (ISO-IR-6) to the UsedSets (as G0). 🌟
  2. Get the next character of the text. If the text is completely analyzed, go to FINISHED. 🔍
  3. If it is in the UsedSets, go to 2). 🔁
  4. Find the set of character sets from PossibleSets in which the character occurs. If it does not occur in any, report failure. 🚨
  5. If it is in a single character set in PossibleSets only, add this set to UsedSets, and go to 2). 🎯
  6. If it is in more than one character set, add these to PossibleSets (if not already present), and increment the counter for that character in all the sets. Go to 2). 🔢

Once you're done with the text, here's the FINISHED part, which is where the real drama happens 🎬

FINISHED)

1)   (FINAL SELECTION) Remove any character set in UsedSets from
     PossibleSets.

     Zero the counters for any character in PossibleSets that also
     occurs in UsedSets.
     WHILE (more characters left)
       Select one character set and move it from PossibleSets to
       UsedSqets.
       Zero the counters for all characters in this set in the other
       PossibleSets.
     END WHILE
     This step can be "tuned" any way you want, for instance by
     choosing the character sets most likely to be understood at
     the destination first, choosing the character sets covering
     the most characters first, avoiding multi-byte character sets
     as long as possible, or any other scheme suitable for the
     application.

5.2. WHAT TO DO ON FAILURE 🥀

Failure will occur in this schema if a character is found that is not in the PossibleSets. It may then be handled in one of the following ways:

  1. Replace the character with the SUB control character. 🫠
  2. Replace the character with Keld Simonsen Mnemonics [8]. This is a reversible transformation as long as the recipient is aware that it has been used, but requires passing out-of-band information to indicate this. 🧠
  3. Replace the lost characters with any suitable fallback or mnemonic scheme intended for human understanding. 🤝
  4. Bounce the message/refuse the conversion/give up. 🙈

The action to be taken may be different based on the percentage of "lost" characters. Choose your battles, queen 👑

If the message has "controls" like "conversion with loss prohibited", only the last possibility may be used. 🔒

Okay bestie, there are 2 steps in the algorithm above that are left for local judgement 🧠. Basically, you get to decide these yourself:

  1. Selection of the sets to appear in PossibleSets.
  2. The algorithm for deciding which character set to select in step 9.

Now, when you're generating X.400 GeneralText messages, the following is recommended ✨:

Sets in PossibleSets:

ISO-IR-6        Esc 28 42 (G0)  US-ASCII, IA5, ISO646
ISO-IR-100      Esc 2D 41 (G1)  ISO-8859-1   West Europe
ISO-IR-101      Esc 2D 42 (G1)  ISO-8859-2   Central/Eastern Europe
ISO-IR-144      Esc 2D 4C (G1)  ISO-8859-5   Cyrillic
ISO-IR-127      Esc 2D 47 (G1)  ISO-8859-6   Arabic
ISO-IR-126      Esc 2D 46 (G1)  ISO-8859-7   Greek
ISO-IR-138      Esc 2D 48 (G1)  ISO-8859-8   Hebrew
ISO-IR-148      Esc 2D 4D (G1)  ISO-8859-9   Turkish

The following multi-byte character sets are recommended 🌈:

ISO-IR-87 (Japanese JIS C6226-1983)     Esc 24 29 42 (G1)
ISO-IR-149 (Korean KS C 5601-1989)      Esc 24 29 43 (G1)
ISO-IR-58 (Chinese GB 2312-80)          Esc 24 29 41 (G1)

It is a STRONG recommendation that character sets not listed above, which do not add any new characters to the total set of characters given by the character sets above, should NOT be used in X.400 interchange. 🙅‍♀️

ISO-IR-87 is the Japanese character set that is allowed in a Teletex string, such as the subject field. 🎀

NOTE: ISO-IR-87 has been "superseded" by ISO-IR-168, which allows two extra Kanji characters. Any application that handles ISO-IR-87 should also be able to handle ISO-IR-168. 🧚

Algorithm for selecting character sets: start at the top of the list above, and add each set only if it is needed. Simple, but it slaps 🦋

6. REFERENCES 📚

  • [1] Information technology - ISO 8-bit code for information interchange - Structure and rules for implementation, Third edition, 1991-12-15.
  • [2] Information technology - 8-bit single-byte coded graphic character sets (parts 1-11; the parts have different dates, the ones referenced here are from RFC 1345).
  • [3] Information technology - Coded graphic character set for text communication (parts 1 and 2; part 2 dated 1983-12-15).
  • [4] Code for the representation of names of languages. 1988 version.
  • [5] CCITT Recommendation X.209(1988): Specification of Basic Encoding Rules for Abstract Syntax Notation One (ASN.1). Technically aligned with ISO 8825 and ISO 8825/AD 1.
  • [6] Information Technology - Universal Multiple-Octet Coded Character Set (UCS) - ISO 10646.
  • [7] Murai, J., Crispin M., and E. van der Poel, "Japanese Character Encoding for Internet Message Bodies", RFC 1468, Keio University, Panda Programming, June 1993.
  • [8] Simonsen, K., "Character Mnemonics & Character Sets", RFC 1345, Rationel Almen Planlaegning, June 1992.
  • [9] Information Technology - Text communication - Message- Oriented Text Interchange Systems (MOTIS) - ISO 10021 - October 1988.
  • [10] ISO DIS documents describing X.400/84 with slight extensions. Now very hard to get copies of, since they failed to become ISes. 😩
  • [11] CCITT Recommendation X.420 (1988), Interpersonal Messaging System.
  • [12] International Standard--Information Processing-- ISO 7-bit and 8-bit coded character sets--Code extension techniques, ISO 2022:1986.

7. Security Considerations 🔐

Security issues are not discussed in this memo. 😭

8. Author's Address 💌

Harald Tveit Alvestrand
SINTEF DELAB
N-7034 Trondheim
NORWAY

EMail: Harald.Alvestrand@delab.sintef.no
📦 where this page comes from