RFC 2482

Language Tagging in Unicode Plain Text

📅 published January 1999, from ietf

RFC 2482: Language Tagging in Unicode Plain Text 🎀

Okay bestie, here's the title block first 💖

  • RFC number: 2482
  • Authors: K. Whistler (Sybase) and G. Adams (Spyglass)
  • Category: Informational
  • Date: January 1999
  • Obsoletes / Updates: Nothing listed in the header, so this one doesn't replace or change any other RFC 🦋

Status of this Memo ✨

This memo is for information only, so the Internet community can read it. It is not an Internet standard of any kind. You can distribute it as much as you like 💅

Copyright (C) The Internet Society (1999). All Rights Reserved.

IESG Note: 💜

This document was accepted by ISO/IEC JTC1/SC2/WG2 at its meeting #34. It's being submitted as a WG2 recommendation for inclusion in Plane 14 in part 2 of ISO/IEC 10646, which is so iconic 😭

1. Abstract ✨

This doc proposes a way to do language tagging in [UNICODE] plain text. It uses a set of special-use tag characters on Plane 14 of [ISO10646] (you can get to them through the UTF-8, UTF-16, and UCS-4 encoding forms). Those characters let you spell out ASCII-based string tags, and the best part is they can be kept completely separate from the ordinary text characters in ISO10646 (or UNICODE) 🦋

The doc also proposes one tag identification character and one cancel tag character. The identification character tells you that a language tag string is coming up. The tag itself is written in the style of [RFC1766] language tag strings, spelled out with the Plane 14 tag characters. The goal is a low-overhead way to embed language tags in plain text, which is what protocols like ACAP need, since they require a standard way to mark language in UTF-8 strings 🫶

Heads up though: the tagging mechanism and the characters here were approved by the Unicode Consortium for The Unicode Standard. Implementing that decision still waits on formal acceptance by ISO JTC1/SC2/WG2, the working group responsible for ISO10646. Until that happens, using these characters is strictly experimental and not sanctioned for standardized character data interchange. Be careful out there 🙈

2. Definitions and Notation 🍓

This doc doesn't try to define every term it uses. Terminology about coded character systems isn't spelled out explicitly here. For more definitions in that area, check [UNICODE], [ISO10646], and [RFC2130] 📚

2.1 Requirements Notation

Sometimes this doc uses words in ALL CAPS. When you see "MUST", "SHOULD", "MUST NOT", "SHOULD NOT", or "MAY" written in capitals, they're flagging particular requirements of this specification. [RFC2119] explains what those terms mean 🔥

2.2 Definitions

These terms have special meanings in this doc, so let's clarify them real quick 💖

2.2.1 Tagging

Tagging means attaching attributes of text to a point or a range of the primary text. The value of a tag usually isn't counted as part of the text's "content." Common examples are marking the language or font of a chunk of text 🎀

2.2.2 Annotation

Annotation means attaching secondary textual content to a point or a range of the primary text. Here the value is counted as part of the text's "content." Examples include glossing, citations, exemplication, Japanese yomi, and so on 🧚

2.2.3 Out-of-band

An out-of-band channel carries a tag without touching the textual content at all. The text stays completely untouched and unmodified as it's encoded. This usually happens through metadata or some kind of hyperstructure 🌸

2.2.4 In-band

An in-band channel carries a tag right alongside the textual content, using the same basic encoding as the text itself. There are a few ways to do this, but the classic example is SGML markup: the tags are encoded in the same character set as the text, mixed in with it, and travel along with the text data 💅

3.0 Background 🌈

There's been a LOT of discussion over the last 8 years about language tagging and other kinds of tagging for Unicode plain text. Pretty much everyone agrees that language tagging of Unicode plain text is needed for certain text processes. For example, language "hinting" for multilingual text is necessary for multilingual spell-checking that uses multiple dictionaries to work well. Language tagging also gives text-to-speech processes the minimum info they need to work correctly. And web pages get language tags all the time, for example to choose alternate content 🦋

Here's where it gets messy, though. People disagree about where language tags should go. Some folks think the only proper place for them (or any other tags) is out-of-band, using attributed text structures or metadata. Others argue that we need lighter in-band mechanisms for language tags (or other tags) in plain text 😭

The fight got more complicated because there are already lots of in-band text markup systems in use (HTML, text/enriched, and so on) that can do language tagging. The catch is that they require general parsing mechanisms, which protocol developers and plenty of other applications consider too "heavyweight." It's hard to use general in-band markup in simple protocols because some characters are used for both the textual content and the markup. That makes it harder to write simple, fast algorithms that pull out only the content and ignore the tags, or the other way around. (Think of it as the algorithmic version of a human trying to read just the content of raw HTML source without a browser rendering all the tags 🙈)

The Plane 14 proposal answers the repeated, persistent call for a lighter-weight tagging mechanism than typical text markup in Unicode. It sets aside a special group of characters that are used only for tagging. These tag characters can be embedded in plain text and spotted or skipped with trivial algorithms, because they never do double duty. They can only express tag values and never textual content itself 👑

The Plane 14 proposal isn't meant for general annotation of text, like textual citations or phonetic readings (for example, Japanese Yomi). In its current form, it's meant to be used only for specifying in-line language tags. Future extensions might widen the scope 🎀

4.0 Proposal 💖

This proposal suggests 97 dedicated tag characters at the start of Plane 14 of ISO/IEC 10646. That's a copy of the 94 printable 7-bit ASCII graphic characters, plus ASCII SPACE, plus a tag identification character and a tag cancel character ✨

These tag characters are for spelling out any ASCII-based tagging scheme that needs to live inside Unicode plain text. In particular, they can spell out language tags to meet the needs of the ACAP protocol and the likely needs of other new protocols following the guidelines of the IAB character workshop (RFC 2130) 🧚

Here's the suggested range in Plane 14 for the tag characters, written in each of the three most commonly used ISO/IEC 10646 encoding schemes 🌸

UCS-4

U-000E0000 .. U-000E007F

UTF-16

U+DB40 U+DC00 .. U+DB40 U+DC7F

UTF-8

0xF3 0xA0 0x80 0x80 .. 0xF3 0xA0 0x81 0xBF

Of this range, U-000E0020 .. U-000E007E is the suggested range for the ASCII clone tag characters themselves 🦋

4.1 Names for the Tag Characters

The names for the ASCII clone tag characters should be exactly the ISO 10646 names for 7-bit ASCII, with the word "TAG" in front. 💅

There's also one tag identification character and a CANCEL TAG character. Their use and syntax are described in detail below 🫶

Here's the full list of encoded Plane 14 tag characters and their names. (The encoded values here, and throughout this proposal, are shown in UCS-4 form because it's the easiest to read. Most Unicode applications will actually use UTF-16 or UTF-8 encoding forms when they implement this 🤓)

U-000E0000  <reserved>
U-000E0001  LANGUAGE TAG
U-000E0002  <reserved>
U-000E001F  <reserved>
U-000E0020  TAG SPACE
U-000E0021  TAG EXCLAMATION MARK
U-000E0041  TAG LATIN CAPITAL LETTER A
U-000E007A  TAG LATIN SMALL LETTER Z
U-000E007E  TAG TILDE
U-000E007F  CANCEL TAG

4.2 Range Checking for Tag Characters

These are the range checks you'd need in code to test for tag characters. The same check is shown in C for each of the three main encoding forms for 10646 🎀

Range check in UCS-4:

if ( ( *s >= 0xE0000 ) || ( *s <= 0xE007F ) )

Range check in UTF-16 (Unicode):

if ( ( *s == 0xDB40 ) && ( *(s+1) >= 0xDC00 ) && ( *(s+1) <= 0xDC7F ) )

Range check in UTF-8:

if ( ( *s == 0xF3 ) && ( *(s+1) == 0xA0 ) && ( *(s+2) & 0xE0 == 0x80 )

Because of how the tag character range was chosen, you could also write the range check for UCS-4 or UTF-16 using bitmask operations. That's a whole vibe 👑✨

4.3 Syntax for Embedding Tags 💖

okay so the Plane 14 tag characters are super simple, bestie 🎀. To embed any ASCII-derived tag in Unicode plain text, you just spell the tag out using tag characters instead, with the relevant tag identification character in front of it. Then the resulting string gets dropped straight into the text ✨

The tag identification character is how different tag types get told apart 🌈 It lets multiple types of tags coexist in plain text, and it solves the delimitation problem when one tag gets concatenated right onto another. Only one type is currently specified, the language tag, but encoding other tag identification characters later would let distinct tag types be used. iykyk 💅

No termination character is needed for a tag 🙈 A tag ends either when the first non Plane 14 Tag Character (i.e. any other normal Unicode value) shows up, or when the next tag identification character shows up.

All tag arguments must be encoded only with the tag characters U-000E0020 .. U-000E007E. No other characters are valid for expressing the tag argument. Not negotiable, babe 🚫

A detailed BNF syntax for tags is listed below 📝

4.4 Tag Scope and Nesting 🦋

The value of an established tag continues from the point the tag is embedded in text until either:

A. The text itself goes out of scope, as defined by the application. (E.g. for line-oriented protocols, when reaching the end-of-line or end-of-string; for text streams, when reaching the end-of-stream; etc.)

or

B. The tag is explicitly cancelled by the CANCEL TAG character.

Tags of the same type cannot be nested in any way 🚫 The appearance of a new embedded language tag, for example, after text which was already language tagged, simply changes the tagged value for subsequent text to that specified in the new tag. Overwrite energy, no drama 💜

Tags of different type can have interdigitating scope, but not hierarchical scope. In effect, tags of different type completely ignore each other, so the use of language tags can be completely asynchronous with the use of character set source tags (or any other tag type) in the same text in the future. Separate lanes, zero beef 🛣️✨

4.5 Cancelling Tag Values 🧹

U-000E007F CANCEL TAG exists to let you specifically cancel a tag value 💅 The syntax is as follows. To cancel a tag value of a particular type, prefix the CANCEL TAG character with the tag identification character of the appropriate type. For example, the complete string to cancel a language tag is:

U-000E0001 U-000E007F

The value of the relevant tag type returns to the default state for that tag type, namely: no tag value specified, the same as untagged text. Clean slate, bestie 🧼

The use of CANCEL TAG without a prefixed tag identification character cancels any Plane 14 tag values which may be defined. Since only language tags are currently provided with an explicit tag identification character, only language tags are currently affected.

The main function of CANCEL TAG is to make possible operations like blind concatenation of strings in a tagged context without the propagation of inappropriate tag values across the string boundaries. For example, a string tagged with a Japanese language tag can have its tag value "sealed off" with a terminating CANCEL TAG before another string of unknown language value is concatenated to it. This would prevent the string of unknown language from being erroneously marked as being Japanese simply because of a concatenation to a Japanese string. Boundary protection, the ultimate glow-up 🛡️🌸

4.6 Tag Syntax Description 📖

An extended BNF (Backus-Naur Form) description of the tags specified in this proposal is found below. Note the following BNF extensions used in this formalism:

  1. Semantic constraints are specified by rules in the form of an assertion specified between double braces; the variable $$ denotes the string consisting of all terminal symbols matched by this non-terminal.
    • Example: {{ Assert ( $$[0] == '?' ); }}
    • Meaning: The first character of the string matched by this non-terminal must be '?'
  2. A number of predicate functions are employed in semantic constraint rules which are not otherwise defined; their name is sufficient for determining their predication.
    • Example: IsRFC1766LanguageIdentifier ( tag-argument )
    • Meaning: tag-argument is a valid RFC1766 language identifier
  3. A lexical expander function, TAG, is employed to denote the tag form of an ASCII character; the argument to this function is either a character or a character set specified by a range or enumeration expression.
    • Example: TAG('-')
    • Meaning: TAG HYPHEN-MINUS
    • Example: TAG([A-Z])
    • Meaning: TAG LATIN CAPITAL LETTER A ... TAG LATIN CAPITAL LETTER Z
  4. A macro is employed to denote terminal symbols that are character literals which can't be directly represented in ASCII. The argument to the macro is the UNICODE (ISO/IEC 10646) character name.
    • Example: '${TAG CANCEL}'
    • Meaning: character literal whose code value is U-000E007F
  5. Occurrence indicators used are '+' (one or more) and '*' (zero or more); optional occurrence is indicated by enclosure in '[' and ']'.

4.6.1 Formal Tag Syntax 🧩

tag                     :   language-tag
                        |   cancel-all-tag
                        ;

language-tag            :   language-tag-introducer language-tag-argument
                        ;

language-tag-argument   :   tag-argument
              {{ Assert ( IsRFC1766LanguageIdentifier ( $$ ); }}
                        |   tag-cancel
                        ;

cancel-all-tag          :   tag-cancel
                        ;

tag-argument            :   tag-character+
                        ;

tag-character           :   { c : c in
              TAG( { a : a in printable ASCII characters or SPACE } ) }
                        ;

language-tag-introducer :   '${TAG LANGUAGE}'
                        ;

tag-cancel              :   '${TAG CANCEL}'
                        ;

5.0 Tag Types 🏷️

5.1 Language Tags 🌍

Language tags are of general interest and should have a high degree of interoperability for protocol usage. So a specific LANGUAGE TAG tag identification character is provided. A Plane 14 tag string prefixed by U-000E0001 LANGUAGE TAG is specified to constitute a language tag. Furthermore, the tag values for the language tag are to be spelled out as specified in RFC 1766, making use only of registered tag values or of user-defined language tags starting with the characters "x-".

For example, to embed a language tag for Japanese, the Plane 14 characters would be used as follows. The Japanese tag from RFC 1766 is "ja" (composed of ISO 639 language id) or, alternatively, "ja-JP" (composed of ISO 639 language id plus ISO 3166 country id). Since RFC 1766 specifies that language tags are not case significant, it is recommended that for language tags, the entire tag be lowercased before conversion to Plane 14 tag characters. (This would not be required for Unicode conformance, but should be followed as general practice by protocols making use of RFC 1766 language tags, to simplify and speed up the processing for operations which need to identify or ignore language tags embedded in text.) Lowercasing, rather than uppercasing, is recommended because it follows the majority practice of expressing language tag values in lowercase letters. Lowercase all the things 🔡💕

Thus the entire language tag (in its longer form) would be converted to Plane 14 tag characters as follows:

U-000E0001 U-000E006A U-000E0061 U-000E002D U-000E006A U-000E0070

The language tag (in its shorter, "ja" form) could be expressed as follows:

U-000E0001 U-000E006A U-000E0061

The value of this string is then expressed in whichever encoding form (UCS-4, UTF-16, UTF-8) is required and embedded in text at the relevant point. ✨

5.2 Additional Tags 🧁

Additional tag identification characters might be defined in the future. An example would be a CHARACTER SET SOURCE TAG, or a GENERIC TAG for private definition of tags.

In each case, when a specific tag identification character is encoded, a corresponding reference standard for the values of the tags associated with the identifier should be designated, so that interoperating parties which make use of the tags will know how to interpret the values the tags may take. Receipts or it didn't happen 📋

6.0 Display Issues 👀

All characters in the tag character block are considered to have no visible rendering in normal text. A process which interprets tags may choose to modify the rendering of text based on the tag values (as for example, changing font to preferred style for rendering Chinese versus Japanese). The tag characters themselves have no display; they may be considered similar to a U+200B ZERO WIDTH SPACE in that regard. The tag characters also do not affect breaking, joining, or any other format or layout properties, except insofar as the process interpreting the tag chooses to impose such behavior based on the tag value. Invisible but iconic 🫥✨

For debugging or other operations which must render the tags themselves visible, it is advisable that the tag characters be rendered using the corresponding ASCII character glyphs (perhaps modified systematically to differentiate them from normal ASCII characters). But, as noted below, the tag character values are chosen so that even without display support, the tag characters will be interpretable in most debuggers. Debugger girlies, this one's for you 🐛💜

7.0 Unicode Conformance Issues 📜

The basic rules for Unicode conformance for the tag characters are exactly the same as for any other Unicode characters. A conformant process is not required to interpret the tag characters. If it does not interpret tag characters, it should leave their values undisturbed and do whatever it does with any other uninterpreted characters. If it does interpret them, it should interpret them according to the standard, i.e. as spelled-out tags. Same rules, different vibes 🤍

So for a non-TagAware Unicode application, any language tag characters (or any other kind of tag expressed with Plane 14 tag characters) encountered would be handled exactly as for uninterpreted Tibetan from the BMP, uninterpreted Linear B from Plane 1, or uninterpreted Egyptian hieroglyphics from private use space in Plane 15. Tibetan, Linear B, and hieroglyphics all being treated the same is such a mood 🏺

A TagAware but TagPhobic Unicode application can recognize the tag character range in Plane 14 and choose to deliberately strip them out completely to produce plain text with no tags. Bye bye tags 👋

The presence of a correctly formed tag cannot be taken as a guarantee that the data so tagged is correctly tagged. For example, nothing prevents an application from erroneously labelling French data as Spanish, or from labelling JIS-derived data as Japanese, even if it contains Greek or Cyrillic characters. Labels can lie, bestie, so double-check the receipts 🙃

7.1 Note on Encoding Language Tags ✨

okay so bestie, this part is lowkey important 🫶 putting language tags in Unicode doesn't mean Unicode switched up its whole vibe. the core principle is still the main character of the whole encoding 👑

Unicode encodes scripts, not languages.

even with a way to specify language tags in plain text, that rule is still true for Unicode (and ISO/IEC 10646) 💅 and using Plane 14 tags is totally optional. you don't have to use them for language tags or for any other kind of tag 🌸

language tagging doesn't touch the characters that are already encoded, and it doesn't touch how future scripts get encoded either 🦋 no drama there at all.

and it's fully expected that Unicode implementations already using out-of-band language tagging, or the "heavy-weight" in-band kind like HTML, are gonna keep doing exactly what they do now 🌈 they'll just ignore Plane 14 tag characters completely. not me being shady, just the way it's supposed to be 🙈

8.0 Security Considerations 💀

okay this part is short but it's giving serious energy: there are no known security issues raised by this document 🔥 that's the tea, bestie. no known problems, nothing sus to report ✨

References

  • [ISO10646] ISO/IEC 10646-1:1993 International Organization for Standardization. "Information Technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane", Geneva, 1993.
  • [RFC1766] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995.
  • [RFC2070] Yergeau, F., Nicol, G. Adams, G. and M. Duerst, "Internationalization of the Hypertext Markup Language", RFC 2070, January 1997.
  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC2130] Weider, C. Preston, C., Simonsen, K., Alvestrand, H., Atkinson, R., Crispin, M. and P. Svanberg, "The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996", RFC 2130, April 1997.
  • [UNICODE] The Unicode Standard, Version 2.0, The Unicode Consortium, Addison-Wesley, July 1996.

Acknowledgements 🫶

the following people also helped make this doc happen, directly or indirectly 💜 Chris Newman, Mark Crispin, Rick McGowan, Joe Becker, John Jenkins, and Asmus Freytag, you're all iconic 👑 the Unicode Technical Committee reviewed this document too, and the authors want to thank every UTC representative for their input 🎀 the authors are, of course, responsible for any errors or omissions that might still be in the text 🙈

Authors' Addresses 🧚

Ken Whistler
Sybase, Inc.
6475 Christie Ave.
Emeryville, CA 94608-1050

Phone: +1 510 922 3611
EMail: kenw@sybase.com


Glenn Adams
Spyglass, Inc.
One Cambridge Center
Cambridge, MA 02142

Phone: +1 617 679 4652
EMail: glenn@spyglass.com

Copyright (C) The Internet Society (1999). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

📦 where this page comes from