RFC 5198

Unicode Format for Network Interchange

📅 published March 2008, from ietf

RFC 5198: Unicode Format for Network Interchange 🎀

okay bestie, this one is basically the internet's "how do we send text in a way everyone can read" spec, but for Unicode 🌈

Here's the tea on the title block. It's Standards Track 🔥, so it's an actual internet standard, dated March 2008. It was written by J. Klensin and M. Padlipsky 🧚. It obsoletes RFC 698 💅 and updates RFC 854 ✨.

Status of This Memo 📜

this is the part that says it's a real internet standards track protocol 💖 and that the community is invited to discuss it and suggest improvements 🫶. If you want to know its official standardization state, check the current edition of the "Internet Official Protocol Standards" (STD 1) 💜. Also, distribution of this memo is unlimited 🦋.

Abstract 🌸

the internet needs a standard way to send internationalized "text," kind of like how ASCII got specified back in the early ARPANET days 🕰️. This document is that spec. It uses UTF-8, along with normalization and specific line-ending sequences 💅.

Table of Contents 🍓

1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  2
  1.1.  Requirement for a Standardized Text Stream Format  . . . .  2
  1.2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  3
2.  Net-Unicode Definition . . . . . . . . . . . . . . . . . . . .  3
3.  Normalization  . . . . . . . . . . . . . . . . . . . . . . . .  5
4.  Versions of Unicode  . . . . . . . . . . . . . . . . . . . . .  5
5.  Applicability and Stability of this Specification  . . . . . .  7
  5.1.  Use in IETF Applications Specifications  . . . . . . . . .  7
  5.2.  Unicode Versions and Applicability . . . . . . . . . . . .  7
6.  Security Considerations  . . . . . . . . . . . . . . . . . . .  9
7.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 10
Appendix A.  History and Context . . . . . . . . . . . . . . . . . 11
Appendix B.  The ASCII NVT Definition  . . . . . . . . . . . . . . 12
Appendix C.  The Line-Ending Problem . . . . . . . . . . . . . . . 14
Appendix D.  A Note about Related Future Work  . . . . . . . . . . 14
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
  Normative References . . . . . . . . . . . . . . . . . . . . . . 15
  Informative References . . . . . . . . . . . . . . . . . . . . . 16

1. Introduction ✨

1.1. Requirement for a Standardized Text Stream Format 💖

okay so historically, internet protocols were mostly ASCII-based 🫶. When protocols said "text," they basically meant ASCII text, specifically in Network Virtual Terminal ("NVT") or "Network ASCII" form (see Appendix A and Appendix B) 📟. Protocols that went beyond ASCII usually added ways to say which character set they're using, and often which language too 🌍.

Now the world is way more international, so "text" just doesn't automatically mean "network ASCII" anymore 😭. The good news is that everyone's converging on Unicode [Unicode] [ISO10646] as one international interchange character coding 🎉. We don't need a separate standard for every script anymore (like one for Arabic, one for Cyrillic, one for Devanagari) 🙌. We also don't need standards for languages that share a script, like French, German, or Swedish 🇫🇷🇩🇪🇸🇪. The bad news? "Use Unicode" is even more ambiguous than "use ASCII" was decades ago 💀. So yes, we totally need a Unicode-based text type for interchange, but it needs care 💅.

Unicode gives every character an integer called its "code point," in the range 0-0x10ffff 🔢. Those integers can be encoded into byte sequences in at least three standard, well-recognized encoding forms. All of them are fully defined in The Unicode Standard and the documents cited below 📚:

  • UTF-8 [RFC3629] is a variable-length encoding that can be applied uniformly to all code points 🦋.
  • UTF-16 [RFC2781] encodes the Unicode characters with code points below 65536 straight up as 16-bit integers. For bigger code points, it has a "surrogate" mechanism that encodes them in 32 bits ✨.
  • UTF-32 (also known as UCS-4) just encodes each code point as a 32-bit integer 🧮.

The older stuff, like the 16-bit UCS-2, is now strongly discouraged 🙅‍♀️.

Like ASCII, any of these forms can be used with different line-ending conventions 📏. That flexibility can cause extra confusion, for example with index (offset) references into documents that are based on character counts 🙈.

So this document proposes "Net-Unicode" as a new standardized text transmission form for the Internet 🌸. It's meant as the internationalized alternative to NVT ASCII, for new protocols and, where it makes sense, updated ones 🧚. UTF-8 [RFC3629] is the coding of choice because it plays really nicely with ASCII, plus other reasons covered in the IETF character set policy [RFC2277] 💜. Net-Unicode is defined in Section 2, and the later sections give background and explanation 📖.

Whenever there's a choice, Unicode is to be used with the text encoding defined here 💖. This combo is preferred over the double-byte encoding of "extended ASCII" [RFC0698] or the random assortment of per-language or per-country character coding systems 🤷‍♀️. Unicode SHOULD be used with the text encoding specified here.

1.2. Terminology 🎀

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 📖✨.

2. Net-Unicode Definition 🦋

okay here's the main event 👑. Net-Unicode (aka the Network Unicode format) goes like this. Some parts are deliberately informal 🫶. They're guidance for specific profiles or rules in the protocols that reference this one, not hard global laws 💅.

  1. Characters MUST be encoded in UTF-8 as defined in [RFC3629] 💖.
  2. If the protocol has the concept of "lines," line-endings MUST be indicated by the sequence Carriage-Return (CR, U+000D) followed by Line-Feed (LF, U+000A), often just called CRLF 📏. CR SHOULD NOT appear except when followed by LF. The only other allowed context where CR is permitted is the combination CR NUL, which is not recommended (see the note at the end of this section) 🙃.
  3. The control characters in the ASCII range (U+0000 to U+001F and U+007F to U+009F) SHOULD generally be avoided ⚠️. Space (SP, U+0020), CR, LF, and Form Feed (FF, U+000C) are exceptions to this, but using all of them except the first needs care, as discussed elsewhere in this document 🧐. The so-called "C1 Controls" (U+0080 through U+009F), which never appeared in ASCII, MUST NOT appear 🚫.

FF should be used only with caution 😬. It doesn't have a standard, universal meaning, and if its use assumes a page length, those assumptions might not fit international contexts (think 8.5x11 inch paper versus A4) 📄. Other control characters are used to affect display format, control devices, or structure files 🖥️. None of those uses are appropriate for streams of plain text 🙅‍♀️.

  1. Before transmission, all character sequences SHOULD be normalized according to Unicode normalization form "NFC" (see Section 3) ✨.
  2. As suggested in Section 6 of RFC 3629, the Byte Order Mark ("BOM") signature MUST NOT appear at the beginning of these text strings 🙅‍♀️.
  3. Systems conforming to this specification MUST NOT transmit any string containing any code point that is unassigned in the version of Unicode they depend on 🚫. The version of NFC and the version of Unicode used by that system MUST be consistent 🔐.

The use of LF without CR is questionable, so check Appendix B for more discussion 🤔. The newer control characters IND (U+0084) and NEL ("Next Line", U+0085) could've been used to sort out the various line-ending situations 🌀. But their use hasn't been established on the Internet, lots of protocols require CRLF, and IND and NEL fall within the "C1 Controls" group (see below), so they MUST NOT be used 🙅‍♀️. The same goes for the even newer line and paragraph separators at U+2028 and U+2029, and any future characters that get defined for these functions 🔮. For this specification and protocols that depend on it, lines end in CRLF and only in CRLF 💖. Anything that doesn't end in CRLF is either not a line or is severely malformed 💀.

The NVT specification had a bunch of extra provisions, like optional backspacing and "bare CR" (sent as CR NUL) for making overstruck character sequences 🖋️. Unicode has way more precomposed characters, plus combining characters, and people increasingly use markup to show things like emphasis (instead of cramming that into special characters) ✨. So those old sequences should be largely unnecessary 🙈. These sequences SHOULD be avoided if at all possible. But since they were optional in NVT applications and this spec is an NVT superset, they can't be banned completely 😤. The most important rule is that CR MUST NOT appear unless it's immediately followed by LF (end of line) or NUL 🔥. And because NUL (an octet with all zeros, i.e., %x00 in the notation of [RFC5234]) is hostile to programming languages that use it as a string delimiter, the CR NUL sequence SHOULD be avoided for that reason too 💀.

3. Normalization 💅

sometimes strings of Unicode are fundamentally equivalent, basically representing the same text 👯‍♀️. Unicode calls these "canonical equivalents." For example, these pairs of strings are canonically equivalent ✨:

U+2126 OHM SIGN
U+03A9 GREEK CAPITAL LETTER OMEGA

U+0061 LATIN SMALL LETTER A, U+0300 COMBINING GRAVE ACCENT
U+00E0 LATIN SMALL LETTER A WITH GRAVE

Comparing strings gets so much easier if every case like this is written as one unique form 🧠. The Unicode Consortium specifies a normalization form called NFC [NFC], which has the mappings and mechanisms to turn all canonically equivalent sequences into that single form 💖. Usually this form makes precomposed characters wherever a sequence can be represented that way. It also reorders other combining marks so they have a unique, unambiguous order 🌈.

Of the normalization forms defined for Unicode, NFC is the closest to what people actually use 🙌. It also minimizes side effects from treating characters as equivalent when they aren't equivalent in every situation, and it usually needs the least work when converting from non-Unicode encodings 🧹.

The section above says that, except in very unusual circumstances, all Net-Unicode strings have to be sent in normalized form 📦. But some applications run on operating system libraries they barely control, and the robustness principle says receivers should be ready to get unnormalized strings and not overreact when they do 🫶. so be chill about it, bestie 🦋.

4. Versions of Unicode ✨

okay so Unicode keeps changing and growing over time, bestie 🌈 Big chunks of code space are reserved for later, and new versions show up at regular intervals with new scripts and characters. Sometimes they also tweak some property definitions 👀 The iconic part is ASCII [ASCII]. When it got picked, its code space was already full by the time the Standard was first published, so there was no practical way to add characters or change code point assignments without being obviously incompatible. Pure restraint 💅

While there are some security issues if people deliberately try to trick the system (see Section 6), Unicode version changes should not have a significant impact on the text stream specification of this document, and here's why 🫶

  • The transformation between Unicode code table positions and the corresponding UTF-8 code is algorithmic. It does not depend on whether a code point has been assigned or not.
  • The normalization recommended here, NFC (see Section 3), performs a very limited set of mappings, much more limited than those of the more extensive NFKC used in, e.g., Nameprep [RFC3491].

The NFC tables may get updated over time as new characters are added, but the Unicode Consortium has guaranteed the stability of all NFC strings 🔒 That means if a string doesn't contain any unassigned characters and it's normalized according to NFC, it will always be normalized according to all future versions of the Unicode Standard. So the stability of the Net-Unicode format is guaranteed when any implementation that converts text into Net-Unicode format does not permit unassigned characters. Stable and sound 🎀

Because Unicode code points that are reserved for private use don't have standard definitions or normalization interpretations, they SHOULD be avoided in strings intended for Internet interchange. Not the vibe for public strings 🙈

Now, if Unicode ever changed in a way that broke these assumptions, like invalidating the byte string order specified in RFC 3629 or changing the stability of NFC as stated above, this specification would not apply. Put differently, this specification applies only to versions of Unicode starting with version 5.0 and extending to, but not including, any version where changes are made in either the UTF-8 definition or to NFC stability. Those changes would break established Unicode policies, so they're unlikely 🥺 But if they ever happened, it would be necessary to evaluate them for compatibility with this specification and other Internet uses of NFC.

If the specification of a protocol references this one, then strings that the protocol receives that appear to be UTF-8 and aren't otherwise identified (e.g., by charset labeling) SHOULD be treated as using UTF-8 in conformance with this specification. Main character energy for UTF-8 ✨

5. Applicability and Stability of this Specification

5.1. Use in IETF Applications Specifications

So during the development of this spec, there was some confusion about where it would actually be useful 🤔 Individual MIME media types used in email and with HTTP have their own rules about UTF-8 character types and normalization, and the application transport protocols impose their own conventions about line endings. Messy, right? There are three answers 💖

The first is that, in retrospect, it would have been better to have those protocols and content types standardized the way specified here, even though it's definitely too late to change them now 😭 The second is that we have several protocols that depend on either the original Telnet design or other arrangements that need a standard, interoperable string definition without specific content-labels of one sort or another. Whois [RFC3912] is an example member of this group. As people consider upgrading them for non-ASCII use, this specification provides a normative reference that gives the same stability that NVT has given the ASCII forms. Stability goals 🧚

This specification is intended for use by other specifications that haven't yet defined how to use Unicode. Having a preferred standard Internet definition for Unicode text streams -- rather than just one for transmission codings -- may help improve the specification and interoperability of protocols developed in the future 🌸 This specification is not intended for use with specifications that already allow the use of UTF-8 and precisely define that use. Those already have it handled, bestie 💅

5.2. Unicode Versions and Applicability

The IETF faces a practical dilemma with regard to Unicode versions 😩 Each new version brings new characters and sometimes new combining characters. Version 5.0 introduces the new concept of sequences of characters named as if they were individual characters (see [NamedSequences]). The normalization represented by NFC is stable if all strings are transmitted and stored in normalized form, if corrections are never made to character definitions or normalization tables, and if unassigned code points are never used. That last one matters a lot, because an unassigned code point always normalizes to itself. However, if the same code point gets assigned to a character in a future version, it may take part in some other normalization mapping (some specific difficulties in this regard are discussed in [RFC4690]). It's worth noting that transmission in normalized form is not required by either the IETF's UTF-8 Standard [RFC3629] or by standards dependent on the current version of Stringprep [RFC3454] 👀

All would be well with this as described in Section 4, except for one problem 🥀 Applications typically don't do their own conversions to Unicode, and they may not do their own normalizations either. Instead they rely on operating system or language library functions, and those functions may be upgraded or otherwise changed without any changes to the application code itself. So there may be no plausible way for an application to know which version of Unicode, or which version of the normalization procedures, it's using, and there's no way it can guarantee that the two will be consistent. Chaos energy 🌀

Because of per-version changes in definitions and tables, Stringprep and documents depending on it are now tied to Unicode Version 3.2 [Unicode32]. Full interoperability of Internet Standard UTF-8 [RFC3629], when used with normalization as specified here, depends on the normalization definitions and the definition of UTF-8 itself not changing after Unicode Version 5.0. These assumptions seem fairly safe, but they're still assumptions 🙃 Rather than being linked to the latest available version of Unicode, version 5.0 [Unicode], or broader concepts of version independence based on specific assumptions and conditions, this specification could reasonably have been tied, like Stringprep and Nameprep, to Unicode 3.2 [Unicode32] or some more recent intermediate version. But beyond the obvious disadvantages of having different IETF standards tied to different versions of Unicode, the library-based application implementation behavior described above makes these version linkages nearly meaningless in practice 🫠

In theory, one can get around this problem in four ways:

  1. Freeze on a particular version of Unicode and try to insist that applications enforce that version by, e.g., containing lists of unassigned characters and prohibiting their use. Of course, this would prohibit evolution to include newly-added scripts, and the tables of unassigned code points would be cumbersome 😵‍💫
  2. Require that every Unicode "text" string or file start with a version indication, somewhat akin to the "byte order mark" indicator. It's unlikely this provision would be practical. More importantly, it would require that each application implementation be prepared to either support multiple normalization tables and versions or reject text from Unicode versions it wasn't prepared to deal with.
  3. Devise a different set of normalization rules that would, e.g., guarantee that no character assigned to a previously-unassigned code point in Unicode was ever normalized to anything but itself, and use those rules instead of NFC. It's not clear whether or not such a set of rules is possible, or whether some other completely stable set of rules could be devised, perhaps in combination with restrictions on the ways characters were added in future versions of Unicode 🤷‍♀️
  4. Devise a normalization process that is otherwise equivalent to NFC but that rejects code points that are unassigned in the current version of Unicode, rather than mapping those code points to themselves. This would still leave some risk of incompatible corrections in Unicode and possibly a few edge cases, but it's probably stable enough for Internet use in the overwhelming number of cases. This process has been discussed in the Unicode Consortium under the name "Stable NFC" 🍓

None of these approaches seems ideal. The ideal procedure would be as stable and predictable as ASCII has been, but that level simply isn't feasible as long as Unicode keeps evolving by adding new code points and scripts 🥹 The fourth option listed above appears to be a reasonable compromise ✨

6. Security Considerations

This specification provides a standard form for the use of Unicode as "network text" 🔐 Most of the same security issues that apply to UTF-8, as discussed in [RFC3629], apply to it too. It should be slightly less subject to some risks, though, because it requires NFC normalization and is generally somewhat more restrictive. However, shifts in Unicode versions, as discussed in Section 5.2, may introduce other security issues 😬

Programs that receive these streams should use extreme caution about assuming that incoming data are normalized, since it might be possible to use unnormalized forms, as well as invalid UTF-8, as part of an attack. In particular, firewalls and other systems that interpret UTF-8 streams should be developed with the clear knowledge that an attacker may deliberately send unnormalized text, for instance, to avoid detection by naive text-matching systems. Bestie, this is serious 💀

NVT contains a requirement, necessarily repeated here (see Section 2), that the CR character be immediately followed by either LF or ASCII NUL (an octet with all bits zero). NUL may be problematic for some programming languages that use it as a string terminator, and hence a trap for the unwary, unless caution is used. This may be an additional reason to avoid CR entirely, except in sequence with LF, as suggested above 🪤

The discussion about Unicode versions above (see Section 4 and Section 5.2) makes several assumptions about future versions of Unicode, about NFC normalization being applied properly, and about UTF-8 being processed and transmitted exactly as specified in RFC 3629. If any of those assumptions turn out to be incorrect, there are cases where strings that would be considered equivalent don't compare equal. Robust code should be prepared for those possibilities 🫶💜

7. Acknowledgments 🫶

Huge thanks to Mark Davis, Martin Duerst, and Michel Suignard for suggestions about Unicode normalization that led to the format described here, and especially to Mark for providing the paragraphs that describe the role of NFC. 💖 Thanks also to Mark, Doug Ewell, Asmus Freytag for corrected text describing Unicode transmission forms, and to Tim Bray, Carsten Bormann, Stephane Bortzmeyer, Martin Duerst, Frank Ellermann, Clive D.W. Feather, Ted Hardie, Bjoern Hoehrmann, Alfred Hoenes, Kent Karlsson, Bill McQuillan, George Michaelson, Chris Newman, and Marcos Sanz for a number of helpful comments and clarification requests. ✨

Appendix A. History and Context 🦋

Okay so this subsection is a little history lesson about how the ARPANET and the early Internet tried to agree on a standard "text" type. 📚 It sets up the context and motivation for the approach taken in this document. It's explanatory rather than normative, so nothing in here is meant to change or update any current specification. If history isn't your thing, you can totally skip it, no judgment. 💅

One of the earlier application design decisions made in the development of ARPANET, and one that got carried forward into the Internet, was to standardize on a single and very specific coding for "text" to be passed across the network [RFC0020]. 🌸 Hosts were then responsible for translating or mapping from whatever character coding conventions they used locally to that common intermediate representation. Sending hosts mapped to it, and receiving hosts mapped from it to their local forms as needed. 🔁

Here's the wild part: at the time, participating host operating systems used at least three different character coding standards. 😭 There was the antiquated BCD (Binary Coded Decimal), the then-dominant major manufacturer-backed EBCDIC (Extended BCD Interchange Code), and the then-still emerging ASCII (American Standard Code for Information Interchange). Since the ARPANET was an "open" project and EBCDIC was intimately linked to a particular hardware vendor, the original Network Working Group agreed that its standard should be ASCII. 🎀 That ASCII form was precisely "7-bit ASCII in an 8-bit field", which was basically a compromise between hosts that were natively 7-bit oriented (for example, with five seven-bit characters in a 36-bit word), those that were 8-bit oriented (using eight-bit characters), and those that placed the seven-bit ASCII characters in 9-bit fields with two leading zero bits (four characters in a 36-bit word). It's giving compromise energy and I'm here for it. 🧚

More standardization was suggested in the first preliminary description of the Telnet protocol [RFC0097]. 🌈 With the iterations of that protocol [RFC0137] [RFC0139], and the drawing together of an essentially formal definition somewhat later [RFC0318], a standard abstraction called the Network Virtual Terminal (NVT) was established. NVT character-coding conventions (initially called "Telnet ASCII" and later called "NVT ASCII", or, more casually, "network ASCII") included the requirement that Carriage Return followed by Line Feed (CRLF) be the common representation for ending lines of text. This was needed because some participating "Host" operating systems used one natively, some used the other, at least one used both, and a few used neither (preferring variable-length lines with counts or special delimiters or markers instead). NVT also specified conventions for some other characters. 🍓 Also, since NVT ASCII was restricted to seven-bit characters, use of the high-order bit in octets was reserved for the transmission of control signaling information. Clever, right? 👑

At a very high level, the concept was that a system could use whatever character coding and line representations were appropriate locally, but text transmitted over the network as text had to conform to the single "network virtual terminal" convention. Virtually all early Internet protocols that presume transfer of "text" assume this virtual terminal model, although different ones assume or limit it in different ways. Telnet, the command stream and ASCII Type in FTP [RFC0542], the message stream in SMTP transfer [RFC2821], and the strings passed to finger [RFC0742] and whois [RFC0954] are the classic examples. 💜 More recently, HTTP [RFC1945] [RFC2616] follows the same general model but permits 8-bit data and leaves the line end sequence unspecified. (The latter has been the source of a significant number of problems. Honestly, same. 😭)

Appendix B. The ASCII NVT Definition 🌸

Okay, so the main body of this specification is meant to be an update to, and an internationalized version of, the Net-ASCII definition. 🌷 It's self-contained, in that the parts of the Net-ASCII definition that are no longer recommended simply aren't included above. Net-ASCII evolved a bit over time, and there's been debate about which specification is the "official" Net-ASCII, so it's worth reviewing the key elements of that definition here. 🔍 This review is informal about the contents of Net-ASCII, and it shouldn't be considered a normative update or summary of the earlier specifications. (Section 2 does specify some normative updates to those specifications, and some comments below are consistent with it. ✨)

The first part of the section titled "THE NVT PRINTER AND KEYBOARD" in RFC 854 [RFC0854] is generally, though not universally, considered to be the normative definition of the (ASCII) Network Virtual Terminal, and hence of Net-ASCII. 💅 It includes not only the graphic ASCII characters but also a number of control characters. The latter get Internet-specific meanings that are often more specific than the definitions in the ASCII specification. In today's usage, and for the present specification, the following clarifications and updates to that list should be noted. Each one comes with a brief explanation of why the original specification is no longer appropriate. 🦋

  1. The "defined but not required" codes -- BEL (U+0007), BS (U+0008), HT (U+0009), VT (U+000B), and FF (U+000C) -- and the undefined control codes ("C0") SHOULD NOT be used unless required by exceptional circumstances. 🙈 Either their original "network printer" definitions are no longer in general use, common practice has evolved away from the formats specified there, or their use to simulate characters that are better handled by Unicode is no longer appropriate. While the appearance of some of these characters on the list may seem surprising, BS now has an ambiguous interpretation in practice (erasing in some systems but not in others), the width associated with HT varies with the environment, and VT and FF do not have a uniform effect with regard to either vertical positioning or the associated horizontal position result. Of course, telnet escapes aren't considered part of the data stream, so they're unaffected by this provision. 🧚
  2. In Net-ASCII, CR MUST NOT appear except when immediately followed by either NUL or LF, with the latter (CR LF) designating the "new line" function. 💖 Today, and as specified above, CR should generally appear only when followed by LF. Page layout is better done in other ways, NUL has a special interpretation in some programming languages, and avoiding other types of confusion matters too. So CR NUL should preferably be avoided, as specified above. 🌈
  3. LF CR SHOULD NOT appear except as a side-effect of multiple CR LF sequences (e.g., CR LF CR LF). 🍓
  4. The historical NVT documents don't call out either "bare LF" (LF without CR) or HT for special treatment. Both have generally been understood to be problematic. 😭 In the case of LF, there's a difference in interpretation as to whether its semantics imply "go to same position on the next line" or "go to the first position on the next line", and interoperability considerations suggest not depending on which interpretation the receiver applies. At the same time, misinterpreting LF is less harmful than misinterpreting "bare" CR. In the CR case, text may be erased or made completely unreadable. In the LF case, the worst consequence is a very funny-looking display. 🤭 Obviously, HT is problematic because there's no standard way to transmit intended tab position or width information in running text. Again, the harm is unlikely to be great if HT is simply interpreted as one or more spaces, but, in general, it can't be relied upon to format information. 👑

It's worth noting that the telnet IAC character (an octet consisting of all ones, i.e., %xFF) itself isn't a problem for UTF-8, since that particular octet can't appear in a valid UTF-8 string. 🎀 However, while few of them have been used, telnet permits other command-introducer characters whose bit sequences in an octet may be part of valid UTF-8 characters. While that causes no ambiguity in UTF-8, Unicode assigns a graphic character ("Latin Small Letter Y with Diaeresis") to U+00FF (octets C3 B0 in UTF-8). So some caution is clearly in order in this area. ⚠️ Be careful, bestie. 🫶

Appendix C. The Line-Ending Problem 🌊

The definition of how a line ending should be denoted in plain text strings on the wire for the Internet has been controversial since before NVT even existed. 😤 Some have argued that recipients should be required to interpret almost anything a sender might intend as a line ending as actually being a line ending. Others have pointed out that this would lead to ambiguities of interpretation and presentation, and would violate the principle that we should minimize the number of forms permitted on the wire, in order to promote interoperability and eliminate the "every recipient needs to understand every sender format" problem. 🙅‍♀️ The design of this specification, like that of NVT, takes the latter approach. Its designers believe there's little point in a standard if it just says "anyone can do whatever they like and the receiver just needs to cope". Respectfully, that's not a standard, that's a vibe. 💀

A further discussion of the nature and evolution of the line-ending problem appears in Section 5.8 of the Unicode Standard [Unicode], and it's suggested as additional reading. 📖 If we were starting with the Internet today, it would probably be sensible to follow the recommendation there and use LS (U+2028) exclusively, in preference to CRLF. 🦋 However, the installed base of CRLF use, and the importance of forward compatibility with NVT and protocols that assume it, makes that impossible. So it's necessary to keep using CRLF as the "New Line Function" ("NLF", see the terminology section in that reference). 🌸

Consideration should be given to a Telnet (or SSH [RFC4251]) option to specify this type of stream, and to an FTP extension [RFC0959] to permit a new "Unicode text" data TYPE. 💜 Future bestie projects, anyone? 🧚✨

References 📚

References 📚

Okay bestie, this is the bibliography, and it's where all the receipts live 🧾✨ Normative references are the ones you genuinely need to read to implement this. Informative ones are just good background reading 🤓

Normative References

  • [ISO10646] International Organization for Standardization, "Information Technology - Universal Multiple-Octet Coded Character Set (UCS) - Part 1: Architecture and Basic Multilingual Plane", ISO/IEC 10646-1:2000, October 2000. 🌍
  • [NFC] Davis, M. and M. Duerst, "Unicode Standard Annex #15: Unicode Normalization Forms", October 2006, <http://www.unicode.org/reports/tr15/>. 💅
  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 🔑
  • [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003. 🎀
  • [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, January 2008. 🧩
  • [Unicode] The Unicode Consortium, "The Unicode Standard, Version 5.0", 2007.

Boston, MA, USA: Addison-Wesley. ISBN 0-321-48091-0 📖

  • [Unicode32] The Unicode Consortium, "The Unicode Standard, Version 3.0", 2000.

(Reading, MA, Addison-Wesley, 2000. ISBN 0-201-61633-5). Version 3.2 consists of the definition in that book as amended by the Unicode Standard Annex #27: Unicode 3.1 (http://www.unicode.org/reports/tr27/) and by the Unicode Standard Annex #28: Unicode 3.2 (http://www.unicode.org/reports/tr28/). 🦋

Informative References

  • [ASCII] American National Standards Institute (formerly United States of America Standards Institute), "USA Code for Information Interchange", ANSI X3.4-1968, 1968.

ANSI X3.4-1968 has been replaced by newer versions with slight modifications, but the 1968 version remains definitive for the Internet. ISO 646 International Reverence Version (IRV) [ISO.646.1991] is usually considered equivalent to ASCII. 🕰️

  • [ISO.646.1991] International Organization for Standardization, "Information technology - ISO 7-bit coded character set for information interchange", ISO Standard 646, 1991.
  • [NamedSequences] The Unicode Consortium, "NamedSequences-4.1.0.txt", 2005, <http://www.unicode.org/Public/UNIDATA/NamedSequences.txt>. 🗂️
  • [RFC0020] Cerf, V., "ASCII format for network interchange", RFC 20, October 1969. 🌱
  • [RFC0097] Melvin, J. and R. Watson, "First Cut at a Proposed Telnet Protocol", RFC 97, February 1971. 🐣
  • [RFC0137] O'Sullivan, T., "Telnet Protocol - a proposed document", RFC 137, April 1971.
  • [RFC0139] O'Sullivan, T., "Discussion of Telnet Protocol", RFC 139, May 1971.
  • [RFC0318] Postel, J., "Telnet Protocols", RFC 318, April 1972.
  • [RFC0542] Neigus, N., "File Transfer Protocol", RFC 542, August 1973.
  • [RFC0698] Mock, T., "Telnet extended ASCII option", RFC 698, July 1975.
  • [RFC0742] Harrenstien, K., "NAME/FINGER Protocol", RFC 742, December 1977.
  • [RFC0854] Postel, J. and J. Reynolds, "Telnet Protocol Specification", STD 8, RFC 854, May 1983.
  • [RFC0954] Harrenstien, K., Stahl, M., and E. Feinler, "NICNAME/WHOIS", RFC 954, October 1985.
  • [RFC0959] Postel, J. and J. Reynolds, "File Transfer Protocol", STD 9, RFC 959, October 1985.
  • [RFC1945] Berners-Lee, T., Fielding, R., and H. Nielsen, "Hypertext Transfer Protocol -- HTTP/1.0", RFC 1945, May 1996. 🌐
  • [RFC2277] Alvestrand, H., "IETF Policy on Character Sets and Languages", BCP 18, RFC 2277, January 1998.
  • [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
  • [RFC2781] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of ISO 10646", RFC 2781, February 2000.
  • [RFC2821] Klensin, J., "Simple Mail Transfer Protocol", RFC 2821, April 2001. 📬
  • [RFC3454] Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, December 2002.
  • [RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)", RFC 3491, March 2003.
  • [RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912, September 2004.
  • [RFC4251] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) Protocol Architecture", RFC 4251, January 2006. 🔐
  • [RFC4690] Klensin, J., Faltstrom, P., Karp, C., and IAB, "Review and Recommendations for Internationalized Domain Names (IDNs)", RFC 4690, September 2006.

Authors' Addresses 📮

Here are the folks who wrote this, with their mailing details, so you know who to thank (or yell at) 💌

John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA  02140
USA

Phone: +1 617 491 5735
EMail: john-ietf@jck.com


Michael A. Padlipsky
8011 Stewart Ave.
Los Angeles, CA  90045
USA

Phone: +1 310-670-4288
EMail: the.map@alum.mit.edu

Copyright (C) The IETF Trust (2008). 🌸

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

The legal bit is next, and it's serious, so I'm keeping it word for word:

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🦄

Okay, the IETF is being very careful here, and honestly that's giving responsible adult energy 🫶 The IETF takes no position on whether any Intellectual Property Rights or other rights that might be claimed to apply to the technology in this document are valid or in scope. It also doesn't say whether any license under those rights might be available, and it hasn't made any independent effort to find out if such rights exist. If you want the procedures for rights in RFC documents, check BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses that will be made available, and the results of any attempt to get a general license or permission for implementers or users of this specification can all be found in the IETF's online IPR repository at http://www.ietf.org/ipr. 📂

And if you spot any copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to implement this standard, please tell the IETF at ietf-ipr@ietf.org. Speak up, queen 👑💖

📦 where this page comes from