RFC 1843

RFC 1843 💖 HZ - A Data Format for Exchanging Files of Arbitrarily Mixed Chinese and ASCII characters

Okay bestie, this one is from F. Lee at Stanford University 🎓. It's an Informational doc from August 1995, so it's giving info and not a formal Internet standard 🎀. The title block doesn't list anything it obsoletes or updates, so it's a solo icon with no drama 💅.

---

Status of This Memo 🌸

This memo gives info to the Internet community ✨. It does not specify an Internet standard of any kind, so no pressure 🙈. Distribution of this memo is unlimited 🦋.

Abstract 💜

Okay so this is basically the same as an article with the same title that the author wrote on September 4, 1989 📝. In this memo, GB means GB2312-80 🇨🇳. The title is only kept for historical reasons, and HZ has been used way more for stuff beyond "file exchange" than the name suggests 🌈.

1. Introduction ✨

Most computer systems that can handle a text file with arbitrarily mixed Chinese and ASCII characters use 8-bit codes 💻. So if you want to send those files through email on ASCII systems, you have to encode them into a 7-bit format 📨. A generic binary-to-ASCII encoder isn't enough, though, because there's no universal standard for those 8-bit codes yet 😭. CCDOS and the Macintosh's Chinese OS, for example, use different internal codes 🍓. The good news is that there's a PRC national standard, GuoBiao (GB), for encoding Chinese characters, and characters from those systems can be easily converted to GB with a simple formula 🫶. (* The ROC standard BIG-5 is outside the scope of this article.)

HZ is a 7-bit data format built for exchanging text files that mix GB and ASCII 💜. It's also meant for designing terminal emulators that show and edit mixed Chinese and ASCII text files in real time 🖥️✨.

---

2. Specification 🎀

Here's how HZ works 👑.

Without loss of generality, we assume all Chinese characters (HanZi) have already been encoded in GB 🇨🇳. A GB code (GB1 and GB2) is a two byte code. The first byte is in the range $21-$77 (hexadecimal), and the second byte is in the range $21-$7E 🔢.

A graphical ASCII character is a byte in the range $21-$7E 🔤. A non-graphical ASCII character is a byte in the range $0-$20 or the value $7F 🙃.

The range of a graphical ASCII character overlaps with a GB byte, so a byte in $21-$7E gets interpreted based on which mode you're in 🧠. There are two modes: ASCII mode and GB mode 🌗.

By convention, a non-graphical ASCII character should only appear in ASCII mode 🙈.

The default mode is ASCII mode 💅.

In ASCII mode, a byte is read as an ASCII character unless you run into a ~, which is the escape character 🐚. By convention, it must be immediately followed ONLY by ~, { or \n (<LF>), and each one has a special meaning:

• ~~ is interpreted as a ~ 💫.
• The escape-to-GB sequence ~{ switches the mode from ASCII to GB 🔀.
• The escape sequence ~\n is a line-continuation marker. It gets consumed and produces no output 🫧.

In GB mode, characters are read two bytes at a time as (pure) GB codes until the escape-from-GB code ~} shows up 🧩. That code switches the mode from GB back to ASCII 🔁. (Note that the escape-from-GB code ~} ($7E7D) is outside the defined GB range 👀.)

The decoding process is clear from the above 🫶.

Encoding is pretty straightforward too 🍰. An (ASCII) ~ is always encoded as ~~ 🙌. A sequence of GB codes gets wrapped in ~{ and ~} 🎁.

---

3. Remarks & Recommendations 🔍

We choose to encode any ASCII character except ~ as-is, rather than as a two byte code. We also pick ASCII as the default mode, and here's why 💖. The computers we used were ASCII based 💻. A HZ file with pure ASCII characters (i.e. no Chinese characters) except for ~ is exactly a pure ASCII file 📄. Overall, the English (ASCII) part of a HZ file is directly readable 👀✨.

The escape character ~ was picked because it's commonly used in the ASCII world 🌍. It's also outside the defined range ($21-$77) of the first byte of a GB code, since ~ ($7E) sits outside that range 🧐.

In ASCII mode, any other two byte sequences that start with ~ (other than ~~, ~{ and ~\n) are currently invalid HZ sequences 🚫. That means they can be used to extend HZ later with total upward compatibility 🦋.

The line-continuation marker ~\n is super handy if you want to split long lines of the original text into shorter lines in this format, without adding extra newline characters when decoding 📏.

There's no limit on the length of a line 🌊. The whole file could even be one long line, or have no newline characters at all 🤯. A decoder for this HZ format shouldn't, and doesn't need to, think in terms of lines 🙅‍♀️.

Encoders and decoders for HZ are easy to write 🧚. An encoder or decoder only needs to look ahead at most one character in the input stream 👀.

Given the current mode, you can also decode a HZ data stream by scanning backward, which is easy and totally doable 🔙. One implication is that "backspaces" can be handled correctly by a terminal emulator ⌫💅.

To make it easier to use programs that support line and page skips, like "more" on UNIX, with a terminal emulator that understands HZ, it is RECOMMENDED that the ENCODER (which outputs HZ) sets a maximum line size of less than 80 characters 📐. Since \n is an ASCII character, the HZ syntax then automatically means that GB codes at the end of a line must be closed with the escape-from-GB code ~}, and the line-continuation marker ~\n should be inserted where it fits 🧵. The tradeoff is that the encoded file is a little bigger 📦.

Now, this part is important 🚨. The recommendation above does NOT change the HZ format at all 🙅‍♀️. It's just an encoding "style" that follows the HZ syntax 🎨. That style isn't built into HZ; it's an extra convention layered "on top of" HZ 🍰. Other applications may need different "styles", but the same basic HZ DECODER will always work 💯. The whole point of HZ is to be a flexible basic data format for files that mix Chinese and ASCII characters, and that's exactly what it does 👑✨.

---

4. Examples 🧁

To show the "stylistic" side of HZ encoding, here are four examples of encoded text that should all decode to the same output. (The recommendation in the last section refers to Example 2 💅.)

Heads up, bestie: the original says four examples but only three actually show up below 💀. I'm just flagging it so you're not confused.

Example 1: (Suppose there is no line size limit.)

This sentence is in ASCII.
The next sentence is in GB.~{<:Ky2;S{#,NpJ)l6HK!#~}Bye.

Example 2: (Suppose the maximum line size is 42.)

This sentence is in ASCII.
The next sentence is in GB.~{<:Ky2;S{#,~}~
~{NpJ)l6HK!#~}Bye.

Example 3: (Suppose a new line is started whenever there is a mode switch.)

This sentence is in ASCII.
The next sentence is in GB.~
~{<:Ky2;S{#,NpJ)l6HK!#~}~
Bye.

---

Acknowledgement 🫶

Edmund Lai was the first person to bring this topic to the author's attention 🙌. Discussions with Ed, Tin-Fook Ngai, Yagui Wei and Ricky Yeung were super helpful in shaping the ideas in this article 💜. A big thank you to Tin-Fook for his careful review of the draft and his many interesting suggestions 🌸.

References 📚

[1] Fung Fung Lee, "HZ - A Data Format for Exchanging Files of Arbitrarily Mixed Chinese and ASCII Characters," September 4, 1989. As part of //ftp.ifcss.org/software/unix/convert/HZ-2.0.tar.gz

Security Considerations 🔥

Security issues are not addressed in this memo. 💀 Honestly, that's a big deal, so don't assume this doc covers it.

Author's Address 💅

• Fung Fung Lee
• Computer Systems Laboratory
• Stanford University
• Stanford, CA 94309
• Phone: +1 415 723 1450
• EMail: lee@csl.stanford.edu