RFC 373

ARBITRARY CHARACTER SETS ✨

• RFC: NWG/RFC #373 🎀 (also NIC 11058)
• Date: 14 July 1972 📅
• Site tag: SU-AI 🌸
• Author: John McCarthy 💅
• Category and status: not stated in the original 🤷‍♀️
• Obsoletes and updates: none listed 🧚

Okay bestie, so this memo is basically the dream. 🦋 Imagine storing documents with any character you want, then showing them on any screen, typing them on any keyboard, and printing them on any printer. Literally iconic. John McCarthy wants to figure out how we get from here to there, with a special focus on the ARPA network. 💖

Where are we now? 🌈

Let's check the vibe first:

1. There's currently a 96 character ASCII set, and everyone agrees it should be included in any bigger set. 💜
2. Lots of installations are stuck on 64 character sets that don't even include the lowercase latin alphabet. 😭
3. At the Stanford Artificial Intelligence Laboratory, they have a 114 character set that includes the 96 character ASCII. It's built into their keyboards, displays, and line printer. ✨
4. Printers are showing up that get their character designs out of memory, like the Xerox XGP printer, and they're getting one. 🖨️
5. The IMLAC type display keeps its character designs in main memory, so changing the displayed set is just reloading the memory. 🔥
6. Lots of display systems share one character generator among many display units. For some of them, like the Datadisc, arbitrary sets are probably doable (with some kludgery we'll get to later). For others, like their III's, arbitrary sets aren't feasible. 💀

One possible approach is to make a bigger standard set of characters, maybe using 8 or 9 bits, and expect new equipment to implement it. But honestly, this has a big problem: it'll be super hard to agree on what comes next, and even if everyone formally agrees, lots of groups will find it in their interest to just ignore the standard. 🙈

So the author's suggestion is that the next step should be arbitrary character sets. Here's how it could work:

1. Set up a registry of characters. Anyone can register a new character. Each character gets a unique number. 17 bits should be enough, even to include Chinese 🧧. Each character also gets a name in ASCII, usually mnemonic. Finally, each character has a design, which is a picture on a 50 by 50 dot matrix. 🎨
2. Alongside the character registry, there's a registry of character sets, which different groups use for different classes of documents. A registered character set has a registry number and a table that maps the character codes (as bit sequences) to the registered character numbers. 📚
3. Each document comes with a statement of the character code it uses. This can be one of the registered codes, or it can include extra modifications described by an auxiliary table that gives the code correspondence with registered character numbers. A character code may have an escape character that says the next character is described by its registry number. The statement of the character code can be a header on the document, or the receiver may have to figure it out some other way, like from its library catalog entry. 🗂️
4. Devices like printers and displays draw characters in different ways, and standardizing that doesn't seem doable right now. So we need a way to go from the standard 50 by 50 dot matrix description of a character to whatever method the device uses. That's up to the programmers supporting the device. Some might manually create files describing how registered characters get implemented. They might find it's too much work to cover every character and keep up with new registrations. Others will write programs that go from the registered descriptions to descriptions their implementations can use. Most people will probably hand tailor the characters they use most and write a program for the rest. 🧚‍♀️
5. The line printer is the easiest device to handle, because it's slow. At the start of a print job, the SPOOL program looks up the character set and loads the printer's memory with the character designs used in that document. Sometimes it may have to go through the network to one of the computers that stores the registry, to find out what to do. 🖨️💖
6. Display systems with a character memory for each display unit can be handled in about the same way. Users will sometimes notice delays when the display programs are surprised by unfamiliar characters. ⏳
7. Display systems that share character memories need more complicated treatment. The goal is to keep the memory large enough to hold all the characters the current set of users is using, and to handle the required table lookups from the different character codes in a nice way. There will be limits on how many different character sets can be in use at the same time. Systems like the Datadisc, which only look up a character when it's first written, can be extended to work with large sets. Systems that have to look up each character code 30 times per second to maintain the display won't work so well. 😬

The author doesn't have any special ideas about making keyboards adaptable to arbitrary sets. Each user may have to fend for themselves. 💅

Typography 🎀

So far, I've ignored typography, which is the fact that in printed documents the same letter can be printed in many fonts. Maybe each character in each font will need its own registered description, with a constant difference between the numbers of the same character in different fonts. Installations will again have to decide which font distinctions they'll implement. 🔤

Some other issues to think about are whether we can automatically adapt texts to the line and page lengths of different devices. 📄

The author thinks the typographical problems probably can't be solved right now. It would be best to adopt conventions for registering character designs now, and leave typography for later. 🦋

Closing thoughts 🌸

In the author's opinion, there's no real obstacle to setting up the registry on the ARPA network now, getting the standards organization to work, and being able to exchange documents in extended character sets as soon as the installations can get the printers and display devices. ✨

And the Stanford Artificial Intelligence Laboratory's current policy is to acquire no more devices that are tied to fixed character sets. That's a whole mood. 💜🫶

       [ This RFC was put into machine readable form for entry ]
       [ into the online RFC archives by BBN Corp. under the   ]
       [ direction of Alex McKenzie.                      1/97 ]