UTF(7)

🌈 UTF — character set and format 💅

ok so this is the page for UTF, Unicode, ASCII, and rune — bestie, it's literally the whole character set and format situation, from the Plan 9 misc info manual. Buckle up 🎀

---

✨ NAME ✨

UTF, Unicode, ASCII, rune — character set and format. that's the whole squad 🫶

---

💖 DESCRIPTION 💖

okay so here's the tea ☕ — the Plan 9 character set and its representation are based on the Unicode Standard and on the ISO multibyte UTF-8 encoding. yes, that's Universal Character Set Transformation Format, 8 bits wide — the full government name, no cap 👑

so like, the Unicode Standard represents its characters in 16 bits, and UTF-8 represents such values in an 8-bit byte stream. cute, right? and throughout this manual, UTF-8 is shortened to UTF — just so we're not typing the whole thing every single time, iykyk 💅

---

🦋 what even is a rune tho

in Plan 9, a rune is a 16-bit quantity representing a Unicode character. internally, programs may store characters as runes. but — and this is the key part, bestie — any external manifestation of textual information, like in files or at the interface between programs, uses a machine-independent, byte-stream encoding called UTF. so runes are the inner-circle vibe, UTF is the public-facing main character energy ✨

---

🌸 the ASCII compatibility slay

UTF is designed so the 7-bit ASCII set (values hexadecimal 00 to 7F) appear only as themselves in the encoding. iconic behaviour honestly.

runes with values above 7F show up as sequences of two or more bytes with values only from 80 to FF.

the UTF encoding of the Unicode Standard is backward compatible with ASCII 🫶:

• programs presented only with ASCII work on Plan 9 even if not written to deal with UTF — periodt
• same goes for programs that deal with uninterpreted byte streams
• however (and this is the plot twist 😭), programs that perform semantic processing on ASCII graphic characters must convert from UTF to runes in order to work properly with non-ASCII input. see rune(3) 💜

---

🧚 the conversion math (yes we're doing binary, stay with me)

letting numbers be binary, a rune x is converted to a multibyte UTF sequence as follows:

01.   x in [00000000.0bbbbbbb] → 0bbbbbbb
10.   x in [00000bbb.bbbbbbbb] → 110bbbbb, 10bbbbbb
11.   x in [bbbbbbbb.bbbbbbbb] → 1110bbbb, 10bbbbbb, 10bbbbbb

conversion 01 provides a one-byte sequence that spans the ASCII character set in a compatible way 💖

conversions 10 and 11 represent higher-valued characters as sequences of two or three bytes with the high bit set.

🚨 important vibe check: Plan 9 does not support the 4, 5, and 6 byte sequences proposed by X-Open. not invited to the party, sorry not sorry 💀

when there are multiple ways to encode a value — for example rune 0 — the shortest encoding is used. efficiency queen 👑

---

🙈 the inverse mapping (reverse uno card)

in the inverse mapping, any sequence except those described above is incorrect and is converted to rune hexadecimal 0080. wrong vibe = bounced straight to 0080, no questions asked 😭

---

🔥 SEE ALSO 🔥

• ascii(1)
• tcs(1)
• rune(3)
• The Unicode Standard.

go read those, bestie, they're literally so good 🍓

---

and that's the page!! this one's from the Plan 9 Miscellaneous Information Manual, filed under UTF(7) — footer and all, same UTF(7) energy at the bottom as at the top 🎀 mwah, ily, go encode something cute 💜🦋