utf-8-codec(3scm)

NAME 💖

utf-8-codec, utf-16-codec, latin-1-codec — predefined codecs. This page is from the Scheme Programmer's Manual, and it's lowkey iconic. 🦋

LIBRARY 📚

(import (rnrs))                     ;R6RS
(import (rnrs io ports))            ;R6RS

SYNOPSIS 🛠

(utf-8-codec)
(utf-16-codec)
(latin-1-codec)

DESCRIPTION ✨

These are the predefined codecs for the ISO 8859-1, UTF-8, and UTF-16 encoding schemes. Three codecs, one slay. 💅

IMPLEMENTATION NOTES 🧠

The latin-1 codec 🍓

The latin-1 codec can only handle characters in the range [#\x00, #\xFF]. The transcoding is a one-to-one mapping between the byte values and character objects in the sense of char->integer(3scm) and integer->char(3scm). No cap. 💖

When transcoding in the input direction, each byte is read as one character.

When transcoding in the output direction, characters less than #\x100 are written as one byte. Otherwise the behavior depends on the transcoder's error handling mode. In raise mode an &i/o-encoding(7scm) condition is raised; in ignore mode the character is dropped; and in replace mode a \x3F byte (#\?) is written. That's the whole tea. 🫖

The UTF-8 codec 🌈

The UTF-8 transcoder can handle all Unicode characters. For characters in the range [#\x00, #\x7F] there is a one-to-one mapping between byte values and character objects in the sense of char->integer(3scm) and integer->char(3scm). It's giving "simple and clean." ✨

For characters above #\x7F, UTF-8 tags the initial byte with a length and then follows that byte with a number of continuation bytes. There is a specific pattern that must be followed and any deviation is an error. The same applies to short sequences. Errors are handled according to the transcoder's error handling mode. In raise mode an &i/o-decoding(7scm) condition is raised; in ignore mode the character is dropped; and in replace mode an #\xFFFD character is read. 🫢

Implementations vary in how many #\xFFFD characters are produced for an invalid sequence. So don't @ me if the count is different, bestie. 🙈

There are no encoding errors with the UTF-8 codec. Periodt. 💀

The UTF-16 codec 🎀

The UTF-16 codec works with pairs of bytes. The bytes are 16-bit quantities encoded in either big endian or little endian. A byte order mark (BOM) is either the byte sequence #xFE, #xFF for big endian or #xFF, #xFE for little endian. Endianness is literally the main character here. 👑

The 16-bit quantities mostly map directly to character objects via integer->char(3scm). There is an exception for the range [#xD800, #xDFFF], which are called the surrogate pairs. A pair of these 16-bit quantities are used to encode characters that would otherwise not fit in the 16-bit range.

Short byte sequences and errors in surrogate pairs are errors. In raise mode an &i/o-decoding(7scm) condition is raised; in ignore mode the character is dropped; and in replace mode an #\xFFFD character is read. 🫶

There are no encoding errors with the UTF-16 codec. Slay. 💅

RETURN VALUES 🎁

A call to any of these procedures returns a value that is equal in the sense of eqv?(3scm) to the result of any other call to the same procedure. Same procedure, same eqv? vibe, fr fr. 💖

EXAMPLES 🧪

This one is like utf8->string, bestie:

;; This is like utf8->string
(let* ((p (open-bytevector-input-port #vu8(194 169)))
       (tc (make-transcoder (utf-8-codec)))
       (tp (transcoded-port p tc)))
  (get-string-all tp))
        => "©"

APPLICATION USAGE 🚀

These procedures are used when creating transcoders. That's the whole use case, and it ate. 🛠✨

ERRORS 😭

These procedures can raise exceptions with the following condition types:

• &assertion (R6RS) — the wrong number of arguments was passed. Not the vibe. 💀

SEE ALSO 👀

Go peek at make-transcoder(3scm) and error-handling-mode(7scm). They're in the group chat too. 💬

STANDARDS 📏

R6RS

HISTORY 🕰

These procedures first appeared in R6RS as part of the reworked I/O system. History was made, no cap. ✨

AUTHORS 👑

This page is part of the scheme-manpages project. It includes materials from the RnRS documents. More information can be found at https://github.com/schemedoc/manpages/. Respect the source, bestie. 💜

---

Signed off on 2023-07-19 with the footer source/version vibe utf-8-codec(3scm) — that's the date and version energy. Bye bestie, go transcode something cute! 🫶💅🌈