encoding(3tcl)

👑 NAME 👑

encoding — manipulate encodings, bestie 💅

(this is the encoding(3tcl) page, straight outta Tcl Built-In Commands, okurrr)

---

✨ SYNOPSIS ✨

encoding option ?arg arg ...?

---

🦋 INTRODUCTION 🦋

Okay so, strings in Tcl? Logically they're a sequence of Unicode characters. Periodt. 💖

But in memory, they're actually represented as a sequence of bytes, and those bytes can be in one of several encodings: modified UTF-8 (which uses 1 to 4 bytes per character), or a custom encoding start as 8 bit binary data. ikyk.

Different operating system interfaces or applications may generate strings in other encodings, like Shift-JIS. And that's literally where encoding comes in — it helps bridge the gap between Unicode and these other formats. no cap. 🌉

---

💖 DESCRIPTION 💖

It performs one of several encoding-related operations, depending on option. The legal options are:

• encoding convertfrom ?encoding? data
• encoding convertfrom ?-profile profile? ?-failindex var? encoding data — converts data, which should be in binary string encoded as per encoding, into a Tcl string. If encoding isn't specified, the current system encoding gets used. 🫶

The -profile option determines what the command does when conversion errors show up. Check the PROFILES section below for the deets. Any premature termination of processing due to errors is reported through an exception if the -failindex option is not specified.

If the -failindex is specified, then instead of an exception being raised on premature termination, the result of the conversion up to the point of the error is returned as the result of the command. Plus, the index of the source byte that triggered the error gets stored in var. And if no errors are encountered, the entire result of the conversion is returned and the value -1 is stored in var. cute, right? 💅

• encoding convertto ?encoding? data
• encoding convertto ?-profile profile? ?-failindex var? encoding data — converts a string to the specified encoding. The result is a Tcl binary string containing the sequence of bytes that represent the converted string in the specified encoding. If encoding isn't specified, the current system encoding is used. ✨

The -profile and -failindex options do the exact same thing as described for the encoding convertfrom command.

• encoding dirs ?directoryList? — Tcl can load encoding data files from the file system that describe extra encodings for it to work with. This command sets the search path for *.enc encoding data files to the list of directories directoryList. If directoryList is omitted, the command returns the current list of directories that make up the search path. It's an error for directoryList to not be a valid list. And if, while a search for an encoding data file is happening, an element in directoryList does not refer to a readable, searchable directory — that element gets ignored. rude but ok 🙈
• encoding names — returns a list containing the names of all the encodings that are currently available. The encodings "utf-8" and "iso8859-1" are guaranteed to be present in that list. they're the main characters fr fr 👑
• encoding profiles — returns a list of the names of encoding profiles. See PROFILES below.
• encoding system ?encoding? — sets the system encoding to encoding. If encoding is omitted, the command returns the current system encoding. The system encoding is used whenever Tcl passes strings to system calls. 🛠️
• encoding user — returns the name of the encoding as per the user's preferences. On Windows systems, this is based on the user's code page settings in the registry. On other platforms, the returned value is the same as what encoding system returns. 💻

---

🌈 PROFILES 🌈

Operations involving encoding transforms may run into several types of errors — like invalid sequences in the source data, characters that cannot be encoded in the target encoding, and so on. A profile prescribes the strategy for dealing with those errors, in one of two ways:

• Terminating further processing of the source data. The profile does not determine how this premature termination is conveyed to the caller. By default, this is signalled by raising an exception. If the -failindex option is specified, errors get reported through that mechanism instead.
• Continue further processing of the source data using a fallback strategy, like replacing or discarding the offending bytes in a profile-defined manner. 🎀

The following profiles are currently implemented, with strict being the default if the -profile isn't specified.

strict 💀

The strict profile always stops processing when a conversion error is encountered. The error is signalled via an exception or the -failindex option mechanism. The strict profile implements a Unicode standard conformant behavior. no shortcuts, no vibes, just law. ⚖️

tcl8 🕰️

The tcl8 profile always follows the first strategy above, and it corresponds to the behavior of encoding transforms in Tcl 8.6.

When converting from an external encoding other than utf-8 to Tcl strings with the encoding convertfrom command, invalid bytes are mapped to their numerically equivalent code points. For example, the byte 0x80 which is invalid in ASCII would be mapped to code point U+0080. When converting from utf-8, invalid bytes that are defined in CP1252 are mapped to their Unicode equivalents, while those that are not fall back to the numerical equivalents. For example, byte 0x80 is defined by CP1252 and is therefore mapped to its Unicode equivalent U+20AC, while byte 0x81 — which is not defined by CP1252 — is mapped to U+0081. As an additional special case, the sequence 0xC0 0x80 is mapped to U+0000. 🤯

When converting from Tcl strings to an external encoding format using encoding convertto, characters that cannot be represented in the target encoding are replaced by an encoding-dependent character, usually the question mark ?.

replace 🔁

Like the tcl8 profile, the replace profile always continues processing on conversion errors — but it follows a Unicode standard conformant method for substitution of invalid source data.

• When converting an encoded byte sequence to a Tcl string using encoding convertfrom, invalid bytes are replaced by the U+FFFD REPLACEMENT CHARACTER code point.
• When encoding a Tcl string with encoding convertto, code points that cannot be represented in the target encoding are transformed to an encoding-specific fallback character: U+FFFD REPLACEMENT CHARACTER for UTF targets, and generally ? for other encodings.

---

🍓 EXAMPLES 🍓

These examples use the utility proc below that prints the Unicode code points comprising a Tcl string.

proc codepoints s {join [lmap c [split $s {}] {
    string cat U+ [format %.6X [scan $c %c]]}]
}

Example 1: convert a byte sequence in Japanese euc-jp encoding to a TCL string:

% codepoints [encoding convertfrom euc-jp "\xA4\xCF"]
U+00306F

The result is the unicode codepoint "\u306F", which is the Hiragana letter HA. 🌸

Example 2: Error handling based on profiles:

The letter A is Unicode character U+0041 and the byte "\x80" is invalid in ASCII encoding.

% codepoints [encoding convertfrom -profile tcl8 ascii A\x80]
U+000041 U+000080
% codepoints [encoding convertfrom -profile replace ascii A\x80]
U+000041 U+00FFFD
% codepoints [encoding convertfrom -profile strict ascii A\x80]
unexpected byte sequence starting at index 1: '\x80'

Example 3: Get partial data and the error location:

% codepoints [encoding convertfrom -profile strict -failindex idx ascii AB\x80]
U+000041 U+000042
% set idx
2

Example 4: Encode a character that is not representable in ISO8859-1:

% encoding convertto iso8859-1 A\u0141
A?
% encoding convertto -profile strict iso8859-1 A\u0141
unexpected character at index 1: 'U+000141'
% encoding convertto -profile strict -failindex idx iso8859-1 A\u0141
A
% set idx
1

---

👀 SEE ALSO 👀

Tcl_GetEncoding(3tcl), fconfigure(3tcl)

---

💜 KEYWORDS 💜

encoding, unicode

---

okay bestie that's the whole page, ily, go forth and encode responsibly 💖✨

This was Tcl 8.1, page encoding(3tcl) — mwah, slay, bye 🎀🦋💅