Tcl_GetEncoding(3tcl)

📦

version 9.0.4 · 3 versions

9.0.4 · ubuntu stonking (latest)
9.0.3 · debian trixie, ubuntu questing, ubuntu resolute
8.6.18 · debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

🌸 Tcl_GetEncoding(3tcl) — Tcl Library Procedures 🌸

okay bestie, sit down, hydrate, because this is the manual page for Tcl_GetEncoding and friends, straight out of the Tcl Library Procedures section of the manual. it's a whole little universe of encoding procedures and ngl it's kind of iconic. this is part 1 of 4, so we're just doing the intro bits: the names, the synopsis, and every single argument. buckle up 💅

✨ NAME ✨

meet the whole squad. these are the procedures for creating and using encodings, and yes, they all show up in this one page:

Tcl_GetEncoding
Tcl_FreeEncoding
Tcl_GetEncodingFromObj
Tcl_ExternalToUtfDString
Tcl_ExternalToUtfDStringEx
Tcl_ExternalToUtf
Tcl_UtfToExternalDString
Tcl_UtfToExternalDStringEx
Tcl_UtfToExternal
Tcl_GetEncodingName
Tcl_SetSystemEncoding
Tcl_GetEncodingNameFromEnvironment
Tcl_GetEncodingNames
Tcl_CreateEncoding
Tcl_GetEncodingSearchPath
Tcl_SetEncodingSearchPath

— procedures for creating and using encodings —

like, that's the whole vibe. 💖

✨ SYNOPSIS ✨

first things first, you gotta pull in the header:

#include <tcl.h>

then here's the getting-and-freeing-your-encoding era:

Tcl_Encoding
Tcl_GetEncoding(interp, name)

Tcl_FreeEncoding(encoding)

int
Tcl_GetEncodingFromObj(interp, objPtr, encodingPtr)

the "DString" girlies that do the converting and hand you back a char *:

char *
Tcl_ExternalToUtfDString(encoding, src, srcLen, dstPtr)

int
Tcl_ExternalToUtfDStringEx(interp, encoding, src, srcLen, flags, dstPtr, errorIdxPtr)

char *
Tcl_UtfToExternalDString(encoding, src, srcLen, dstPtr)

int
Tcl_UtfToExternalDStringEx(interp, encoding, src, srcLen, flags, dstPtr, errorIdxPtr)

the big main-character conversions with the full param list, she's long but she's worth it:

int
Tcl_ExternalToUtf(interp, encoding, src, srcLen, flags, statePtr, dst, dstLen, srcReadPtr, dstWrotePtr, dstCharsPtr)

int
Tcl_UtfToExternal(interp, encoding, src, srcLen, flags, statePtr, dst, dstLen, srcReadPtr, dstWrotePtr, dstCharsPtr)

and the naming / system / search-path / creation department, aka the backstage crew:

const char *
Tcl_GetEncodingName(encoding)

Tcl_Size
Tcl_GetEncodingNulLength(encoding)

int
Tcl_SetSystemEncoding(interp, name)

const char *
Tcl_GetEncodingNameFromEnvironment(bufPtr)

Tcl_GetEncodingNames(interp)

Tcl_Encoding
Tcl_CreateEncoding(typePtr)

Tcl_Obj *
Tcl_GetEncodingSearchPath()

int
Tcl_SetEncodingSearchPath(searchPath)

yes queen, Tcl_GetEncodingNulLength really does show up in the synopsis. we love a surprise guest ⭐

✨ ARGUMENTS ✨

every single one, explained, no skips, no cuts. this is the main character energy section:

Tcl_Interp *interp (in) — the interpreter to use for error reporting, or NULL if you literally don't want error reporting at all. your call bestie 🫶
const char *name (in) — the name of the encoding to load. that's it, that's the whole job.
Tcl_Encoding encoding (in) — the encoding you're querying, freeing, or using to convert text. and here's the plot twist: if encoding is NULL, the current system encoding gets used instead.
Tcl_Obj *objPtr (in) — the name of the encoding you want to get a token for.
Tcl_Encoding *encodingPtr (out) — points to the storage where the encoding token is gonna be written. the output goes here, so be ready 📝
const char *src (in) — for the Tcl_ExternalToUtf functions, this is an array of bytes in the specified encoding that are to be converted to TUTF-8. for the Tcl_UtfToExternal function, it's a TUTF-8 byte sequence to be converted to the specified encoding.
const TCHAR *tsrc (in) — an array of Windows TCHAR characters to convert to TUTF-8. windows girlies, this one's for you.
Tcl_Size srcLen (in) — length of src or tsrc in bytes. if the length is negative, the encoding-specific length of the string is used instead. negative = "you figure it out, girl"
Tcl_DString *dstPtr (out) — pointer to an uninitialized or free Tcl_DString where the converted result will be stored. uninitialized or free, okay? don't come at me with a half-used one 💀
int flags (in) — this is a bit mask you pass in to control how the encoding functions operate. okay so, breaking it down:
- TCL_ENCODING_START means the source buffer is the first block in a (potentially multi-block) input stream. it tells the conversion routine to reset to an initial state and perform any initialization that needs to occur before the first byte is converted.
- TCL_ENCODING_END means the source buffer is the last block in a (potentially multi-block) input stream. it tells the conversion routine to perform any finalization that needs to occur after the last byte is converted, and then to reset to an initial state.
- the TCL_PROFILE_* bits defined in the PROFILES section below control the encoding profile to be used for dealing with invalid data or other errors in the encoding transform.
- the flag TCL_ENCODING_STOPONERROR has no effect — it only has meaning in Tcl 8.x. not me still being haunted by 8.x 😭
- some flags bits may not be usable with some functions, as noted in the function descriptions below. read the fine print, babes.
Tcl_EncodingState *statePtr (in/out) — used when converting a (generally long or indefinite length) byte stream in a piece-by-piece fashion. the conversion routine stores its current state in *statePtr after src (the buffer containing the current piece) has been converted; that state information must be passed back when converting the next piece of the stream, so the conversion routine knows what state it was in when it left off at the end of the last piece. it may be NULL, in which case the value specified for flags is ignored and the source buffer is assumed to contain the complete string to convert. (so like, if you're a one-and-done converter, NULL is fine, no drama 🌸)
char *dst (out) — the buffer where the converted result will be stored. no more than dstLen bytes will be stored in dst. respect the limit, periodt.
Tcl_Size dstLen (in) — the maximum length of the output buffer dst, in bytes. that's your ceiling, don't exceed her 📏

bestie we are SO not done, there's more coming — this was just the setup. keep scrolling for the rest 💖

int *srcReadPtr (out) — gets filled with the number of bytes from src that actually got converted. Bestie, this might be less than the original source length if something went sideways converting some of the source characters. Can be NULL. 🫶
int *dstWrotePtr (out) — gets filled with the number of bytes that actually landed in the output buffer because of the conversion. Can be NULL.
int *dstCharsPtr (out) — gets filled with the number of characters that correspond to however many bytes got stored in the output buffer. Can be NULL.
Tcl_Size *errorIdxPtr (out) — gets filled with the index of the byte or character that made the encoding transform fail. Can be NULL. 💀
Tcl_DString *bufPtr (out) — storage for the prescribed system encoding name.
const Tcl_EncodingType *typePtr (in) — the structure that defines a whole new type of encoding.
Tcl_Obj *searchPath (in) — a list of filesystem directories to go hunting through for encoding data files.
const char *path (in) — a path pointing at the location of the encoding file.

🌸 INTRODUCTION 🌸

N.B. Go read the Tcl_UniChar documentation page for the whole rundown on the TUTF-8 encoding and all the related terms that get name-dropped in here. Not optional if you want the lore, fr fr.

So these routines convert between TUTF-8 and other character representations using encodings like standard UTF-8, UTF-16, ASCII, or Shift-JIS — basically whatever encoding some system interface or other software component is expecting you to show up with. ✨

Picture it: on a Japanese Unix workstation, a user might get a filename represented in the EUC-JP file encoding, and then translate those characters to the jisx0208 font encoding so the filename can actually be displayed in a Tk widget. The whole point of the encoding package is to help bridge that translation gap. TUTF-8 is the intermediate staging ground for all the various encodings. In the example above, text would get translated into TUTF-8 from whatever file encoding the operating system is using. Then it would get translated from TUTF-8 into whatever font encoding the display routines require. It's giving middleman, but make it iconic. 💅

Some basic encodings are compiled right into Tcl. Others can be defined by the user, or dynamically loaded from encoding files in a platform-independent manner. Obsessed.

💖 DESCRIPTION 💖

Tcl_GetEncoding finds an encoding given its name. That name may refer to a built-in Tcl encoding, a user-defined encoding registered by calling Tcl_CreateEncoding, or a dynamically-loadable encoding file. What you get back is a token that represents the encoding and can be used in subsequent calls to procedures like Tcl_GetEncodingName, Tcl_FreeEncoding, and Tcl_UtfToExternal. But if the name didn't refer to any known or loadable encoding? You get NULL, and an error message gets returned in interp. 🙈

The encoding package keeps a whole database of all encodings currently in use. The first time name is seen, Tcl_GetEncoding returns an encoding with a reference count of 1. If the same name gets requested again later, the reference count for that encoding just gets incremented — no overhead of allocating a brand new encoding and all its associated data structures. Efficiency queen. 👑

When an encoding is no longer needed, Tcl_FreeEncoding should be called to release it. And when an encoding is no longer in use anywhere — like, it's been freed as many times as it's been gotten — Tcl_FreeEncoding will release all the storage that encoding was using and delete it from the database. Periodt.

Tcl_GetEncodingFromObj treats the string representation of objPtr as an encoding name, and finds an encoding with that name, just like Tcl_GetEncoding does. When an encoding is found, it gets cached inside the objPtr value for future reference, the Tcl_Encoding token is written to the storage pointed to by encodingPtr, and the value TCL_OK is returned. If no such encoding is found, the value TCL_ERROR is returned, and no writing to *encodingPtr takes place. And just like with Tcl_GetEncoding, the caller should call Tcl_FreeEncoding on the resulting encoding token once that token won't be used anymore. 🫶

Tcl_ExternalToUtfDString converts a source buffer src from the specified encoding into TUTF-8. The converted bytes get stored in dstPtr, which then gets null-terminated. The caller should eventually call Tcl_DStringFree to free any information stored in dstPtr. When converting, if any of the characters in the source buffer can't be represented in the target encoding, a default fallback character gets used. The return value is a pointer to the value stored in the DString.

Tcl_ExternalToUtfDStringEx is the more flexible version of the older Tcl_ExternalToUtfDString function — the glow-up, if you will. It takes three additional parameters: interp, flags and errorIdxPtr. The flags parameter may be used to specify the profile to be used for the transform. Heads up though: the TCL_ENCODING_START and TCL_ENCODING_END bits in flags are ignored, because the function assumes the entire source string to be decoded is passed into the function. On success, the function returns TCL_OK with the converted string stored in *dstPtr. For errors other than conversion errors — like invalid flags — the function returns TCL_ERROR with an error message in interp if it is not NULL.

For conversion errors, Tcl_ExternalToUtfDStringEx returns one of the TCL_CONVERT_* errors listed below for Tcl_ExternalToUtf. When one of these conversion errors is returned, an error message is stored in interp only if errorIdxPtr is NULL. Otherwise, no error message is stored, because the function expects that the caller is interested in the decoded data up to that point and is not treating this as an immediate error condition. The index of the error location gets stored in *errorIdxPtr.

The caller must call Tcl_DStringFree to free up the *dstPtr resources irrespective of the return value from the function. No skipping this one, bestie. 😭

Tcl_ExternalToUtf converts a source buffer src from the specified encoding into TUTF-8. Up to srcLen bytes are converted from the source buffer, and up to dstLen converted bytes are stored in dst. In all cases, *srcReadPtr gets filled with the number of bytes that were successfully converted from src, and *dstWrotePtr gets filled with the corresponding number of bytes that were stored in dst. The return value is one of the following:

TCL_OK — all bytes of src were converted. Slay. ✨
TCL_CONVERT_NOSPACE — the destination buffer was not large enough for all of the converted data; as many characters as could fit were converted though.
TCL_CONVERT_MULTIBYTE — the last few bytes in the source buffer were the beginning of a multibyte sequence, but more bytes were needed to complete this sequence. A subsequent call to the conversion routine should pass a buffer containing the unconverted bytes that remained in src plus some further bytes from the source stream, so the formerly split-up multibyte sequence can properly convert. It's giving "to be continued…" 🦋
TCL_CONVERT_SYNTAX — the source buffer contained an invalid byte or character sequence. This may occur if the input stream has been damaged, or if the input encoding method was misidentified.
TCL_CONVERT_UNKNOWN — the source buffer contained a character that could not be represented in the target encoding. 💀

Tcl_UtfToExternalDString converts a source buffer src from TUTF-8 into the specified encoding. The converted bytes get stored in dstPtr, which then gets terminated with the appropriate encoding-specific null. The caller should eventually call Tcl_DStringFree to free any information stored in dstPtr. When converting, if any of the characters in the source buffer can't be represented in the target encoding, a default fallback character gets used. The return value is a pointer to the value stored in the DString.

Tcl_UtfToExternalDStringEx is the enhanced version of Tcl_UtfToExternalDString that transforms TUTF-8 encoded source data to a specified encoding. Except for the direction of the transform, the parameters and return values are identical to those of Tcl_ExternalToUtfDStringEx. Go read that function above for the details on all of the same. 👆

Irrespective of the return code from the function, the caller must free resources associated with *dstPtr when the function returns. Every single time. No cap. 🎀

Tcl_UtfToExternal is the ✨glow-up✨ move: it takes a source buffer src and flips it from TUTF-8 into whatever encoding you named. Up to srcLen bytes get converted out of src, and up to dstLen converted bytes get parked in dst. 💾 Either way (every single time, no exceptions bestie), *srcReadPtr gets filled with how many bytes actually made it out of src, and *dstWrotePtr gets filled with the matching count of bytes that landed in dst. And the return values? Same energy as Tcl_ExternalToUtf, literally identical. 🔁

Tcl_GetEncodingName is basically the inverse of Tcl_GetEncoding, like the uno reverse card of this whole friend group. Hand it an encoding and it hands you back the name argument you used to create that encoding. 👑 But heads up: the string it returns is only guaranteed to stick around until the encoding gets deleted. So don't get attached. And the caller must not modify that string — hands off, it's not yours. 🙅‍♀️

Tcl_GetEncodingNulLength tells you the length of the terminating nul byte sequence for strings in the encoding you specified. That's it, that's the whole vibe check. 🎀

Tcl_SetSystemEncoding sets the default encoding that gets used whenever the user passes a NULL value for the encoding argument to any of the other encoding functions. It's the "if nobody says anything, use THIS" setting. 💅

If name is NULL, the system encoding resets to the default system encoding, binary. 🫶
If the name didn't refer to any known or loadable encoding, TCL_ERROR is returned and an error message gets left in interp. 😭
Otherwise? This procedure increments the reference count of the new system encoding, decrements the reference count of the old system encoding, and returns TCL_OK. Balance restored. ⚖️

Tcl_GetEncodingNameFromEnvironment gives the Tcl library a way to report the encoding name it believes is the correct one to use as the system encoding, based on system calls and a good long look at the environment suitable for the platform. 🔍 It accepts bufPtr, which is a pointer to an uninitialized or freed Tcl_DString, and writes the encoding name into it. It returns the Tcl_DStringValue. ✨

Tcl_GetEncodingNames sets the interp result to a list consisting of the names of all the encodings that are currently defined or can be dynamically loaded, searching the encoding path specified by Tcl_SetEncodingSearchPath. 🌈 Now here's the tea: this procedure does not verify that the dynamically-loadable encoding files actually contain valid data — it just checks that they exist. Existence check only, no vibe check on the contents. 🙈

Tcl_CreateEncoding defines a brand new encoding and registers the C procedures that get called back to convert between that encoding and TUTF-8. Encodings made this way are thereafter visible in the database that Tcl_GetEncoding uses. Just like with Tcl_GetEncoding, the return value is a token representing the encoding, and you can use it in later calls to other encoding functions. 🎟️

Tcl_CreateEncoding returns an encoding with a reference count of 1. And if an encoding with the specified name already exists? Its entry in the database gets replaced with the new one. The token for the old encoding stays valid and keeps behaving exactly as before, but anybody using the new token will now be calling the new encoding procedures. Old token, old behavior — new token, new behavior. Split custody fr fr. 💀

The typePtr argument to Tcl_CreateEncoding carries the info about the encoding's name and the procedures that'll be called to convert between this encoding and TUTF-8. It's defined like this:

typedef struct {
    const char *encodingName;
    Tcl_EncodingConvertProc *toUtfProc;
    Tcl_EncodingConvertProc *fromUtfProc;
    Tcl_EncodingFreeProc *freeProc;
    void *clientData;
    Tcl_Size nullSize;
} Tcl_EncodingType;

Breakdown time bestie, every field matters: 🧚

encodingName — gives a string name for the encoding, so you can refer to it in other procedures like Tcl_GetEncoding. 🏷️
toUtfProc — the callback procedure to invoke to convert text from this encoding into TUTF-8.
fromUtfProc — the callback procedure to invoke to convert text from TUTF-8 into this encoding.
freeProc — the callback procedure to invoke when this encoding is deleted. May be NULL.
clientData — holds an arbitrary one-word value that gets passed to toUtfProc, fromUtfProc, and freeProc whenever they're called. Typically this is a pointer to a data structure containing encoding-specific info that the callback procedures can use. For instance, two very similar encodings like ascii and macRoman may use the same callback procedure but different clientData values to control its behavior. Same function, different personality. 💅
nullSize — specifies the number of zero bytes that signify end-of-string in this encoding. It must be 1 (for single-byte or multi-byte encodings like ASCII or Shift-JIS) or 2 (for double-byte encodings like Unicode). ⚠️ Constant-sized encodings with 3 or more bytes per character (such as CNS11643) are not accepted. Not invited to the party, sorry. 🚫

The callback procedures toUtfProc and fromUtfProc should match the type Tcl_EncodingConvertProc:

typedef int Tcl_EncodingConvertProc(
        void *clientData,
        const char *src,
        int srcLen,
        int flags,
        Tcl_EncodingState *statePtr,
        char *dst,
        int dstLen,
        int *srcReadPtr,
        int *dstWrotePtr,
        int *dstCharsPtr);

The toUtfProc and fromUtfProc procedures get called by the Tcl_ExternalToUtf or Tcl_UtfToExternal family of functions to do the actual conversion. 🔥 The clientData parameter to these procedures is the very same as the clientData field you specified to Tcl_CreateEncoding back when the encoding was created — it remembers, it's loyal. 🫶

The remaining arguments to the callback procedures are the same as the arguments documented at the top, the ones for Tcl_ExternalToUtf or Tcl_UtfToExternal, with the following exceptions:

If the srcLen argument to one of those high-level functions is negative, the value passed to the callback procedure will be the appropriate encoding-specific string length of src. 📏
If any of the srcReadPtr, dstWrotePtr, or dstCharsPtr arguments to one of the high-level functions is NULL, the corresponding value passed to the callback procedure will be a non-NULL location. (So your callback never has to worry about a NULL pointer sneaking in there. 💖)

The callback procedure freeProc, if non-NULL, should match the type Tcl_EncodingFreeProc:

typedef void Tcl_EncodingFreeProc(
        void *clientData);

This freeProc function gets called when the encoding is deleted. 🪦 The clientData parameter is the same as the clientData field specified to Tcl_CreateEncoding when the encoding was created.

Tcl_GetEncodingSearchPath and Tcl_SetEncodingSearchPath are called to access and set the list of filesystem directories that get searched for encoding data files. 📂 It's giving very "where did I put my stuff" energy.

The value returned by Tcl_GetEncodingSearchPath is the value stored by the last successful call to Tcl_SetEncodingSearchPath. If no calls to Tcl_SetEncodingSearchPath have ever happened, Tcl will compute an initial value based on the environment. 🌱 And important detail: there is one encoding search path for the entire process, shared by all threads in the process. One path to rule them all, shared custody, no exceptions. 👑

Tcl_SetEncodingSearchPath stores searchPath and returns TCL_OK — unless searchPath is not a valid Tcl list, which causes TCL_ERROR to be returned. 😭 Note that the elements of searchPath are not verified as existing readable filesystem directories. So when the search for encoding data files actually takes place, any non-existent or non-readable filesystem directories on the searchPath get silently ignored. No error, no warning, just quietly pretending it never saw them. 🫥

✨ ENCODING FILES ✨

okay so here's the tea bestie 💅 — if Tcl tried to precompile every encoding algorithm it could ever need straight into the binary, space would prohibit it. so instead, a whole bunch of encodings just live on disk as dynamically-loadable encoding files. and honestly? that's lowkey iconic, because it means you can make your own extra encoding files and load them through the exact same mechanism. 🎀

these files hold the info about the tables and/or escape sequences that map between an external encoding and Unicode. and the external encoding can be made of single-byte, multi-byte, or double-byte characters — all vibes, all valid. 🌈

every dynamically-loadable encoding is represented as a text file. the very first line, which starts with a “#” symbol, is a comment giving a human-readable description of the file (it's giving cover letter 💀). the next line says what type of encoding file it is, and it's one of these letters:

S — a single-byte encoding, where one character is always one byte long in the encoding. example: iso8859-1, used by so many European languages. 🍓
D — a double-byte encoding, where one character is always two bytes long. example: big5, used for Chinese text.
M — a multi-byte encoding, where one character may be either one or two bytes long. certain bytes are lead bytes, meaning "another byte is required to follow me, and together we make one character." other bytes are not lead bytes and just represent themselves. example: shiftjis, used by a ton of Japanese computers. ✨
E — an escape-sequence encoding, which says that certain sequences of bytes don't represent characters at all, but commands telling you how the following bytes should be interpreted.

the rest of the lines in the file depend on which type you picked. no cap.

🧚 the table-based girlies (cases `[1]`, `[2]`, and `[3]`)

cases [1], [2], and [3] are collectively referred to as table-based encoding files. 🙌 the lines in a table-based encoding file are in the same format as this example taken from the shiftjis encoding (and bestie, this is not the complete file):

# Encoding file: shiftjis, multi-byte
M
003F 0 40
00
0000000100020003000400050006000700080009000A000B000C000D000E000F
0010001100120013001400150016001700180019001A001B001C001D001E001F
0020002100220023002400250026002700280029002A002B002C002D002E002F
0030003100320033003400350036003700380039003A003B003C003D003E003F
0040004100420043004400450046004700480049004A004B004C004D004E004F
0050005100520053005400550056005700580059005A005B005C005D005E005F
0060006100620063006400650066006700680069006A006B006C006D006E006F
0070007100720073007400750076007700780079007A007B007C007D203E007F
0080000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000FF61FF62FF63FF64FF65FF66FF67FF68FF69FF6AFF6BFF6CFF6DFF6EFF6F
FF70FF71FF72FF73FF74FF75FF76FF77FF78FF79FF7AFF7BFF7CFF7DFF7EFF7F
FF80FF81FF82FF83FF84FF85FF86FF87FF88FF89FF8AFF8BFF8CFF8DFF8EFF8F
FF90FF91FF92FF93FF94FF95FF96FF97FF98FF99FF9AFF9BFF9CFF9DFF9EFF9F
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
81
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
300030013002FF0CFF0E30FBFF1AFF1BFF1FFF01309B309C00B4FF4000A8FF3E
FFE3FF3F30FD30FE309D309E30034EDD30053006300730FC20152010FF0F005C
301C2016FF5C2026202520182019201C201DFF08FF0930143015FF3BFF3DFF5B
FF5D30083009300A300B300C300D300E300F30103011FF0B221200B100D70000
00F7FF1D2260FF1CFF1E22662267221E22342642264000B0203220332103FFE5
FF0400A200A3FF05FF03FF06FF0AFF2000A72606260525CB25CF25CE25C725C6
25A125A025B325B225BD25BC203B301221922190219121933013000000000000
000000000000000000000000000000002208220B2286228722822283222A2229
000000000000000000000000000000002227222800AC21D221D4220022030000
0000000000000000000000000000000000000000222022A52312220222072261
2252226A226B221A223D221D2235222B222C0000000000000000000000000000
212B2030266F266D266A2020202100B6000000000000000025EF000000000000

so the third line of the file is three numbers. let's break it down 🫶

the first number is the fallback character (in base 16) to use when converting from TUTF-8 to this encoding.
the second number is a 1 if this file represents the encoding for a symbol font, or 0 otherwise.
the last number (in base 10) is how many pages of data follow.

the lines after that in the example up there are the pages, and they describe how to map from the encoding into 2-byte Unicode. the first line in a page identifies the page number. 👑

following that page number you get 256 double-byte numbers, arranged as 16 rows of 16 numbers. here's the magic: given a character in the encoding, the high byte of that character picks which page, and the low byte picks the index into that page to select one of the double-byte numbers — and the value you get is the corresponding Unicode character. ✨

if you squint at the example above, you can see that the characters 0x7E and 0x8163 in shiftjis map to 203E and 2026 in Unicode, respectively. slay.

after the first page comes all the other pages, each in the exact same format as the first: one number identifying the page, followed by 256 double-byte Unicode characters. if a character in the encoding maps to the Unicode character 0000, that means the character doesn't actually exist. and if every single character on a page would map to 0000, you can just leave that whole page out. 💀 efficient queen.

🌸 case `[4]`: the escape-sequence encoding file

case [4] is the escape-sequence encoding file. the lines in this type of file follow the same format as this example taken from the iso2022-jp encoding:

# Encoding file: iso2022-jp, escape-driven
E
init           {}
final          {}
iso8859-1      \x1B(B
jis0201        \x1B(J
jis0208        \x1B$@
jis0208        \x1B$B
jis0212        \x1B$(D
gb2312         \x1B$A
ksc5601        \x1B$(C

in the file, the first column is an option and the second column is the value that goes with it. 🎀

init is a string to emit or expect before the first character gets converted.
final is a string to emit or expect after the last character.
every other option is the name of a table-based encoding, and its associated value is the escape-sequence that marks that encoding.

Tcl syntax is used for the values — so in the example above, for instance, “{}” represents the empty string and “\x1B” represents character 27. 💅

🔥 how it actually loads stuff

when Tcl_GetEncoding encounters an encoding name that hasn't been loaded, it attempts to load an encoding file called name.enc from the encoding subdirectory of each directory that Tcl searches for its script library. and if the encoding file exists but is malformed? an error message gets left in interp. 😭 so like, heads up.

💖 REFERENCE COUNT MANAGEMENT 💖

Tcl_GetEncodingFromObj does not modify the reference count of its objPtr argument — it only reads, ok? but note that this function may set the interpreter result, and if that result is the only place holding a reference to the object, then bestie… it will be deleted. 🫠
Tcl_GetEncodingSearchPath returns an object with a reference count of at least 1. periodt.

👑 PROFILES 👑

encoding profiles define the manner in which errors in the encoding transforms are handled by the encoding functions. an application can specify which profile to use by OR-ing the flags parameter passed to the function with at most one of TCL_ENCODING_PROFILE_TCL8, TCL_ENCODING_PROFILE_STRICT or TCL_ENCODING_PROFILE_REPLACE. these correspond to the tcl8, strict and replace profiles respectively. ✨

if you don't specify any of them, a version-dependent default profile is used. and for Tcl 9.0, the default profile is strict. 💅

for the full deets about profiles, go check the PROFILES section in the documentation of the encoding command. iykyk.

🌈 SEE ALSO 🌈

encoding(3tcl)

⭐ KEYWORDS ⭐

utf, encoding, convert

and that's the whole page bestie, we ate and left no crumbs 🎀✨ this was Tcl_GetEncoding(3tcl) from Tcl, version 8.1, dated back in the day — the original footer said Tcl 8.1 and Tcl_GetEncoding(3tcl) and we're honoring that exactly, no notes, mwah 💖🦋

📦 where this page comes from

ubuntu stonking: tcl9.0-doc 9.0.4+dfsg-1
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms