Tcl_ParseCommand(3)

💖 Tcl_ParseCommand — the Tcl Library Procedures glow-up 💖

okay bestie, gather round, because we're about to get intimate with the Tcl parser. this is the Tcl_ParseCommand(3) page, and she is long, so buckle up. part 1 of 2, no skimming allowed. 🫶

---

✨ NAME ✨

Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, Tcl_ParseVarName, Tcl_ParseVar, Tcl_FreeParse, Tcl_EvalTokensStandard — parse Tcl scripts and expressions 💅

---

🌸 SYNOPSIS 🌸

like, the receipts, exactly as they are:

#include <tcl.h>

int
Tcl_ParseCommand(interp, start, numBytes, nested, parsePtr)

int
Tcl_ParseExpr(interp, start, numBytes, parsePtr)

int
Tcl_ParseBraces(interp, start, numBytes, parsePtr, append, termPtr)

int
Tcl_ParseQuotedString(interp, start, numBytes, parsePtr, append, termPtr)

int
Tcl_ParseVarName(interp, start, numBytes, parsePtr, append)

const char *
Tcl_ParseVar(interp, start, termPtr)

Tcl_FreeParse(usedParsePtr)

int
Tcl_EvalTokensStandard(interp, tokenPtr, numTokens)

---

🎀 ARGUMENTS 🎀

here's the cast, every single one, no cuts:

• Tcl_Interp *interp (out) — for every procedure except Tcl_FreeParse and Tcl_EvalTokensStandard, this is only used for error reporting. if it's NULL, then no error messages get left behind after errors. but for Tcl_EvalTokensStandard, it actually determines the context for evaluating the script and it's used for error reporting too — so it must not be NULL. periodt. 💀
• const char *start (in) — pointer to the first character in the string we're parsing. the main character energy, fr. 👑
• Tcl_Size numBytes (in) — how many bytes are in the string to parse, not counting any terminating null character. and if it's less than 0, then the script is literally all the characters following start up to the first null character. iykyk. ✨
• int nested (in) — non-zero means the script is part of a command substitution, so an unquoted close bracket should be treated as a command terminator. if it's zero, close brackets have no special meaning whatsoever. just vibes. 🦋
• int append (in) — non-zero means *parsePtr already holds valid tokens and the new tokens should be appended to the ones already there. zero means *parsePtr is uninitialized and anything in it gets ignored. this argument is normally 0. 🌸
• Tcl_Parse *parsePtr (out) — points to the structure we fill in with info about the parsed command, expression, variable name, etc. any previous info in this structure is ignored, unless append is non-zero in a call to Tcl_ParseBraces, Tcl_ParseQuotedString, or Tcl_ParseVarName. 🫶
• const char **termPtr (out) — if it's not NULL, this points to a location where Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVar will stash a pointer to the character just after the terminating character (that's the close-brace, the last character of the variable name, or the close-quote respectively) — but only if the parse succeeded. 🔥
• Tcl_Parse *usedParsePtr (in) — points to a structure that was filled in by a previous call to Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseVarName, etc. recycling, we love to see it. ♻️

---

💜 DESCRIPTION 💜

so these procedures parse Tcl commands, or little pieces of Tcl commands like expressions or references to variables. each procedure takes a pointer to a script (or a portion of one) and fills in the structure pointed to by parsePtr with a collection of tokens describing the info that was parsed. normally they return TCL_OK. but if an error happens, they return TCL_ERROR, leave an error message in interp's result (if interp is not NULL), and leave nothing in parsePtr. 💅

---

🌟 Tcl_ParseCommand 🌟

Tcl_ParseCommand parses Tcl scripts. give it a pointer to a script and it parses the first command from the script. if the command parsed successfully, Tcl_ParseCommand returns TCL_OK and fills in the structure pointed to by parsePtr with info about the structure of the command (more on that below, keep reading bestie). if an error occurred while parsing the command, then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr. 😭

---

🌈 Tcl_ParseExpr 🌈

Tcl_ParseExpr parses Tcl expressions. give it a pointer to a script containing an expression and it parses the expression. if the expression parsed successfully, Tcl_ParseExpr returns TCL_OK and fills in the structure pointed to by parsePtr with info about the structure of the expression (again, below). if an error occurred while parsing the command, then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr. 💀

---

🎀 Tcl_ParseBraces 🎀

Tcl_ParseBraces parses a string or command argument enclosed in braces, like {hello} or {string \t with \t tabs}, starting from the beginning of its start argument. the first character of start must be {. if the braced string parsed successfully, Tcl_ParseBraces returns TCL_OK, fills in the structure pointed to by parsePtr with info about the structure of the string (you know the drill), and stores a pointer to the character just after the terminating } in the location given by *termPtr. if an error occurs while parsing the string, then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr or *termPtr. 🦋

---

✨ Tcl_ParseQuotedString ✨

Tcl_ParseQuotedString parses a double-quoted string like "sum is [expr {$a+$b}]" from the beginning of the start argument. the first character of start must be ". if the double-quoted string parsed successfully, Tcl_ParseQuotedString returns TCL_OK, fills in the structure pointed to by parsePtr with info about the structure of the string (yes, also below), and stores a pointer to the character just after the terminating " in the location given by *termPtr. if an error occurs while parsing the string, then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr or *termPtr. 😤

---

🌸 Tcl_ParseVarName 🌸

Tcl_ParseVarName parses a Tcl variable reference like $abc or $x([expr {$index + 1}]) from the beginning of its start argument. the first character of start must be $. if a variable name parsed successfully, Tcl_ParseVarName returns TCL_OK and fills in the structure pointed to by parsePtr with info about the structure of the variable name (yes bestie, below). if an error occurs while parsing the command, then TCL_ERROR is returned, an error message is left in interp's result (if interp is not NULL), and no information is left at *parsePtr. 🫠

---

💜 Tcl_ParseVar 💜

Tcl_ParseVar parses a Tcl variable reference like $abc or $x([expr {$index + 1}]) from the beginning of its start argument. the first character of start must be $. if the variable name parsed successfully, Tcl_ParseVar returns a pointer to the string value of the variable. if an error occurs while parsing, then NULL is returned and an error message is left in interp's result. 🫶

---

🧚 memory management moment 🧚

the info left at *parsePtr by Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName may include dynamically allocated memory. if these five parsing procedures return TCL_OK, then the caller must invoke Tcl_FreeParse to release the storage at *parsePtr. these procedures ignore any existing information in *parsePtr (unless append is non-zero), so if you're making repeated calls to any of them, then Tcl_FreeParse must be invoked once after each call. like, don't leak, bestie, it's not cute. 🚨

---

🔥 Tcl_EvalTokensStandard 🔥

Tcl_EvalTokensStandard evaluates a sequence of parse tokens from a Tcl_Parse structure. the tokens typically consist of all the tokens in a word, or all the tokens that make up the index for a reference to an array variable. Tcl_EvalTokensStandard performs the substitutions requested by the tokens and concatenates the resulting values. the return value from Tcl_EvalTokensStandard is a Tcl completion code with one of the values TCL_OK, TCL_ERROR, TCL_RETURN, TCL_BREAK, or TCL_CONTINUE, or possibly some other integer value originating in an extension. on top of that, a result value or error message is left in interp's result; it can be retrieved using Tcl_GetObjResult. slay. 💅

---

👑 TCL_PARSE STRUCTURE 👑

okay so Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName return parse information in two data structures: Tcl_Parse and Tcl_Token. the duo. the iconic pair. 🌈

typedef struct {
    const char *commentStart;
    Tcl_Size commentSize;
    const char *commandStart;
    Tcl_Size commandSize;
    Tcl_Size numWords;
    Tcl_Token *tokenPtr;
    Tcl_Size numTokens;
    ...
} Tcl_Parse;

typedef struct {
    int type;
    const char *start;
    Tcl_Size size;
    Tcl_Size numComponents;
} Tcl_Token;

the first five fields of a Tcl_Parse structure are filled in only by Tcl_ParseCommand. these fields are not used by the other parsing procedures. so like, don't come for them, they're spoken for. 🙈

---

okay pausing here because this page is genuinely a two-parter and part 2 is where the rest of the Tcl_Parse / Tcl_Token deets live 💖

...and your type field is gonna be exactly one of these, bestie. Each one has its own whole personality, so let's get into it ✨

• TCL_TOKEN_WORD — the main character 💅 Usually this bad boy describes one single word of a command, but plot twist: it can also describe a quoted or braced string sitting inside an expression. What it's really describing is a component of the script that exists because you concatenated a whole sequence of subcomponents together — and every one of those subcomponents gets its own subtoken. The token kicks off at the first non-blank character of the component (so yeah, that could be a double-quote or an open brace 👀) and it swallows every character in the component right up to — but not including — the space, semicolon, close bracket, close quote, or close brace that terminates the component. And numComponents? That's counting the total number of sub-tokens making up the word, and yes that includes the sub-tokens of TCL_TOKEN_VARIABLE and TCL_TOKEN_BS tokens too. no skipping 🫶
• TCL_TOKEN_SIMPLE_WORD — literally the same meaning as TCL_TOKEN_WORD, except the word is guaranteed to consist of a single TCL_TOKEN_TEXT sub-token. So numComponents is always 1. always. iconic consistency ✨
• TCL_TOKEN_EXPAND_WORD — again, same meaning as TCL_TOKEN_WORD, except the command parser clocked that this word began with the expansion prefix {*}, which means that after substitution, the list value of this word should be expanded to form multiple arguments in command evaluation. and ngl, this token type can only be created by Tcl_ParseCommand. it's exclusive, it's rare, it's her 💅
• TCL_TOKEN_TEXT — describes a range of literal text that's part of a word. numComponents is always 0. simple, clean, no drama 🌸
• TCL_TOKEN_BS — describes a backslash sequence, like \n or \0xA3. numComponents is always 0. 💜
• TCL_TOKEN_COMMAND — describes a command whose result has to get substituted into the word. The token includes the square brackets that surround the command. numComponents is always 0 — and hear me out, the nested command is not parsed, okay? if you want to see its tokens you have to call Tcl_ParseCommand recursively yourself. do the work bestie 😭
• TCL_TOKEN_VARIABLE — describes a variable substitution, and that means the $, the variable name, and the array index (if there is one) all the way up through the close parenthesis that terminates the index. This token is followed by one or more additional tokens describing the variable name and array index. So: if numComponents is 1, the variable is a scalar, and the next token is a TCL_TOKEN_TEXT token giving the variable name. If numComponents is greater than 1, it's an array — the first sub-token is a TCL_TOKEN_TEXT token giving the array name, and the remaining sub-tokens are TCL_TOKEN_TEXT, TCL_TOKEN_BS, TCL_TOKEN_COMMAND, and TCL_TOKEN_VARIABLE tokens that must be concatenated to produce the array index. And numComponents includes nested sub-tokens that are part of TCL_TOKEN_VARIABLE tokens in the array index. yeah. it nests. it's giving inception 🌀
• TCL_TOKEN_SUB_EXPR — describes one subexpression of an expression (or, if you're feeling fancy, an entire expression). A subexpression may consist of a value — like an integer literal, a variable substitution, or a parenthesized subexpression — and it may also consist of an operator plus its operands. The token starts with the first non-blank character of the subexpression, up to but not including the space, brace, close-paren, or bracket that terminates the subexpression. This token is followed by one or more additional tokens describing the subexpression. Here's the tea: if the first sub-token after the TCL_TOKEN_SUB_EXPR token is a TCL_TOKEN_OPERATOR token, then the subexpression is an operator and its token operands — and if that operator has no operands, the subexpression consists of just the TCL_TOKEN_OPERATOR token, periodt. Each operand is described by a TCL_TOKEN_SUB_EXPR token. Otherwise, the subexpression is a value described by one of TCL_TOKEN_WORD, TCL_TOKEN_TEXT, TCL_TOKEN_BS, TCL_TOKEN_COMMAND, TCL_TOKEN_VARIABLE, and TCL_TOKEN_SUB_EXPR. numComponents counts the total number of sub-tokens that make up the subexpression, and yes, that includes the sub-tokens for any nested TCL_TOKEN_SUB_EXPR tokens 🔥
• TCL_TOKEN_OPERATOR — describes one operator of an expression, like && or hypot. A TCL_TOKEN_OPERATOR token is always preceded by a TCL_TOKEN_SUB_EXPR token that describes the operator and its operands, and you can use that TCL_TOKEN_SUB_EXPR token's numComponents field to figure out how many operands there are. A binary operator like * is followed by two TCL_TOKEN_SUB_EXPR tokens describing its operands. A unary operator like - is followed by a single TCL_TOKEN_SUB_EXPR token for its operand. If the operator is a math function like log10, the TCL_TOKEN_OPERATOR token gives its name and the following TCL_TOKEN_SUB_EXPR tokens describe its operands — and if there are no operands (as with rand), no TCL_TOKEN_SUB_EXPR tokens follow at all. And there's exactly one trinary operator, ?, which shows up in if-then-else subexpressions like x?y:z — in that case the ? TCL_TOKEN_OPERATOR token is followed by three TCL_TOKEN_SUB_EXPR tokens for the operands x, y, and z. The numComponents field for a TCL_TOKEN_OPERATOR token is always 0 ⭐

---

okay now the "after it returns" section, this is where it all comes together besties 👑

After Tcl_ParseCommand returns, the very first token pointed to by the tokenPtr field of the Tcl_Parse structure always has type TCL_TOKEN_WORD or TCL_TOKEN_SIMPLE_WORD or TCL_TOKEN_EXPAND_WORD. It's followed by the sub-tokens that must be concatenated to produce the value of that word. Then the next token is the TCL_TOKEN_WORD or TCL_TOKEN_SIMPLE_WORD of TCL_TOKEN_EXPAND_WORD token for the second word, followed by sub-tokens for that word, and so on and so on until all numWords have been accounted for. every single one. no stragglers 🌈

After Tcl_ParseExpr returns, the first token pointed to by the tokenPtr field of the Tcl_Parse structure always has type TCL_TOKEN_SUB_EXPR. It's followed by the sub-tokens that must be evaluated to produce the value of the expression. And heads up: only the token information in the Tcl_Parse structure gets modified here — the commentStart, commentSize, commandStart, and commandSize fields are not modified by Tcl_ParseExpr. hands off 🫶

After Tcl_ParseBraces returns, the array of tokens pointed to by the tokenPtr field of the Tcl_Parse structure will contain a single TCL_TOKEN_TEXT token if the braced string does not contain any backslash-newlines. If the string does contain backslash-newlines, the array of tokens will contain one or more TCL_TOKEN_TEXT or TCL_TOKEN_BS sub-tokens that must be concatenated to produce the value of the string. And if the braced string was just {} (that is, the string was empty), the single TCL_TOKEN_TEXT token will have a size field containing zero — which is lowkey a serve, because it ensures that at least one token appears to describe the braced string. Again: only the token information in the Tcl_Parse structure is modified; the commentStart, commentSize, commandStart, and commandSize fields are not modified by Tcl_ParseBraces 💖

After Tcl_ParseQuotedString returns, the array of tokens pointed to by the tokenPtr field of the Tcl_Parse structure depends on the contents of the quoted string. It'll consist of one or more TCL_TOKEN_TEXT, TCL_TOKEN_BS, TCL_TOKEN_COMMAND, and TCL_TOKEN_VARIABLE sub-tokens. The array always contains at least one token — for example, if the argument start is empty, the array returned consists of a single TCL_TOKEN_TEXT token with a zero size field. And yes, same deal: only the token information in the Tcl_Parse structure is modified, and the commentStart, commentSize, commandStart, and commandSize fields are not modified 🌸

After Tcl_ParseVarName returns, the first token pointed to by the tokenPtr field of the Tcl_Parse structure always has type TCL_TOKEN_VARIABLE. It's followed by the sub-tokens that make up the variable name as described above. The total length of the variable name is contained in the size field of the first token. And as in Tcl_ParseExpr, only the token information in the Tcl_Parse structure is modified by Tcl_ParseVarName — the commentStart, commentSize, commandStart, and commandSize fields are not modified 💜

All of the character pointers in the Tcl_Parse and Tcl_Token structures refer to characters in the start argument passed to Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName. one source of truth, we love to see it ⭐

okay real talk, important one: there are additional fields in the Tcl_Parse structure after the numTokens field, but those are for the private use of Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName. They should not be referenced by code outside of these procedures. like. do not touch. it's not yours bestie 🙈

---

🎀 KEYWORDS 🎀

• backslash substitution
• braces
• command
• expression
• parse
• token
• variable substitution

---

and that's the whole page babes, you ate it 💅✨ this one's from the Tcl 8.3 era, filed under Tcl_ParseCommand(3) — go forth and parse responsibly, mwah 🫶💖🦋