Tcl_ParseCommand(3)

💅 Tcl_ParseCommand(3) — Tcl Library Procedures 🎀

okay so bestie, welcome to the Tcl Library Procedures glow-up of Tcl_ParseCommand(3). this is the page where Tcl learns to read your scripts and expressions like the main character it is. grab a strawberry drink 🍓 and let's get into it.

---

✨ NAME ✨

Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, Tcl_ParseVarName, Tcl_ParseVar, Tcl_FreeParse, Tcl_EvalTokens, Tcl_EvalTokensStandard — literally the whole friend group that parses Tcl scripts and expressions 🫶

---

🌈 SYNOPSIS 🌈

#include <tcl.h>

int
Tcl_ParseCommand(interp, start, numBytes, nested, parsePtr)

int
Tcl_ParseExpr(interp, start, numBytes, parsePtr)

int
Tcl_ParseBraces(interp, start, numBytes, parsePtr, append, termPtr)

int
Tcl_ParseQuotedString(interp, start, numBytes, parsePtr, append, termPtr)

int
Tcl_ParseVarName(interp, start, numBytes, parsePtr, append)

const char *
Tcl_ParseVar(interp, start, termPtr)

Tcl_FreeParse(usedParsePtr)

Tcl_Obj *
Tcl_EvalTokens(interp, tokenPtr, numTokens)

int
Tcl_EvalTokensStandard(interp, tokenPtr, numTokens)

---

🦋 ARGUMENTS 🦋

• Tcl_Interp *interp (out) — for every procedure except Tcl_FreeParse, Tcl_EvalTokens and Tcl_EvalTokensStandard, this is used only for error reporting. if it's NULL, then no error messages get left behind after errors. for Tcl_EvalTokens and Tcl_EvalTokensStandard, it determines the context for evaluating the script and is also used for error reporting — and it must not be NULL, no cap 🫶
• const char *start (in) — pointer to the first character in the string to parse. like, the starting point of the whole vibe check ✨
• int numBytes (in) — number of bytes in the string to parse, not including any terminating null character. if it's less than 0, then the script consists of all characters following start up to the first null character. iykyk 😌
• int nested (in) — non-zero means the script is part of a command substitution, so an unquoted close bracket should be treated as a command terminator. if zero, close brackets have no special meaning. periodt 💅
• int append (in) — non-zero means *parsePtr already contains valid tokens, and the new tokens should be appended to those already present. zero means *parsePtr is uninitialized; any information in it is ignored. this argument is normally 0. ngl you'll almost always leave it alone 🙈
• Tcl_Parse *parsePtr (out) — points to the structure to fill in with information about the parsed command, expression, variable name, etc. any previous information in this structure is ignored, unless append is non-zero in a call to Tcl_ParseBraces, Tcl_ParseQuotedString, or Tcl_ParseVarName. she's the main character structure 👑
• const char **termPtr (out) — if not NULL, points to a location where Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVar will store a pointer to the character just after the terminating character — the close-brace, the last character of the variable name, or the close-quote (respectively) — if the parse was successful. so like, the "and that's where it ended" pointer 🎀
• Tcl_Parse *usedParsePtr (in) — points to a structure that was filled in by a previous call to Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseVarName, etc. it's the "we've been here before" pointer 💀

---

🌸 DESCRIPTION 🌸

these procedures parse Tcl commands or portions of Tcl commands such as expressions or references to variables. each procedure takes a pointer to a script (or portion thereof) and fills in the structure pointed to by parsePtr with a collection of tokens describing the information that was parsed. the procedures normally return TCL_OK. however, if an error occurs then they return TCL_ERROR, leave an error message in interp's result (if interp is not NULL), and leave nothing in parsePtr. so on a bad day, parsePtr gets absolutely nothing. iconic but tragic 😭

Tcl_ParseCommand is the procedure that parses Tcl scripts. given a pointer to a script, it parses the first command from the script. if the command was parsed successfully, Tcl_ParseCommand returns TCL_OK and fills in the structure pointed to by parsePtr with information about the structure of the command (see below for details). if an error occurred in parsing the command then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr. she said "first command only, bestie" 💅

Tcl_ParseExpr parses Tcl expressions. given a pointer to a script containing an expression, Tcl_ParseExpr parses the expression. if the expression was parsed successfully, Tcl_ParseExpr returns TCL_OK and fills in the structure pointed to by parsePtr with information about the structure of the expression (see below for details). if an error occurred in parsing the command then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr. math girlies rise ✨

Tcl_ParseBraces parses a string or command argument enclosed in braces such as {hello} or {string \t with \t tabs} from the beginning of its start argument. the first character of start must be {. if the braced string was parsed successfully, Tcl_ParseBraces returns TCL_OK, fills in the structure pointed to by parsePtr with information about the structure of the string (see below for details), and stores a pointer to the character just after the terminating } in the location given by *termPtr. if an error occurs while parsing the string then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr or *termPtr. curly braces are her whole aesthetic 🎀

Tcl_ParseQuotedString parses a double-quoted string such as "sum is [expr {$a+$b}]" from the beginning of the argument start. the first character of start must be ". if the double-quoted string was parsed successfully, Tcl_ParseQuotedString returns TCL_OK, fills in the structure pointed to by parsePtr with information about the structure of the string (see below for details), and stores a pointer to the character just after the terminating " in the location given by *termPtr. if an error occurs while parsing the string then TCL_ERROR is returned, an error message is left in interp's result, and no information is left at *parsePtr or *termPtr. quotes are lowkey her signature move 🌸

Tcl_ParseVarName parses a Tcl variable reference such as $abc or $x([expr {$index + 1}]) from the beginning of its start argument. the first character of start must be $. if a variable name was parsed successfully, Tcl_ParseVarName returns TCL_OK and fills in the structure pointed to by parsePtr with information about the structure of the variable name (see below for details). if an error occurs while parsing the command then TCL_ERROR is returned, an error message is left in interp's result (if interp is not NULL), and no information is left at *parsePtr. dollar signs are the blueprint, fr fr 💸

Tcl_ParseVar parses a Tcl variable reference such as $abc or $x([expr {$index + 1}]) from the beginning of its start argument. the first character of start must be $. if the variable name is parsed successfully, Tcl_ParseVar returns a pointer to the string value of the variable. if an error occurs while parsing, then NULL is returned and an error message is left in interp's result. she gives you the value, not just the name — main character energy 🌟

the information left at *parsePtr by Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName may include dynamically allocated memory. if these five parsing procedures return TCL_OK then the caller must invoke Tcl_FreeParse to release the storage at *parsePtr. these procedures ignore any existing information in *parsePtr (unless append is non-zero), so if repeated calls are being made to any of them then Tcl_FreeParse must be invoked once after each call. okay so literally do not forget this or you're leaking memory and that's not slay, that's a mess 😭

Tcl_EvalTokensStandard evaluates a sequence of parse tokens from a Tcl_Parse structure. the tokens typically consist of all the tokens in a word or all the tokens that make up the index for a reference to an array variable. Tcl_EvalTokensStandard performs the substitutions requested by the tokens and concatenates the resulting values. the return value from Tcl_EvalTokensStandard is a Tcl completion code with one of the values TCL_OK, TCL_ERROR, TCL_RETURN, TCL_BREAK, or TCL_CONTINUE, or possibly some other integer value originating in an extension. in addition, a result value or error message is left in interp's result; it can be retrieved using Tcl_GetObjResult. completion code buffet, no cap 🍓

Tcl_EvalTokens differs from Tcl_EvalTokensStandard only in the return convention used: it returns the result in a new Tcl_Obj. the reference count of the value returned as result has been incremented, so the caller must invoke Tcl_DecrRefCount when it is finished with the value. if an error or other exception occurs while evaluating the tokens (such as a reference to a non-existent variable) then the return value is NULL and an error message is left in interp's result. the use of Tcl_EvalTokens is deprecated. periodt — she's deprecated, put her in the group chat archive 💀

---

👑 TCL_PARSE STRUCTURE 👑

Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName return parse information in two data structures, Tcl_Parse and Tcl_Token:

typedef struct Tcl_Parse {
    const char *commentStart;
    int commentSize;
    const char *commandStart;
    int commandSize;
    int numWords;
    Tcl_Token *tokenPtr;
    int numTokens;
    ...
} Tcl_Parse;

typedef struct Tcl_Token {
    int type;
    const char *start;
    int size;
    int numComponents;
} Tcl_Token;

Okay so quick vibe check on these two structs bestie, because the fields do not all get filled in by the same functions and that's lowkey important:

• the first five fields of Tcl_Parse — commentStart, commentSize, commandStart, commandSize, numWords — are filled in only by Tcl_ParseCommand. The other parsing procedures literally don't touch them. Not their business. 💅

---

💅 the token types 💅

So Tcl_ParseCommand fills in a Tcl_Parse structure with the deets about one Tcl command and any comments that come before it. Let's break it down:

• if there are comments, commentStart points at the # character that begins the first comment, and commentSize is how many bytes all the comments before the command take up — including the newline character that terminates the last comment.
• if the command has no comments before it? commentSize is just 0. Nothing to see here. 🫶
• commandStart points to the first character of the first word in the command — skipping any comments and leading space, bestie, we're getting straight to business.
• commandSize is the total number of bytes in the command, from the character that commandStart points at, up to and including the newline, close bracket, or semicolon character that terminates the command.
• numWords gives the total number of words in the command. Count 'em. 🔥

All parsing procedures set the remaining fields, tokenPtr and numTokens. So:

• tokenPtr points to the first in an array of Tcl_Token structures that describe the components of whatever entity is being parsed.
• numTokens gives the total number of tokens present in that array.

Every token has four fields, no more no less:

• type picks one of the several token types described down below 👇
• start points to the first character in the token
• size gives the total number of characters in the token
• numComponents — some token types, like TCL_TOKEN_WORD and TCL_TOKEN_VARIABLE, are made of several component tokens that immediately follow the parent token, and this field says how many of those there are.

And type has one of these values, periodt:

• TCL_TOKEN_WORD — this token ordinarily describes one word of a command, but it can also describe a quoted or braced string in an expression. It describes a component of the script that's the result of concatenating a sequence of subcomponents together, each one described by a separate subtoken. The token starts with the first non-blank character of the component (which might be a double-quote or an open brace) and includes all the characters in the component up to but not including the space, semicolon, close bracket, close quote, or close brace that terminates the component. The numComponents field counts the total number of sub-tokens that make up the word — and yes, that includes sub-tokens of TCL_TOKEN_VARIABLE and TCL_TOKEN_BS tokens. 🧚
• TCL_TOKEN_SIMPLE_WORD — same energy as TCL_TOKEN_WORD, except this word is guaranteed to consist of a single TCL_TOKEN_TEXT sub-token. The numComponents field is always 1. No cap, just one. ✨
• TCL_TOKEN_EXPAND_WORD — same meaning as TCL_TOKEN_WORD, except the command parser clocked that this word began with the expansion prefix {*}, which means after substitution, the list value of this word should be expanded to form multiple arguments in command evaluation. Iconic. This token type can only be created by Tcl_ParseCommand — nobody else gets to make it. 👑
• TCL_TOKEN_TEXT — describes a range of literal text that's part of a word. The numComponents field is always 0. 🎀
• TCL_TOKEN_BS — describes a backslash sequence such as \n or \0xA3. The numComponents field is always 0. 🦋
• TCL_TOKEN_COMMAND — describes a command whose result must be substituted into the word. The token includes the square brackets that surround the command. The numComponents field is always 0 — the nested command is not parsed, so if you actually want to see its tokens you gotta call Tcl_ParseCommand recursively yourself, bestie. 🙈
• TCL_TOKEN_VARIABLE — describes a variable substitution, including the $, the variable name, and the array index (if there is one) up through the close parenthesis that terminates the index. This token is followed by one or more additional tokens that describe the variable name and array index. So:
  • if numComponents is 1, the variable is a scalar and the next token is a TCL_TOKEN_TEXT token giving the variable name.
  • if numComponents is greater than 1, the variable is an array: the first sub-token is a TCL_TOKEN_TEXT token giving the array name, and the remaining sub-tokens are TCL_TOKEN_TEXT, TCL_TOKEN_BS, TCL_TOKEN_COMMAND, and TCL_TOKEN_VARIABLE tokens that must be concatenated to produce the array index.
  • and the numComponents field includes nested sub-tokens that are part of TCL_TOKEN_VARIABLE tokens in the array index. It's giving nested, fr fr. 🌸
• TCL_TOKEN_SUB_EXPR — describes one subexpression of an expression (or an entire expression, main character energy). A subexpression may consist of a value such as an integer literal, variable substitution, or parenthesized subexpression; it may also consist of an operator and its operands. The token starts with the first non-blank character of the subexpression, up to but not including the space, brace, close-paren, or bracket that terminates the subexpression. This token is followed by one or more additional tokens that describe the subexpression:
  • if the first sub-token after the TCL_TOKEN_SUB_EXPR token is a TCL_TOKEN_OPERATOR token, then the subexpression consists of an operator and its token operands. If the operator has no operands, the subexpression consists of just the TCL_TOKEN_OPERATOR token. Each operand is described by a TCL_TOKEN_SUB_EXPR token.
  • otherwise, the subexpression is a value described by one of the token types TCL_TOKEN_WORD, TCL_TOKEN_TEXT, TCL_TOKEN_BS, TCL_TOKEN_COMMAND, TCL_TOKEN_VARIABLE, and TCL_TOKEN_SUB_EXPR.
  • the numComponents field counts the total number of sub-tokens that make up the subexpression, and that includes the sub-tokens for any nested TCL_TOKEN_SUB_EXPR tokens. Deep. 💜
• TCL_TOKEN_OPERATOR — describes one operator of an expression, such as && or hypot. A TCL_TOKEN_OPERATOR token is always preceded by a TCL_TOKEN_SUB_EXPR token that describes the operator and its operands, and you can use that TCL_TOKEN_SUB_EXPR token's numComponents field to figure out how many operands there are. The tea:
  • a binary operator such as * is followed by two TCL_TOKEN_SUB_EXPR tokens describing its operands.
  • a unary operator like - is followed by a single TCL_TOKEN_SUB_EXPR token for its operand.
  • if the operator is a math function such as log10, the TCL_TOKEN_OPERATOR token gives its name and the following TCL_TOKEN_SUB_EXPR tokens describe its operands; if there are no operands (as with rand), no TCL_TOKEN_SUB_EXPR tokens follow.
  • there is exactly one trinary operator, ?, that shows up in if-then-else subexpressions such as x?y:z; in this case the ? TCL_TOKEN_OPERATOR token is followed by three TCL_TOKEN_SUB_EXPR tokens for the operands x, y, and z.
  • the numComponents field for a TCL_TOKEN_OPERATOR token is always 0. ⭐

---

🦋 what comes back after each parse call 🦋

After Tcl_ParseCommand returns, the first token pointed to by the tokenPtr field of the Tcl_Parse structure always has type TCL_TOKEN_WORD or TCL_TOKEN_SIMPLE_WORD or TCL_TOKEN_EXPAND_WORD. It's followed by the sub-tokens that must be concatenated to produce the value of that word. Then the next token is the TCL_TOKEN_WORD or TCL_TOKEN_SIMPLE_WORD of TCL_TOKEN_EXPAND_WORD token for the second word, followed by sub-tokens for that word, and so on until all numWords have been accounted for. One by one, no skipping. 💀

After Tcl_ParseExpr returns, the first token pointed to by the tokenPtr field of the Tcl_Parse structure always has type TCL_TOKEN_SUB_EXPR. It's followed by the sub-tokens that must be evaluated to produce the value of the expression. And heads up: only the token information in the Tcl_Parse structure is modified — the commentStart, commentSize, commandStart, and commandSize fields are not modified by Tcl_ParseExpr. Hands off. 🙅‍♀️

After Tcl_ParseBraces returns, the array of tokens pointed to by the tokenPtr field of the Tcl_Parse structure will contain a single TCL_TOKEN_TEXT token if the braced string doesn't contain any backslash-newlines. But if the string does contain backslash-newlines, the array of tokens will contain one or more TCL_TOKEN_TEXT or TCL_TOKEN_BS sub-tokens that must be concatenated to produce the value of the string. And if the braced string was just {} (that is, the string was empty), the single TCL_TOKEN_TEXT token will have a size field containing zero — this ensures that at least one token appears to describe the braced string, even when there's literally nothing in there. Representation matters. 🎀

Same deal as before: only the token information in the Tcl_Parse structure is modified — the commentStart, commentSize, commandStart, and commandSize fields are not modified by Tcl_ParseBraces. 🌈

Okay so after Tcl_ParseQuotedString returns, the array of tokens pointed to by the tokenPtr field is fully dependent on what's inside that quoted string. It's gonna consist of one or more TCL_TOKEN_TEXT, TCL_TOKEN_BS, TCL_TOKEN_COMMAND, and TCL_TOKEN_VARIABLE sub-tokens. 💅 And no cap: the array always contains at least one token — like, if the start argument is empty, the array returned consists of a single TCL_TOKEN_TEXT token with a zero size field. Only the token information in the Tcl_Parse structure gets modified; commentStart, commentSize, commandStart, and commandSize are literally not modified. ✨

Then when Tcl_ParseVarName returns, the first token pointed to by the tokenPtr field of the Tcl_Parse structure always has type TCL_TOKEN_VARIABLE — main character energy only. It's followed by the sub-tokens that make up the variable name as described above, and the total length of the variable name is contained in the size field of that first token. Same as in Tcl_ParseExpr: only the token information in the Tcl_Parse structure is modified by Tcl_ParseVarName, and the commentStart, commentSize, commandStart, and commandSize fields are not modified. Periodt. 🦋

Every single character pointer in the Tcl_Parse and Tcl_Token structures refers to characters in the start argument that got passed to Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName. Like, all of them, bestie. 🌸

There are additional fields in the Tcl_Parse structure after the numTokens field, but highkey those are for the private use of Tcl_ParseCommand, Tcl_ParseExpr, Tcl_ParseBraces, Tcl_ParseQuotedString, and Tcl_ParseVarName. They should not be referenced by code outside of these procedures — that's a private group chat, iykyk. 🙈

---

🔑 KEYWORDS 🔑

• backslash substitution
• braces
• command
• expression
• parse
• token
• variable substitution

---

💖 That's the whole page, bestie — footer vibe check: Tcl 8.3, Tcl_ParseCommand(3), with source, version, and date all snuggled into that little line. Slay. 🎀