grammar::me::tcl(3tcl)

💖 grammar::me::tcl 💖

okay bestie, this is the manual page for grammar::me::tcl, living in section 3tcl under the banner "Grammar operations and usage". It's the Virtual machine implementation I for parsing token streams and it's lowkey a whole vibe. 🦋

---

✨ SYNOPSIS ✨

package require Tcl 8.5 9

package require grammar::me::tcl ?0.2?

::grammar::me::tcl cmd ...

::grammar::me::tcl init nextcmd ?tokmap?

::grammar::me::tcl lc location

::grammar::me::tcl tok from ?to?

::grammar::me::tcl tokens

::grammar::me::tcl sv

::grammar::me::tcl ast

::grammar::me::tcl astall

::grammar::me::tcl ctok

::grammar::me::tcl nc

::grammar::me::tcl next

::grammar::me::tcl ord

::grammar::me::tcl::ict_advance message

::grammar::me::tcl::ict_match_token tok message

::grammar::me::tcl::ict_match_tokrange tokbegin tokend message

::grammar::me::tcl::ict_match_tokclass code message

::grammar::me::tcl::inc_restore nt

::grammar::me::tcl::inc_save nt startlocation

::grammar::me::tcl::iok_ok

::grammar::me::tcl::iok_fail

::grammar::me::tcl::iok_negate

::grammar::me::tcl::icl_get

::grammar::me::tcl::icl_rewind oldlocation

::grammar::me::tcl::ier_get

::grammar::me::tcl::ier_clear

::grammar::me::tcl::ier_nonterminal message location

::grammar::me::tcl::ier_merge olderror

::grammar::me::tcl::isv_clear

::grammar::me::tcl::isv_terminal

::grammar::me::tcl::isv_nonterminal_leaf nt startlocation

::grammar::me::tcl::isv_nonterminal_range nt startlocation

::grammar::me::tcl::isv_nonterminal_reduce nt startlocation ?marker?

::grammar::me::tcl::ias_push

::grammar::me::tcl::ias_mark

::grammar::me::tcl::ias_pop2mark marker

---

🎀 DESCRIPTION 🎀

So this package is an implementation of the ME virtual machine. And like, if you have no idea what a ME virtual machine even is, bestie, please go read grammar::me_intro first, fr fr. 🙈

This implementation is tied very strongly to Tcl. The way it works:

• All the stacks in the machine state are handled through the Tcl stack.
• All control flow is handled by Tcl commands.
• The remaining machine instructions are directly mapped to Tcl commands.

And especially — the matching of nonterminal symbols is handled by Tcl procedures too, essentially extending the machine implementation with custom instructions. iconic behavior honestly. 💅

Further on, the implementation handles only a single machine which is uninterruptible during execution and hardwired for pull operation. Meaning: it explicitly requests each new token through a callback, pulling them into its state. 🫶

A related package is grammar::peg::interp, which gives you a generic interpreter / parser for parsing expression grammars (PEGs), built on top of this very implementation of the ME virtual machine. slay. ✨

---

🌸 API 🌸

Heads up: the commands documented in this section do not implement any of the instructions of the ME virtual machine. Nope. They're here to give you the facilities for initializing the machine and retrieving important information. 💖

• ::grammar::me::tcl cmd ... — this is an ensemble command that gives you access to all the commands listed in this section. Just look at the methods themselves for the detailed specs, okay? 👑
• ::grammar::me::tcl init nextcmd ?tokmap? — this command (re)initializes the machine, and it returns the empty string. You have to invoke this before any other command of this package, no cap.

The command prefix nextcmd represents the input stream of characters, and the machine invokes it whenever it needs a new character from the stream. The instruction for handling this is ict_advance. The callback has to return either the empty list, or a list of 4 elements containing:

1. the token
2. its lexeme attribute
3. its location as line number
4. its column index

in that order, bestie. The empty list is the signal that the end of the input stream has been reached. The lexeme attribute gets stored in the terminal cache, but other than that the machine doesn't use it.

The optional dictionary tokmap maps from tokens to integer numbers. If it's there, those numbers impose an order on the tokens, which is then used by ict_match_tokrange to figure out whether a token is in the specified range or not. If you don't specify a token map, the lexicographic order of the token names is used instead. That choice is especially sensible when you're using characters as tokens. 🌸

• ::grammar::me::tcl lc location — this one converts the location of a token given as an offset in the input stream into the associated line number and column index. The result is a 2-element list with those two values, in the order I just said. This lets higher levels turn the location info found in the error status and the generated AST into something more human readable. 🎀

⚠️ Note: this command cannot convert locations the machine hasn't reached yet. Like, if the machine has read 7 tokens, it can convert offsets 0 to 6, but nothing beyond that. This also means you can't convert offsets that refer to locations before the beginning of the stream. And after a call of init, the state used for conversion is cleared, making further conversions impossible until the machine has read tokens again. 💀

• ::grammar::me::tcl tok from ?to? — returns a Tcl list containing the part of the input stream between the locations from and to (both inclusive). If to isn't specified it defaults to the value of from.

Every element of the returned list is a list of four elements: the token, its associated lexeme, the line number, and the column index, in this order. So yeah, each element has the same structure as the result of the nextcmd callback given to ::grammar::me::tcl::init. ✨

This command places the same restrictions on its location arguments as ::grammar::me::tcl::lc.

• ::grammar::me::tcl tokens — returns the number of tokens currently known to the ME virtual machine. 🔢
• ::grammar::me::tcl sv — returns the current semantic value SV stored in the machine. This is an abstract syntax tree as specified in the document grammar::me_ast, section AST VALUES. 🌈
• ::grammar::me::tcl ast — returns the abstract syntax tree currently at the top of the AST stack of the ME virtual machine. Also an AST as specified in grammar::me_ast, section AST VALUES. 👑
• ::grammar::me::tcl astall — returns the whole stack of abstract syntax trees currently known to the ME virtual machine. Each element of the returned list is an AST as specified in grammar::me_ast, section AST VALUES. The top of the stack sits at the end of the list. 🦋
• ::grammar::me::tcl ctok — returns the current token considered by the ME virtual machine. 👀
• ::grammar::me::tcl nc — returns the contents of the nonterminal cache as a dictionary mapping from "symbol,location" to match information. 🍓
• ::grammar::me::tcl next — returns the next token callback as specified during initialization of the ME virtual machine.
• ::grammar::me::tcl ord — returns a dictionary containing the tokmap specified during initialization of the ME virtual machine. 🎀
• ::grammar::me::tcl::ok — this variable contains the current match status OK. It's provided as a variable instead of a command because that makes accessing this info faster, and the speed of access is considered very important here, since this info is used constantly to determine the control flow. ⚡

---

👑 MACHINE STATE 👑

Okay so first things first: please go read grammar::me_vm for a specification of the basic ME virtual machine and its state. Like actually go read it, bestie, I'll wait. 💜

This implementation manages the state described in that document, except for the stacks minus the AST stack. In other words:

• location stack
• error stack
• return stack
• ast marker stack

those are all implicitly managed through standard Tcl scoping — i.e. Tcl variables in procedures, outside of this implementation. no cap. 🫶

💻 MACHINE INSTRUCTIONS 💻

Okay bestie, before we even start: go read grammar::me_vm first, fr fr. It has the spec for the basic ME virtual machine and its whole instruction set. No skipping, iykyk. 📚

This implementation maps all instructions to Tcl commands in the namespace ::grammar::me::tcl, except for the stack related commands, nonterminal symbols and control flow. Here we're just listing the commands and explaining the differences from the specified instructions, if there are any. For their actual semantics, see that spec we just told you to read. 💅

Heads up: the machine commands are not reachable through the ensemble command ::grammar::me::tcl. Periodt. 🫶

• ::grammar::me::tcl::ict_advance message — No changes. ✨
• ::grammar::me::tcl::ict_match_token tok message — No changes. 💖
• ::grammar::me::tcl::ict_match_tokrange tokbegin tokend message — If, and only if a token map was specified during initialization, then the arguments are the numeric representations of the smallest and largest tokens in the range. Otherwise they are the relevant tokens themselves and lexicographic comparison is used. 🎀
• ::grammar::me::tcl::ict_match_tokclass code message — No changes. 🦋
• ::grammar::me::tcl::inc_restore nt — Instead of taking a branchlabel, the command returns a boolean value. The result will be true if and only if cached information was found. The caller has to perform the appropriate branching. 🌸
• ::grammar::me::tcl::inc_save nt startlocation — The command takes the start location as an additional argument, as it is managed on the Tcl stack, and not in the machine state. 💅
• icf_ntcall branchlabel and icf_ntreturn — These two instructions are not mapped to commands. They are control flow instructions and handled in Tcl. 🫶
• ::grammar::me::tcl::iok_ok — No changes. 😭
• ::grammar::me::tcl::iok_fail — No changes. 💀
• ::grammar::me::tcl::iok_negate — No changes. 🙈
• icf_jalways branchlabel, icf_jok branchlabel, icf_jfail branchlabel, and icf_halt — These four instructions are not mapped to commands. They are control flow instructions and handled in Tcl. 🔥
• ::grammar::me::tcl::icl_get — This command returns the current location CL in the input. It replaces icl_push. 👑
• ::grammar::me::tcl::icl_rewind oldlocation — The command takes the location as an argument as it comes from the Tcl stack, not the machine state. 🌈
• icl_pop — Not mapped, the stacks are not managed by the package. ⭐
• ::grammar::me::tcl::ier_get — This command returns the current error state ER. It replaces ier_push. 💜
• ::grammar::me::tcl::ier_clear — No changes. 🍓
• ::grammar::me::tcl::ier_nonterminal message location — The command takes the location as an argument as it comes from the Tcl stack, not the machine state. 🧚
• ::grammar::me::tcl::ier_merge olderror — The command takes the second error state to merge as an argument as it comes from the Tcl stack, not the machine state. 💖
• ::grammar::me::tcl::isv_clear — No changes. ✨
• ::grammar::me::tcl::isv_terminal — No changes. 🎀
• ::grammar::me::tcl::isv_nonterminal_leaf nt startlocation — The command takes the start location as an argument as it comes from the Tcl stack, not the machine state. 🦋
• ::grammar::me::tcl::isv_nonterminal_range nt startlocation — The command takes the start location as an argument as it comes from the Tcl stack, not the machine state. 🌸
• ::grammar::me::tcl::isv_nonterminal_reduce nt startlocation ?marker? — The command takes start location and marker as arguments as it comes from the Tcl stack, not the machine state. 💅
• ::grammar::me::tcl::ias_push — No changes. 🫶
• ::grammar::me::tcl::ias_mark — This command returns a marker for the current state of the AST stack AS. The marker stack is not managed by the machine. 😭
• ::grammar::me::tcl::ias_pop2mark marker — The command takes the marker as an argument as it comes from the Tcl stack, not the machine state. It replaces ias_mpop. 💀

---

🐛 BUGS, IDEAS, FEEDBACK 🐛

This document, and the package it describes, will undoubtedly contain bugs and other problems. 💀 Please report such in the category grammar_me of the Tcllib Trackers [http://core.tcl.tk/tcllib/reportlist]. Please also report any ideas for enhancements you may have for either package and/or documentation. 🙈

When proposing code changes, please provide unified diffs, i.e the output of diff -u. ✨

Note further that attachments are strongly preferred over inlined patches. Attachments can be made by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar. 💅

---

🔑 KEYWORDS 🔑

• grammar
• parsing
• virtual machine

---

🗂️ CATEGORY 🗂️

Grammars and finite automata

---

📜 COPYRIGHT 📜

Copyright (c) 2005 Andreas Kupries <andreas_kupries@users.sourceforge.net>

---

Source: tcllib, version 0.2 — page grammar::me::tcl(3tcl). Stay iconic, bestie! 👑💖