pt::pgen(n)

✨ pt::pgen — Parser Generator ✨

okay so bestie, welcome to pt::pgen 💖 it lives inside Parser Tools (that's the manual you're holding rn) and it's genuinely one of the most iconic little commands in the whole toolkit. no cap.

---

🌸 SYNOPSIS 🌸

package require Tcl 8.5 9

package require pt::pgen ?1.2?

::pt::pgen inputformat text resultformat ?options...?

---

💅 DESCRIPTION 💅

Are you lost ? Do you have trouble understanding this document ? 😭 if that's you, pls go read the overview in the Introduction to Parser Tools first — that one's the entrypoint to the whole system this package is part of. fr fr, it'll help.

So here's the vibe: this package gives you a command that implements a parser generator — you feed it parsing expression grammars and it gives you a parser back. that's the whole thing. slay.

It's the implementation of method generate of pt, aka the Parser Tools Application. so the intended audience here is people who want to modify and/or extend this part of pt's functionality. If you're just a user of pt? bestie, go read the application's manpage instead — the Parser Tools Application one. 🫶

It resides in the User Package Layer of Parser Tools.

IMAGE: arch_user_pkg

---

🎀 API 🎀

the one command

• ::pt::pgen inputformat text resultformat ?options...? — takes the parsing expression grammar sitting in text (formatted however inputformat says), and hands you back that exact same grammar in the resultformat format, as the result of the command. ✨

The two input formats we know about are peg and json. Introductions to them — formal specifications included — live in the PEG Language Tutorial and The JSON Grammar Exchange Format. The packages doing the actual parsing of these formats are:

• peg — pt::peg::from::peg
• json — pt::peg::from::json

On the output side, the known formats and the packages that generate them are:

• c — pt::peg::to::cparam
• container — pt::peg::to::container
• critcl — pt::peg::to::cparam + pt::cparam::configuration::critcl
• json — pt::peg::to::json
• oo — pt::peg::to::tclparam + pt::tclparam::configuration::tcloo
• peg — pt::peg::to::peg
• snit — pt::peg::to::tclparam + pt::tclparam::configuration::snit

The options supported by each of these formats are documented with their respective packages. go check those out if you need the deets 💜

---

🦋 EXAMPLE 🦋

okay this section is the main character moment — we're doing a complete example, starting from a PEG grammar and ending with us running the parser we generated over some input, following the outline in the figure below:

IMAGE: flow

Our grammar, assumed to be stored in the file "calculator.peg", is:

PEG calculator (Expression)
    Digit      <- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9'       ;
    Sign       <- '-' / '+'                                     ;
    Number     <- Sign? Digit+                                  ;
    Expression <- Term (AddOp Term)*                            ;
    MulOp      <- '*' / '/'                                     ;
    Term       <- Factor (MulOp Factor)*                        ;
    AddOp      <- '+'/'-'                                       ;
    Factor     <- '(' Expression ')' / Number                   ;
END;

From this we create a snit-based parser using the script "gen":

package require Tcl 8.5
package require fileutil
package require pt::pgen

lassign $argv name
set grammar [fileutil::cat $name.peg]
set pclass  [pt::pgen peg $gr snit -class $name -file  $name.peg -name  $name]
fileutil::writeFile $name.tcl $pclass
exit 0

and we call it like:

tclsh8.5 gen calculator

which leaves us with the parser package and class written to the file "calculator.tcl". 🎉 Assuming that package is then properly installed somewhere Tcl can find it, we can now use this class via a script like:

package require calculator

lassign $argv input
set channel [open $input r]

set parser [calculator]
set ast [$parser parse $channel]
$parser destroy
close $channel

... now process the returned abstract syntax tree ...

where the abstract syntax tree stored in the variable will look like:

set ast {Expression 0 4
    {Factor 0 4
        {Term 0 2
            {Number 0 2
                {Digit 0 0}
                {Digit 1 1}
                {Digit 2 2}
            }
        }
        {AddOp 3 3}
        {Term 4 4
            {Number 4 4
                {Digit 4 4}
            }
        }
    }
}

assuming the input file and channel contained the text:

 120+5

A more graphical representation of the tree would be:

.nf +- Digit 0 0 | 1 |            | +- Term 0 2 --- Number 0 2 -+- Digit
1 1 | 2 |                           |            | |
+- Digit 2 2 | 0 |                                        | Expression 0
4 --- Factor 0 4 -+----------------------------- AddOp 3 3 | + |
| +- Term 4 4 --- Number 4 4 --- Digit 4 4 | 5 .fi

ngl that graphical tree is a little cursed-looking in the raw doc, but that's exactly how it's printed — reproducing it faithfully, no edits from me. 💀

Regardless, at this point it's the user's responsibility to work with the tree to reach whatever goal she desires. i.e. analyze it, transform it, etc. The package pt::ast should be of help here — it gives you commands to walk such AST structures in various ways. 🫶

One important thing to note: the parsers used here return a data structure representing the structure of the input per the grammar underlying the parser. There are no callbacks during the parsing process, i.e. no parsing actions, as most other parsers will have. so don't go looking for them. 🙈

Going back to the last snippet of code — the execution of the parser for some input — note how the parser instance follows the specified Parser API. periodt.

---

💀 BUGS, IDEAS, FEEDBACK 💀

This document, and the package it describes, will undoubtedly contain bugs and other problems. (yes bestie, they said it themselves, not me 😭) Please report such in the category pt of the Tcllib Trackers: http://core.tcl.tk/tcllib/reportlist. Please also report any ideas for enhancements you may have for either the package and/or the documentation. we want them. give us them. 💖

When proposing code changes, please provide unified diffs, i.e the output of diff -u.

Note further that attachments are strongly preferred over inlined patches. Attachments can be made by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar. iykyk 🎀

---

⭐ KEYWORDS ⭐

EBNF, LL(k), PEG, TDPL, context-free languages, expression, grammar, matching, parser, parsing expression, parsing expression grammar, push down automaton, recursive descent, state, top-down parsing languages, transducer

---

👑 CATEGORY 👑

Parsing and Grammars

---

🌈 COPYRIGHT 🌈

Copyright (c) 2009 Andreas Kupries <andreas_kupries@users.sourceforge.net>

---

okay that's the whole page bestie 🫶 this one came out of tcllib, version 1.2, as part of the pt::pgen(n) manual — go forth and generate some parsers, you've got this 💅✨