pt::peg::to::param(n)

๐Ÿ“ฆ
version 2.0 ยท 2 versions
, in fedora 43, fedora 44, nixos 26.05, nixpkgs unstable

okay so bestie, this is pt::peg::to::param ๐Ÿ’– โ€” a page from the Parser Tools manual (the (n) is just the manual section, iykyk). She's a converter, she's book smart, and she's about to make PARAM markup appear out of thin air. Let's get into it.


๐ŸŒธ NAME ๐ŸŒธ

pt::peg::to::param โ€” PEG Conversion. Write PARAM format โœจ


โœจ SYNOPSIS โœจ

package require Tcl 8.5 9

package require pt::peg::to::param ?1.1?

package require pt::peg

package require pt::pe

pt::peg::to::param reset

pt::peg::to::param configure

pt::peg::to::param configure option

pt::peg::to::param configure option value...

pt::peg::to::param convert serial

๐Ÿฆ‹ DESCRIPTION ๐Ÿฆ‹

bestie, are you lost?? ๐Ÿ˜ญ Having trouble understanding this document? Say less โ€” go read the overview in the Introduction to Parser Tools. That doc is the entrypoint to the whole entire system this package is part of, no cap. ๐Ÿซถ

So what is this package, actually? It implements the converter from parsing expression grammars to PARAM markup. That's the whole gig. โœจ

Where does she live? In the Export section of the Core Layer of Parser Tools, and you can use her two different ways:

  • directly with the other packages of this layer, or
  • indirectly through the export manager provided by pt::peg::export.

That second route? It's intented for use in untrusted environments ๐Ÿ™ˆ and it goes through the corresponding export plugin pt::peg::export::param, which sits between the converter and the export manager like the world's most responsible group-chat moderator. ๐Ÿ‘‘

๐Ÿ“ธ the doc drops an image reference here: IMAGE: arch_core_eplugins

๐Ÿ’… API ๐Ÿ’…

The API this package hands you satisfies the spec of the Converter API found in the Parser Tools Export API specification. It's giving certified. ๐ŸŽ€

  • pt::peg::to::param reset โ€” this command resets the configuration of the package to its default settings. fresh-start energy, we love a clean slate ๐Ÿงผ
  • pt::peg::to::param configure โ€” this command returns a dictionary containing the current configuration of the package. it's just telling you what's going on ๐Ÿ’ฌ
  • pt::peg::to::param configure option โ€” this command returns the current value of the specified configuration option of the package. wondering which options are even legal? read the Options section, bestie ๐Ÿ‘€
  • pt::peg::to::param configure option value... โ€” this command sets the given configuration options of the package to the specified values. again: for the set of legal options, please read the Options section.
  • pt::peg::to::param convert serial โ€” this command takes the canonical serialization of a parsing expression grammar (as specified in the PEG serialization format section, contained in serial) and generates PARAM markup encoding the grammar, per the current package configuration. the created string is then returned as the result of the command. slay ๐Ÿ’…

๐ŸŽ€ OPTIONS ๐ŸŽ€

The converter to PARAM markup recognizes the following configuration variables and changes its behaviour as they specify. ๐Ÿ‘‘

  • -template string โ€” the value of this configuration variable is a string into which to put the generated text and the other configuration settings. the various locations for user-data are expected to be specified with the placeholders listed below. the default value is "@code@".
    • @user@ โ€” to be replaced with the value of the configuration variable -user.
    • @format@ โ€” to be replaced with the constant PARAM.
    • @file@ โ€” to be replaced with the value of the configuration variable -file.
    • @name@ โ€” to be replaced with the value of the configuration variable -name.
    • @code@ โ€” to be replaced with the generated text.
  • -name string โ€” the value of this configuration variable is the name of the grammar for which the conversion is run. the default value is a_pe_grammar.
  • -user string โ€” the value of this configuration variable is the name of the user for which the conversion is run. the default value is unknown.
  • -file string โ€” the value of this configuration variable is the name of the file or other entity from which the grammar came, for which the conversion is run. the default value is unknown.

๐ŸŒˆ PARAM CODE REPRESENTATION OF PARSING EXPRESSION GRAMMARS ๐ŸŒˆ

The PARAM code representation of parsing expression grammars is assembler-like text using the instructions of the virtual machine documented in the PackRat Machine Specification, plus a few more for control flow: jump ok, jump fail, call symbol, return. ๐Ÿงš

ngl, it is not really useful โ€” except possibly as a tool demonstrating how a grammar is compiled in general, without getting distracted by the incidentals of a framework, i.e. like the supporting C and Tcl code generated by the other PARAM-derived formats. ๐Ÿ’€

It has no direct formal specification beyond what was said above. periodt. โœจ

๐Ÿ“ EXAMPLE ๐Ÿ“

Assuming the following PEG for simple mathematical expressions:

       PEG calculator (Expression)
           Digit      <- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9'       ;
           Sign       <- '-' / '+'                                     ;
           Number     <- Sign? Digit+                                  ;
           Expression <- Term (AddOp Term)*                            ;
           MulOp      <- '*' / '/'                                     ;
           Term       <- Factor (MulOp Factor)*                        ;
           AddOp      <- '+'/'-'                                       ;
           Factor     <- '(' Expression ')' / Number                   ;
       END;

one possible PARAM serialization for it is:

       # -*- text -*-
       # Parsing Expression Grammar 'TEMPLATE'.
       # Generated for unknown, from file 'TEST'

       #
       # Grammar Start Expression
       #

       <<MAIN>>:
                call              sym_Expression
                halt

       #
       # value Symbol 'AddOp'
       #

       sym_AddOp:
       # /
       #     '-'
       #     '+'

                symbol_restore    AddOp
         found! jump              found_7
                loc_push

                call              choice_5

          fail! value_clear
            ok! value_leaf        AddOp
                symbol_save       AddOp
                error_nonterminal AddOp
                loc_pop_discard

       found_7:
            ok! ast_value_push
                return

       choice_5:
       # /
       #     '-'
       #     '+'

                error_clear

                loc_push
                error_push

                input_next        "t -"
            ok! test_char         "-"

                error_pop_merge
            ok! jump              oknoast_4

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t +"
            ok! test_char         "+"

                error_pop_merge
            ok! jump              oknoast_4

                loc_pop_rewind
                status_fail
                return

       oknoast_4:
                loc_pop_discard
                return
       #
       # value Symbol 'Digit'
       #

       sym_Digit:
       # /
       #     '0'
       #     '1'
       #     '2'
       #     '3'
       #     '4'
       #     '5'
       #     '6'
       #     '7'
       #     '8'
       #     '9'

                symbol_restore    Digit
         found! jump              found_22
                loc_push

                call              choice_20

          fail! value_clear
            ok! value_leaf        Digit
                symbol_save       Digit
                error_nonterminal Digit
                loc_pop_discard

       found_22:
            ok! ast_value_push
                return

       choice_20:
       # /
       #     '0'
       #     '1'
       #     '2'
       #     '3'
       #     '4'
       #     '5'
       #     '6'
       #     '7'
       #     '8'
       #     '9'

                error_clear

                loc_push
                error_push

                input_next        "t 0"
            ok! test_char         "0"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 1"
            ok! test_char         "1"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 2"
            ok! test_char         "2"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 3"
            ok! test_char         "3"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 4"
            ok! test_char         "4"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 5"
            ok! test_char         "5"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 6"
            ok! test_char         "6"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 7"
            ok! test_char         "7"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 8"
            ok! test_char         "8"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                loc_push
                error_push

                input_next        "t 9"
            ok! test_char         "9"

                error_pop_merge
            ok! jump              oknoast_19

                loc_pop_rewind
                status_fail
                return

(this example keeps going, bestie โœจ)

              oknoast_19:
                       loc_pop_discard
                       return
              #
              # value Symbol 'Expression'
              #

              sym_Expression:
              # /
              #     x
              #         '\('
              #         (Expression)
              #         '\)'
              #     x
              #         (Factor)
              #         *
              #             x
              #                 (MulOp)
              #                 (Factor)

                       symbol_restore    Expression
                found! jump              found_46
                       loc_push
                       ast_push

                       call              choice_44

                 fail! value_clear
                   ok! value_reduce      Expression
                       symbol_save       Expression
                       error_nonterminal Expression
                       ast_pop_rewind
                       loc_pop_discard

              found_46:
                   ok! ast_value_push
                       return

okay so bestie, that first label oknoast_19 is literally just wrapping up a rule from the part before this one โ€” it pops the location and returns, that's the whole vibe ๐Ÿ’…

then we get to the main character of this chunk: sym_Expression ๐ŸŽ€ and see how those # comment lines right under it are literally the grammar rule written out? like / = ordered choice, x = sequence, * = star/kleene, ? = optional โ€” it's the recipe, and the assembly below is the recipe cooked fr fr โœจ

the way that symbol_restore Expression + found! jump found_46 works is giving memoization energy โ€” if we already built this nonterminal, we jump straight to found_46, hit ast_value_push, and bounce ๐Ÿฆ‹ no re-parsing, obsessed.

and if it isn't cached? we loc_push, ast_push, call choice_44, and then the outcome branches:

  • on failure โ†’ value_clear and unwind
  • on success โ†’ value_reduce Expression, symbol_save Expression, error_nonterminal Expression, ast_pop_rewind, loc_pop_discard ๐ŸŒธ

no cap, that's the whole pattern. it repeats for literally every nonterminal in this file and honestly? iconic consistency ๐Ÿ’œ


now choice_44 is the ordered-choice dispatch โ€” it saves state, tries sequence_27 first, and if that dies it rewinds and tries sequence_40 instead, then finally gives up with status_fail if both flop ๐Ÿ˜ญ

              choice_44:
              # /
              #     x
              #         '\('
              #         (Expression)
              #         '\)'
              #     x
              #         (Factor)
              #         *
              #             x
              #                 (MulOp)
              #                 (Factor)

                       error_clear

                       ast_push
                       loc_push
                       error_push

                       call              sequence_27

                       error_pop_merge
                   ok! jump              ok_43

                       ast_pop_rewind
                       loc_pop_rewind
                       ast_push
                       loc_push
                       error_push

                       call              sequence_40

                       error_pop_merge
                   ok! jump              ok_43

                       ast_pop_rewind
                       loc_pop_rewind
                       status_fail
                       return

              ok_43:
                       ast_pop_discard
                       loc_pop_discard
                       return

lowkey the ok_43 landing pad is so clean โ€” ast_pop_discard, loc_pop_discard, return, done ๐Ÿซถ


here's sequence_27 โ€” that's the parenthesized branch, '\(' (Expression) '\)' โ€” it literally does test_char "(", calls sym_Expression, then test_char ")" ๐Ÿงš and sequence_40 is the other branch: (Factor) * (MulOp) (Factor), so sym_Factor then kleene_37 on repeat ๐Ÿ’ซ

              sequence_27:
              # x
              #     '\('
              #     (Expression)
              #     '\)'

                       loc_push
                       error_clear

                       error_push

                       input_next        "t ("
                   ok! test_char         "("

                       error_pop_merge
                 fail! jump              failednoast_29
                       ast_push
                       error_push

                       call              sym_Expression

                       error_pop_merge
                 fail! jump              failed_28
                       error_push

                       input_next        "t )"
                   ok! test_char         ")"

                       error_pop_merge
                 fail! jump              failed_28

                       ast_pop_discard
                       loc_pop_discard
                       return

              failed_28:
                       ast_pop_rewind

              failednoast_29:
                       loc_pop_rewind
                       return

ngl the failed_28 / failednoast_29 two-step cleanup is giving parallel-universe energy โ€” one path had an AST pushed and needs ast_pop_rewind, the other never got that far and only needs loc_pop_rewind ๐Ÿ™ˆ like yes queen, know your unwind depth.

              sequence_40:
              # x
              #     (Factor)
              #     *
              #         x
              #             (MulOp)
              #             (Factor)

                       ast_push
                       loc_push
                       error_clear

                       error_push

                       call              sym_Factor

                       error_pop_merge
                 fail! jump              failed_41
                       error_push

                       call              kleene_37

                       error_pop_merge
                 fail! jump              failed_41

                       ast_pop_discard
                       loc_pop_discard
                       return

              failed_41:
                       ast_pop_rewind
                       loc_pop_rewind
                       return

then kleene_37 is the star loop โ€” (MulOp) (Factor) zero or more times ๐Ÿ” and sequence_34 is a single (MulOp) (Factor) pair. watch the loop: it calls sequence_34, and on failure it jumps to failed_38, which rewinds and โ€” plot twist โ€” returns status_ok, because zero repetitions is still a valid star โœจ and on success it discards the location and jump kleene_37 right back around ๐Ÿ’–

              kleene_37:
              # *
              #     x
              #         (MulOp)
              #         (Factor)

                       loc_push
                       error_push

                       call              sequence_34

                       error_pop_merge
                 fail! jump              failed_38
                       loc_pop_discard
                       jump              kleene_37

              failed_38:
                       loc_pop_rewind
                       status_ok
                       return
              sequence_34:
              # x
              #     (MulOp)
              #     (Factor)

                       ast_push
                       loc_push
                       error_clear

                       error_push

                       call              sym_MulOp

                       error_pop_merge
                 fail! jump              failed_35
                       error_push

                       call              sym_Factor

                       error_pop_merge
                 fail! jump              failed_35

                       ast_pop_discard
                       loc_pop_discard
                       return

              failed_35:
                       ast_pop_rewind
                       loc_pop_rewind
                       return
              #
              # value Symbol 'Factor'
              #

              sym_Factor:
              # x
              #     (Term)
              #     *
              #         x
              #             (AddOp)
              #             (Term)

                       symbol_restore    Factor
                found! jump              found_60
                       loc_push
                       ast_push

                       call              sequence_57

                 fail! value_clear
                   ok! value_reduce      Factor
                       symbol_save       Factor
                       error_nonterminal Factor
                       ast_pop_rewind
                       loc_pop_discard

              found_60:
                   ok! ast_value_push
                       return

and boom, sym_Factor arrives โ€” same memoized symbol_restore glow-up as Expression, just with found_60 as its landing pad ๐ŸŽ€ it calls sequence_57, and on success does the value_reduce Factor โ†’ symbol_save Factor โ†’ error_nonterminal Factor โ†’ unwind combo ๐Ÿ’…

sequence_57 is (Term) * ((AddOp) (Term)), kleene_54 is the star, sequence_51 is one (AddOp) (Term) โ€” exact same shape as the multiplication layer, just with AddOp and Term swapped in ๐Ÿฆ‹

              sequence_57:
              # x
              #     (Term)
              #     *
              #         x
              #             (AddOp)
              #             (Term)

                       ast_push
                       loc_push
                       error_clear

                       error_push

                       call              sym_Term

                       error_pop_merge
                 fail! jump              failed_58
                       error_push

                       call              kleene_54

                       error_pop_merge
                 fail! jump              failed_58

                       ast_pop_discard
                       loc_pop_discard
                       return

              failed_58:
                       ast_pop_rewind
                       loc_pop_rewind
                       return

              kleene_54:
              # *
              #     x
              #         (AddOp)
              #         (Term)

                       loc_push
                       error_push

                       call              sequence_51

                       error_pop_merge
                 fail! jump              failed_55
                       loc_pop_discard
                       jump              kleene_54

              failed_55:
                       loc_pop_rewind
                       status_ok
                       return

              sequence_51:
              # x
              #     (AddOp)
              #     (Term)

                       ast_push
                       loc_push
                       error_clear

                       error_push

                       call              sym_AddOp

                       error_pop_merge
                 fail! jump              failed_52
                       error_push

                       call              sym_Term

                       error_pop_merge
                 fail! jump              failed_52

                       ast_pop_discard
                       loc_pop_discard
                       return

              failed_52:
                       ast_pop_rewind
                       loc_pop_rewind
                       return
              #
              # value Symbol 'MulOp'
              #

              sym_MulOp:
              # /
              #     '*'
              #     '/'

                       symbol_restore    MulOp
                found! jump              found_67
                       loc_push

                       call              choice_65

                 fail! value_clear
                   ok! value_leaf        MulOp
                       symbol_save       MulOp
                       error_nonterminal MulOp
                       loc_pop_discard

              found_67:
                   ok! ast_value_push
                       return

ok so sym_MulOp is a leaf, bestie โ€” notice it's value_leaf MulOp instead of value_reduce, and there's no ast_push / ast_pop_rewind anywhere in it, because a leaf has literally nothing to recurse into ๐Ÿ’€ it's just symbol_restore โ†’ choice_65 โ†’ memoize โ†’ found_67 โ†’ ast_value_push โ†’ bye ๐Ÿ’–

and choice_65 is the tiniest ordered choice imaginable: try test_char "*", else rewind and try test_char "/", else status_fail ๐Ÿ™ˆ its success pad is oknoast_64, which is the no-AST variant โ€” loc_pop_discard and return, no ast_pop_discard needed because this choice never pushed an AST ๐Ÿงš

              choice_65:
              # /
              #     '*'
              #     '/'

                       error_clear

                       loc_push
                       error_push

                       input_next        "t *"
                   ok! test_char         "*"

                       error_pop_merge
                   ok! jump              oknoast_64

                       loc_pop_rewind
                       loc_push
                       error_push

                       input_next        "t /"
                   ok! test_char         "/"

                       error_pop_merge
                   ok! jump              oknoast_64

                       loc_pop_rewind
                       status_fail
                       return

              oknoast_64:
                       loc_pop_discard
                       return
              #
              # value Symbol 'Number'
              #

              sym_Number:
              # x
              #     ?
              #         (Sign)
              #     +
              #         (Digit)

                       symbol_restore    Number
                found! jump              found_80
                       loc_push
                       ast_push

                       call              sequence_77

                 fail! value_clear
                   ok! value_reduce      Number
                       symbol_save       Number
                       error_nonterminal Number
                       ast_pop_rewind
                       loc_pop_discard

              found_80:
                   ok! ast_value_push
                       return

finally for this stretch, sym_Number ๐Ÿ‘‘ โ€” the rule is ? (Sign) followed by + (Digit), so an optional sign and then one-or-more digits. same memoize-or-build skeleton, landing on found_80, and sequence_77 is the assembly: optional_70 for the sign, then poskleene_73 for the positive kleene (that's the +) ๐Ÿ“

              sequence_77:
              # x
              #     ?
              #         (Sign)
              #     +
              #         (Digit)

                       ast_push
                       loc_push
                       error_clear

                       error_push

                       call              optional_70

                       error_pop_merge
                 fail! jump              failed_78
                       error_push

                       call              poskleene_73

                       error_pop_merge
                 fail! jump              failed_78

                       ast_pop_discard
                       loc_pop_discard
                       return

              failed_78:
                       ast_pop_rewind
                       loc_pop_rewind
                       return

and that's exactly where this part runs out of code โ€” optional_70 and poskleene_73 are the next girls up, so stay tuned bestie ๐Ÿ’œโœจ

              optional_70:
              # ?
              #     (Sign)

                       loc_push
                       error_push

                       call              sym_Sign

                       error_pop_merge
                 fail! loc_pop_rewind
                   ok! loc_pop_discard
                       status_ok
                       return

              poskleene_73:
              # +
              #     (Digit)

                       loc_push

                       call              sym_Digit

                 fail! jump              failed_74

              loop_75:
                       loc_pop_discard
                       loc_push
                       error_push

                       call              sym_Digit

                       error_pop_merge
                   ok! jump              loop_75
                       status_ok

              failed_74:
                       loc_pop_rewind
                       return
              #
              # value Symbol 'Sign'
              #

              sym_Sign:
              # /
              #     '-'
              #     '+'

                       symbol_restore    Sign
                found! jump              found_86
                       loc_push

                       call              choice_5

                 fail! value_clear
                   ok! value_leaf        Sign
                       symbol_save       Sign
                       error_nonterminal Sign
                       loc_pop_discard

              found_86:
                   ok! ast_value_push
                       return
              #
              # value Symbol 'Term'
              #

              sym_Term:
              # (Number)

                       symbol_restore    Term
                found! jump              found_89
                       loc_push
                       ast_push

                       call              sym_Number

                 fail! value_clear
                   ok! value_reduce      Term
                       symbol_save       Term
                       error_nonterminal Term
                       ast_pop_rewind
                       loc_pop_discard

              found_89:
                   ok! ast_value_push
                       return

              #
              #

โœจ PEG SERIALIZATION FORMAT โœจ

okay bestie, new section, new vibe ๐Ÿ’– This is where we spill the tea on the exact format that the Parser Tools use to serialize Parsing Expression Grammars โ€” like, turning them into immutable values so they can be shipped around, compared, whatever. โœจ

And there's a whole two-tier situation going on, so pay attention ๐Ÿซถ: we distinguish between regular and canonical serializations. A PEG can have more than one regular serialization, but exactly one of them is canonical. Periodt. ๐Ÿ’…


๐Ÿ’พ regular serialization

  1. The serialization of any PEG is a nested Tcl dictionary. ๐ŸŒธ
  2. This dictionary holds a single key, pt::grammar::peg, and its value. That value holds the contents of the grammar. ๐Ÿ‘‘
  3. The contents of the grammar are a Tcl dictionary holding the set of nonterminal symbols and the starting expression. The relevant keys and their values are:
    • rules โ€” The value is a Tcl dictionary whose keys are the names of the nonterminal symbols known to the grammar. ๐ŸŽ€
      1. Each nonterminal symbol may occur only once.
      2. The empty string is not a legal nonterminal symbol. (not me trying to name a rule "" ๐Ÿ’€)
      3. The value for each symbol is a Tcl dictionary itself. The relevant keys and their values in this dictionary are:
        • is โ€” The value is the serialization of the parsing expression describing the symbols sentennial structure, as specified in the section PE serialization format. โœจ
        • mode โ€” The value can be one of three values specifying how a parser should handle the semantic value produced by the symbol. ๐Ÿฆ‹
          • value โ€” The semantic value of the nonterminal symbol is an abstract syntax tree consisting of a single node node for the nonterminal itself, which has the ASTs of the symbol's right hand side as its children. ๐ŸŒˆ
          • leaf โ€” The semantic value of the nonterminal symbol is an abstract syntax tree consisting of a single node node for the nonterminal, without any children. Any ASTs generated by the symbol's right hand side are discarded. ๐Ÿ™ˆ
          • void โ€” The nonterminal has no semantic value. Any ASTs generated by the symbol's right hand side are discarded (as well). ๐Ÿ’€
    • start โ€” The value is the serialization of the start parsing expression of the grammar, as specified in the section PE serialization format. โญ
  4. The terminal symbols of the grammar are specified implicitly as the set of all terminal symbols used in the start expression and on the RHS of the grammar rules. ๐Ÿงš

๐Ÿ‘‘ canonical serialization

The canonical serialization of a grammar has the format as specified in the previous item, and then additionally satisfies the constraints below โ€” which is what makes it unique among all the possible serializations of this grammar. One grammar, one canonical form. Main character energy only. ๐Ÿ’…

  1. The keys found in all the nested Tcl dictionaries are sorted in ascending dictionary order, as generated by Tcl's builtin command lsort -increasing -dict. ๐ŸŽ€
  2. The string representation of the value is the canonical representation of a Tcl dictionary. I.e. it does not contain superfluous whitespace. No extra crumbs, we keep it clean. โœจ

๐Ÿ“ EXAMPLE

Say we're working with the following PEG for simple mathematical expressions ๐Ÿ‘‡

PEG calculator (Expression)
    Digit      <- '0'/'1'/'2'/'3'/'4'/'5'/'6'/'7'/'8'/'9'       ;
    Sign       <- '-' / '+'                                     ;
    Number     <- Sign? Digit+                                  ;
    Expression <- Term (AddOp Term)*                            ;
    MulOp      <- '*' / '/'                                     ;
    Term       <- Factor (MulOp Factor)*                        ;
    AddOp      <- '+'/'-'                                       ;
    Factor     <- '(' Expression ')' / Number                   ;
END;

Then its canonical serialization (except for whitespace) is this absolute serve ๐Ÿ’–๐Ÿ‘‡

pt::grammar::peg {
    rules {
        AddOp      {is {/ {t -} {t +}}                                                                mode value}
        Digit      {is {/ {t 0} {t 1} {t 2} {t 3} {t 4} {t 5} {t 6} {t 7} {t 8} {t 9}}                mode value}
        Expression {is {x {n Term} {* {x {n AddOp} {n Term}}}}                                        mode value}
        Factor     {is {/ {x {t (} {n Expression} {t )}} {n Number}}                                  mode value}
        MulOp      {is {/ {t *} {t /}}                                                                mode value}
        Number     {is {x {? {n Sign}} {+ {n Digit}}}                                                 mode value}
        Sign       {is {/ {t -} {t +}}                                                                mode value}
        Term       {is {x {n Factor} {* {x {n MulOp} {n Factor}}}}                                    mode value}
    }
    start {n Expression}
}

So here's the tea โ˜•: this is the format the Parser Tools use to serialize Parsing Expressions as immutable values โ€” like, for transport, comparison, and all that jazz. ๐Ÿ’–

We distinguish between regular and canonical serializations. A parsing expression can have more than one regular serialization, but only exactly one of them will ever be canonical. No cap. ๐Ÿ’…

โœจ Regular serialization โœจ

๐Ÿงฌ Atomic Parsing Expressions ๐Ÿงฌ

  1. epsilon โ€” this string is an atomic parsing expression. It matches the empty string. Periodt. ๐Ÿซง
  2. dot โ€” also an atomic parsing expression. It matches any character. Literally any. ๐ŸŒˆ
  3. alnum โ€” atomic parsing expression, matches any Unicode alphabet or digit character. This is a custom extension of PEs based on Tcl's builtin command string is. Iykyk. ๐Ÿ’…
  4. alpha โ€” matches any Unicode alphabet character. Also a custom extension based on Tcl's string is. ๐Ÿ‘‘
  5. ascii โ€” matches any Unicode character below U0080. Custom extension based on string is. ๐Ÿ’ป
  6. control โ€” matches any Unicode control character. Custom extension based on string is. ๐ŸŽฎ
  7. digit โ€” matches any Unicode digit character. And yes, that includes characters outside the [0..9] range, bestie. ๐Ÿคฏ Custom extension based on string is.
  8. graph โ€” matches any Unicode printing character, except for space. Custom extension based on string is. ๐Ÿ“Š
  9. lower โ€” matches any Unicode lower-case alphabet character. Custom extension based on string is. ๐Ÿ”ก
  10. print โ€” matches any Unicode printing character, including space. Custom extension based on string is. ๐Ÿ–จ๏ธ
  11. punct โ€” matches any Unicode punctuation character. Custom extension based on string is. โ€ผ๏ธ
  12. space โ€” matches any Unicode space character. Custom extension based on string is. ๐ŸŒŒ
  13. upper โ€” matches any Unicode upper-case alphabet character. Custom extension based on string is. ๐Ÿ” 
  14. wordchar โ€” matches any Unicode word character. That's any alphanumeric character (see alnum) plus any connector punctuation characters, like underscore. Custom extension based on string is. ๐Ÿ“
  15. xdigit โ€” matches any hexadecimal digit character. Custom extension based on string is. ๐Ÿ”ข
  16. ddigit โ€” matches any decimal digit character. This one is a custom extension based on Tcl's builtin command regexp. ๐Ÿ‘€
  17. The expression [list t x] is an atomic parsing expression. It matches the terminal string x. ๐ŸŽฏ
  18. The expression [list n A] is an atomic parsing expression. It matches the nonterminal A. ๐Ÿท๏ธ

๐Ÿงฉ Combined Parsing Expressions ๐Ÿงฉ

  1. For parsing expressions e1, e2, ..., the result of [list / e1 e2 ... ] is a parsing expression too. This is the ordered choice, aka prioritized choice. Choose your fighter. ๐ŸฅŠ
  2. For parsing expressions e1, e2, ..., the result of [list x e1 e2 ... ] is a parsing expression as well. This is the sequence. One after the other. โžก๏ธ
  3. For a parsing expression e, the result of [list * e] is a parsing expression as well. This is the kleene closure, describing zero or more repetitions. Star power. โญ
  4. For a parsing expression e, the result of [list + e] is a parsing expression as well. This is the positive kleene closure, describing one or more repetitions. Plus sign energy. โž•
  5. For a parsing expression e, the result of [list & e] is a parsing expression as well. This is the and lookahead predicate. Peek ahead. ๐Ÿ”ฎ
  6. For a parsing expression e, the result of [list ! e] is a parsing expression as well. This is the not lookahead predicate. Nope, not today. ๐Ÿšซ
  7. For a parsing expression e, the result of [list ? e] is a parsing expression as well. This is the optional input. Maybe, maybe not. ๐Ÿคท

๐Ÿ‘‘ Canonical serialization ๐Ÿ‘‘

The canonical serialization of a parsing expression uses the format from the previous item, but then it also has to satisfy the constraints below. These make it unique among all possible serializations of that parsing expression. So it's the one true queen. ๐Ÿ‘‘

  1. The string representation of the value is the canonical representation of a pure Tcl list. I.e., it does not contain superfluous whitespace. No extra spaces, bestie. โœ‚๏ธ
  2. Terminals are not encoded as ranges (where the start and end of the range are identical). Keep it simple. ๐ŸŽ€

๐ŸŒŸ EXAMPLE ๐ŸŒŸ

Assuming the parsing expression shown on the right-hand side of the rule

Expression <- Term (AddOp Term)*

then its canonical serialization (except for whitespace) is

{x {n Term} {* {x {n AddOp} {n Term}}}}

๐Ÿž BUGS, IDEAS, FEEDBACK ๐Ÿž

This document, and the package it describes, will undoubtedly contain bugs and other problems. I know, shocking. ๐Ÿ˜ญ Please report such in the category pt of the Tcllib Trackers at http://core.tcl.tk/tcllib/reportlist. Also, if you have ideas for enhancements for either the package and/or the documentation, please report those too. We love a glow-up. โœจ

When proposing code changes, please provide unified diffs, i.e., the output of diff -u. That's the move. ๐Ÿ’…

Also note: attachments are strongly preferred over inlined patches. You can attach files by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar. Easy peasy. ๐Ÿ‹

๐Ÿ”‘ KEYWORDS ๐Ÿ”‘

  • EBNF
  • LL(k)
  • PARAM
  • PEG
  • TDPL
  • context-free languages
  • conversion
  • expression
  • format conversion
  • grammar
  • matching
  • parser
  • parsing expression
  • parsing expression grammar
  • push down automaton
  • recursive descent
  • serialization
  • state
  • top-down parsing languages
  • transducer

๐Ÿ“‚ CATEGORY ๐Ÿ“‚

Parsing and Grammars. That's the category. Slay. ๐Ÿ’…

Copyright (c) 2009 Andreas Kupries <andreas_kupries@users.sourceforge.net>. Respect the OG. ๐Ÿ™Œ


And that's the whole vibe. This page is from tcllib 1.1, for pt::peg::to::param(n). Stay slay, bestie. ๐Ÿ’–โœจ

๐Ÿ“ฆ where this page comes from