Regexp::Grammars(3pm)
💖 Regexp::Grammars(3pm) 💖
okay bestie, welcome to Regexp::Grammars(3pm) — the manual page that's literally about adding grammatical parsing features to Perl 5.10 regexes. let's get into it 💅
✨ NAME ✨
Regexp::Grammars — Add grammatical parsing features to Perl 5.10 regexes
🎀 VERSION 🎀
This document describes Regexp::Grammars version 1.058 💖
🌸 SYNOPSIS 🌸
use Regexp::Grammars;
my $parser = qr{
(?:
<Verb> # Parse and save a Verb in a scalar
<.ws> # Parse but don't save whitespace
<Noun> # Parse and save a Noun in a scalar
<type=(?{ rand > 0.5 ? 'VN' : 'VerbNoun' })>
# Save result of expression in a scalar
|
(?:
<[Noun]> # Parse a Noun and save result in a list
(saved under the key 'Noun')
<[PostNoun=ws]> # Parse whitespace, save it in a list
(saved under the key 'PostNoun')
)+
<Verb> # Parse a Verb and save result in a scalar
(saved under the key 'Verb')
<type=(?{ 'VN' })> # Save a literal in a scalar
|
<debug: match> # Turn on the integrated debugger here
<.Cmd= (?: mv? )> # Parse but don't capture a subpattern
(name it 'Cmd' for debugging purposes)
<[File]>+ # Parse 1+ Files and save them in a list
(saved under the key 'File')
<debug: off> # Turn off the integrated debugger here
<Dest=File> # Parse a File and save it in a scalar
(saved under the key 'Dest')
)
################################################################
<token: File> # Define a subrule named File
<.ws> # - Parse but don't capture whitespace
<MATCH= ([\w-]+) > # - Parse the subpattern and capture
# matched text as the result of the
# subrule
<token: Noun> # Define a subrule named Noun
cat | dog | fish # - Match an alternative (as usual)
<rule: Verb> # Define a whitespace-sensitive subrule
eats # - Match a literal (after any space)
<Object=Noun>? # - Parse optional subrule Noun and
# save result under the key 'Object'
| # Or else...
<AUX> # - Parse subrule AUX and save result
<part= (eaten|seen) > # - Match a literal, save under 'part'
<token: AUX> # Define a whitespace-insensitive subrule
(has | is) # - Match an alternative and capture
(?{ $MATCH = uc $^N }) # - Use captured text as subrule result
}x;
# Match the grammar against some text...
if ($text =~ $parser) {
# If successful, the hash %/ will have the hierarchy of results...
process_data_in( %/ );
}🌈 QUICKSTART CHEATSHEET 🌈
💖 In your program... 💖
use Regexp::Grammars;— Allow enhanced regexes in lexical scope%/— Result-hash for successful grammar match
💖 Defining and using named grammars... 💖
<grammar: GRAMMARNAME>— Define a named grammar that can be inherited<extends: GRAMMARNAME>— Current grammar inherits named grammar's rules
💖 Defining rules in your grammar... 💖
<rule: RULENAME>— Define rule with magic whitespace<token: RULENAME>— Define rule without magic whitespace<objrule: CLASS= NAME>— Define rule that blesses return-hash into class<objtoken: CLASS= NAME>— Define token that blesses return-hash into class<objrule: CLASS>— Shortcut for above (rule name derived from class)<objtoken: CLASS>— Shortcut for above (token name derived from class)
💖 Matching rules in your grammar... 💖
<RULENAME>— Call named subrule (may be fully qualified) save result to$MATCH{RULENAME}<RULENAME(...)>— Call named subrule, passing args to it<!RULENAME>— Call subrule and fail if it matches<!RULENAME(...)>— (shorthand for(?!<.RULENAME>))<:IDENT>— Match contents of$ARG{IDENT}as a pattern<\:IDENT>— Match contents of$ARG{IDENT}as a literal</:IDENT>— Match closing delimiter for$ARG{IDENT}<%HASH>— Match longest possible key of hash<%HASH {PAT}>— Match any key of hash that also matches PAT</IDENT>— Match closing delimiter for$MATCH{IDENT}<\_IDENT>— Match the literal contents of$MATCH{IDENT}<ALIAS= RULENAME>— Call subrule, save result in$MATCH{ALIAS}<ALIAS= %HASH>— Match a hash key, save key in$MATCH{ALIAS}<ALIAS= ( PATTERN )>— Match pattern, save match in$MATCH{ALIAS}<ALIAS= (?{ CODE })>— Execute code, save value in$MATCH{ALIAS}<ALIAS= 'STR' >— Save specified string in$MATCH{ALIAS}<ALIAS= 42 >— Save specified number in$MATCH{ALIAS}<ALIAS= /IDENT>— Match closing delim, save as$MATCH{ALIAS}<ALIAS= \_IDENT>— Match'$MATCH{IDENT}', save as$MATCH{ALIAS}<.SUBRULE>— Call subrule (one of the above forms), but don't save the result in%MATCH<[SUBRULE]>— Call subrule (one of the above forms), but append result instead of overwriting it<SUBRULE1>+ % <SUBRULE2>— Match one or more repetitions of SUBRULE1 as long as they're separated by SUBRULE2<SUBRULE1> ** <SUBRULE2>— Same (only for backwards compatibility)<SUBRULE1>* % <SUBRULE2>— Match zero or more repetitions of SUBRULE1 as long as they're separated by SUBRULE2<SUBRULE1>* %% <SUBRULE2>— Match zero or more repetitions of SUBRULE1 as long as they're separated by SUBRULE2 and allow an optional trailing SUBRULE2
💖 In your grammar's code blocks... 💖
$CAPTURE— Alias for$^N(the most recent paren capture)$CONTEXT— Another alias for$^N$INDEX— Current index of next matching position in string%MATCH— Current rule's result-hash$MATCH— Magic override value (returned instead of result-hash)%ARG— Current rule's argument hash$DEBUG— Current match-time debugging mode
💖 Directives... 💖
<require: (?{ CODE }) >— Fail if code evaluates false<timeout: INT >— Fail after specified number of seconds<debug: COMMAND >— Change match-time debugging mode<logfile: LOGFILE >— Change debugging log file (default:STDERR)<fatal: TEXT|(?{CODE})>— Queue error message and fail parse<error: TEXT|(?{CODE})>— Queue error message and backtrack<warning: TEXT|(?{CODE})>— Queue warning message and continue<log: TEXT|(?{CODE})>— Explicitly add a message to debugging log<ws: PATTERN >— Override automatic whitespace matching<minimize:>— Simplify the result of a subrule match<context:>— Switch on context substring retention<nocontext:>— Switch off context substring retention
💜 DESCRIPTION 💜
This module adds a small number of new regex constructs that can be used within Perl 5.10 patterns to implement complete recursive-descent parsing. ✨
Perl 5.10 already supports recursive-descent matching, via the new (?<name>...) and (?&name) constructs. For example, here is a simple matcher for a subset of the LaTeX markup language:
$matcher = qr{
(?&File)
(?(DEFINE)
(?<File> (?&Element)* )
(?<Element> \s* (?&Command)
| \s* (?&Literal)
)
(?<Command> \\ \s* (?&Literal) \s* (?&Options)? \s* (?&Args)? )
(?<Options> \[ \s* (?:(?&Option) (?:\s*,\s* (?&Option) )*)? \s* \])
(?<Args> \{ \s* (?&Element)* \s* \} )
(?<Option> \s* [^][\$&%#_{}~^\s,]+ )
(?<Literal> \s* [^][\$&%#_{}~^\s]+ )
)
}xmsThis technique makes it possible to use regexes to recognize complex, hierarchical--and even recursive--textual structures. The problem is that Perl 5.10 doesn't provide any support for extracting that hierarchical data into nested data structures. In other words, using Perl 5.10 you can match complex data, but not parse it into an internally useful form. 😭
An additional problem when using Perl 5.10 regexes to match complex data formats is that you have to make sure you remember to insert whitespace-matching constructs (such as \s*) at every possible position where the data might contain ignorable whitespace. This reduces the readability of such patterns, and increases the chance of errors (typically caused by overlooking a location where whitespace might appear). 💀
The Regexp::Grammars module solves both those problems. 👑
If you import the module into a particular lexical scope, it preprocesses any regex in that scope, so as to implement a number of extensions to the standard Perl 5.10 regex syntax. These extensions simplify the task of defining and calling subrules within a grammar, and allow those subrule calls to capture and retain the components of they match in a proper hierarchical manner.
For example, the above LaTeX matcher could be converted to a full LaTeX parser (and considerably tidied up at the same time), like so:
use Regexp::Grammars;
$parser = qr{
<File>
<rule: File> <[Element]>*
<rule: Element> <Command> | <Literal>
<rule: Command> \\ <Literal> <Options>? <Args>?
<rule: Options> \[ <[Option]>+ % (,) \]
<rule: Args> \{ <[Element]>* \}
<rule: Option> [^][\$&%#_{}~^\s,]+
<rule: Literal> [^][\$&%#_{}~^\s]+
}xmsNote that there is no need to explicitly place \s* subpatterns throughout the rules; that is taken care of automatically.
If the Regexp::Grammars version of this regex were successfully matched against some appropriate LaTeX document, each rule would call the subrules specified within it, and then return a hash containing whatever result each of those subrules returned, with each result indexed by the subrule's name. 💖
So like, if the rule named Command got invoked, it would first try to match a backslash, then it'd call the three subrules <Literal>, <Options>, and <Args> — in that exact sequence, no skipping, no reordering. And if all three matched successfully? The Command rule would hand back a hash with three keys: 'Literal', 'Options', and 'Args'. And the value for each of those entries? Literally whatever result-hash the subrules themselves returned when they matched. ✨
So the way this stacks up: every single level of the hierarchical regex can generate hashes recording everything its own subrules matched. Which means when the entire pattern matches, you get a whole tree of nested hashes that represent the structured data the pattern matched. It's giving data structure realness. 🌈
Okay so for example — if that previous regex grammar were matched against a string containing:
\documentclass[a4paper,11pt]{article}
\author{D. Conway}it would automatically extract a data structure equivalent to this (but with several extra "empty" keys, which are described down in "Subrule results"):
{
'file' => {
'element' => [
{
'command' => {
'literal' => 'documentclass',
'options' => {
'option' => [ 'a4paper', '11pt' ],
},
'args' => {
'element' => [ 'article' ],
}
}
},
{
'command' => {
'literal' => 'author',
'args' => {
'element' => [
{
'literal' => 'D.',
},
{
'literal' => 'Conway',
}
]
}
}
}
]
}
}And the data structure that Regexp::Grammars produces from a regex match is available to the surrounding program in the magic variable %/. Yes bestie, the magic variable. 👑
Regexp::Grammars provides many features that simplify the extraction of hierarchical data via a regex match, and also some features that can simplify the processing of that data once it has been extracted. The following sections explain each of those features, and some of the parsing techniques they support. Let's goooo. 💅
🎀 Setting up the module 🎀
Just add:
use Regexp::Grammars;to any lexical scope. Any regexes within that scope will automatically now implement the new parsing constructs:
use Regexp::Grammars;
my $parser = qr/ regex with $extra <chocolatey> grammar bits /;Note that you do not have to use the /x modifier when declaring a regex grammar (though you certainly may). But even if you don't, the module quietly adds a /x to every regex within the scope of its usage. Otherwise, the default "a whitespace character matches exactly that whitespace character" behaviour of Perl regexes would mess up your grammar's parsing. 😭 If you need the non-/x behaviour, you can still use the (?-x) or (?-x:...) directives to switch off /x within one or more of your grammar's components.
Once the grammar has been processed, you can then match text against the extended regexes, in the usual manner (i.e. via a =~ match):
if ($input_text =~ $parser) {
...
}After a successful match, the variable %/ will contain a series of nested hashes representing the structured hierarchical data captured during the parse. Obsessed. 💖
🦋 Structure of a Regexp::Grammars grammar 🦋
A Regexp::Grammars specification consists of a start-pattern (which may include both standard Perl 5.10 regex syntax, as well as special Regexp::Grammars directives), followed by one or more rule or token definitions.
For example:
use Regexp::Grammars;
my $balanced_brackets = qr{
# Start-pattern...
<paren_pair> | <brace_pair>
# Rule definition...
<rule: paren_pair>
\( (?: <escape> | <paren_pair> | <brace_pair> | [^()] )* \)
# Rule definition...
<rule: brace_pair>
\{ (?: <escape> | <paren_pair> | <brace_pair> | [^{}] )* \}
# Token definition...
<token: escape>
\\ .
}xms;The start-pattern at the beginning of the grammar acts like the "top" token of the grammar, and must be matched completely for the grammar to match. No half-matches, no "close enough". Periodt.
This pattern is treated like a token for whitespace matching behaviour (see "Tokens vs rules (whitespace handling)"). That is, whitespace in the start-pattern is treated like whitespace in any normal Perl regex.
The rules and tokens are declarations only and they are not directly matched. Instead, they act like subroutines, and are invoked by name from the initial pattern (or from within a rule or token). It's giving functions but make it regex. 🌈
Each rule or token extends from the directive that introduces it up to either the next rule or token directive, or (in the case of the final rule or token) to the end of the grammar.
✨ Tokens vs rules (whitespace handling) ✨
The difference between a token and a rule is that a token treats any whitespace within it exactly as a normal Perl regular expression would. That is, a sequence of whitespace in a token is ignored if the /x modifier is in effect, or else matches the same literal sequence of whitespace characters (if /x is not in effect).
In a rule, most sequences of whitespace are treated as matching the implicit subrule <.ws>, which is automatically predefined to match optional whitespace (i.e. \s*). Sneaky little auto-subrule. 🙈
Exceptions to this behaviour are whitespaces before a | or a code block or an explicit space-matcher (such as <ws> or \s), or at the very end of the rule)
In other words, a rule such as:
<rule: sentence> <noun> <verb>
| <verb> <noun>is equivalent to a token with added non-capturing whitespace matching:
<token: sentence> <.ws> <noun> <.ws> <verb>
| <.ws> <verb> <.ws> <noun>You can explicitly define a <ws> token to change that default behaviour. For example, you could alter the definition of "whitespace" to include Perlish comments, by adding an explicit <token: ws>:
<token: ws>
(?: \s+ | #[^\n]* )*But be careful not to define <ws> as a rule, as this will lead to all kinds of infinitely recursive unpleasantness. 💀 Like, do not. I'm serious bestie.
🌸 Per-rule whitespace handling 🌸
Redefining the <ws> token changes its behaviour throughout the entire grammar, within every rule definition. Usually that's appropriate, but sometimes you need finer-grained control over whitespace handling.
So Regexp::Grammars provides the <ws:> directive, which allows you to override the implicit whitespace-matches-whitespace behaviour only within the current rule.
Note that this directive does not redefine <ws> within the rule; it simply specifies what to replace each whitespace sequence with (instead of replacing each with a <ws> call).
For example, if a language allows one kind of comment between statements and another within statements, you could parse it with:
<rule: program>
# One type of comment between...
<ws: (\s++ | \# .*? \n)* >
# ...colon-separated statements...
<[statement]>+ % ( ; )
<rule: statement>
# Another type of comment...
<ws: (\s*+ | \#{ .*? }\# )* >
# ...between comma-separated commands...
<cmd> <[arg]>+ % ( , )Note that each directive only applies to the rule in which it is specified. In every other rule in the grammar, whitespace would still match the usual <ws> subrule. Scoped, babes. 🎀
💖 Calling subrules 💖
To invoke a rule to match at any point, just enclose the rule's name in angle brackets (like in Perl 6). There must be no space between the opening bracket and the rulename. For example::
qr{
file: # Match literal sequence 'f' 'i' 'l' 'e' ':'
<name> # Call <rule: name>
<options>? # Call <rule: options> (it's okay if it fails)
<rule: name>
# etc.
}x;If you need to match a literal pattern that would otherwise look like a subrule call, just backslash-escape the leading angle:
qr{
file: # Match literal sequence 'f' 'i' 'l' 'e' ':'
\<name> # Match literal sequence '<' 'n' 'a' 'm' 'e' '>'
<options>? # Call <rule: options> (it's okay if it fails)
<rule: name>
# etc.
}x;That \ is doing so much work and honestly? Iconic. 🫶
🍓 Subrule results 🍓
If a subrule call successfully matches, the result of that match is a reference to a hash. That hash reference is stored in the current rule's own result-hash, under the name of the subrule that was invoked. The hash will, in turn, contain the results of any more deeply nested subrule calls, each stored under the name by which the nested subrule was invoked. Hashes in hashes in hashes, it's giving inception. 🌀
In other words, if the rule sentence is defined:
<rule: sentence>
<noun> <verb> <object>then successfully calling the rule:
<sentence>causes a new hash entry at the current nesting level. That entry's key will be 'sentence' and its value will be a reference to a hash, which in turn will have keys: 'noun', 'verb', and 'object'.
In addition each result-hash has one extra key: the empty string. The value for this key is whatever substring the entire subrule call matched. This value is known as the context substring.
So, for example, a successful call to <sentence> might add something like the following to the current result-hash:
sentence => {
"" => 'I saw a dog',
noun => 'I',
verb => 'saw',
object => {
"" => 'a dog',
article => 'a',
noun => 'dog',
},
}Note, however, that if the result-hash at any level contains only the empty-string key (i.e. the subrule did not call any sub-subrules or save any of their nested result-hashes), then the hash is "unpacked" and just the context substring itself is returned.
For example, if <rule: sentence> had been defined:
<rule: sentence>
I see dead peoplethen a successful call to the rule would only add:
sentence => 'I see dead people'to the current result-hash. No nested hash, no drama — just the string. Clean, minimal, slay. 💅
okay so this whole thing is genuinely a slay feature, because it stops a long chain of nested subrule calls from building you the most unhinged data structures imaginable. Like, without this automatic unpacking, even our cute lil' earlier example:
<rule: sentence>
<noun> <verb> <object>would hand you back something needlessly extra, like:
sentence => {
"" => 'I saw a dog',
noun => {
"" => 'I',
},
verb => {
"" => 'saw',
},
object => {
"" => 'a dog',
article => {
"" => 'a',
},
noun => {
"" => 'dog',
},
},
}✨ Turning off the context substring ✨
The context substring is so handy for debugging and for cooking up error messages, no cap. But bestie, in a big grammar, or when you're parsing a long string, capturing and storing all those nested substrings can get prohibitively expensive real quick. It's giving "my RAM said please stop".
So Regexp::Grammars hands you a directive to stop context substrings from being kept around at all. Any rule or token that has the directive <nocontext:> anywhere in its body will not retain any context substring it matches... unless that substring would be the only entry in its result hash (which only ever happens inside objrules and objtokens).
If a <nocontext:> directive shows up before the first rule or token definition (aka as part of the main pattern), then the entire grammar yeets all context substrings from every single one of its rules and tokens.
But plot twist: you can override that universal prohibition with a second directive, <context:>. If that directive appears in any rule or token, that rule or token will save its context substring, even if a global <nocontext:> is in effect. Main character behaviour honestly.
Which means this grammar:
qr{
<Command>
<rule: Command>
<nocontext:>
<Keyword> <arg=(\S+)>+ % <.ws>
<token: Keyword>
<Move> | <Copy> | <Delete>
# etc.
}xand this grammar:
qr{
<nocontext:>
<Command>
<rule: Command>
<Keyword> <arg=(\S+)>+ % <.ws>
<token: Keyword>
<context:>
<Move> | <Copy> | <Delete>
# etc.
}xwill behave identically (saving context substrings for keywords, but not for commands), except that the first version will also retain the global context substring (i.e. $/{""}), whereas the second version will not. Tiny difference, huge vibe.
Heads up bestie:<context:>and<nocontext:>have no effect on, or even any interaction with, the various result distillation mechanisms. Those keep working exactly the usual way whether you use either directive, both, or neither. 🫶
💅 Renaming subrule results
It's not always convenient to have subrule results stored under the same name as the rule itself, you know? Rule names should be optimized for understanding how the parser behaves, whereas result names should be optimized for understanding the structure of the data. Often those two goals are identical, but not always. Sometimes rule names need to describe what the data looks like, while result names need to describe what the data means. It's giving two different jobs.
For example, sometimes you need to call the same rule twice, to match two syntactically identical components whose positions give them semantically distinct meanings:
<rule: copy_cmd>
copy <file> <file>The problem here? If the second call to <file> succeeds, its result-hash gets stored under the key 'file', clobbering the data that came back from the first call to <file>. Not it. 💀
To dodge that whole mess, Regexp::Grammars lets you alias any subrule call, so it's still invoked by the original name, but its result-hash gets stored under a different key. The syntax for that is: <alias=rulename>. Like so:
<rule: copy_cmd>
copy <from=file> <to=file>Here, <rule: file> is called twice, with the first result-hash stored under the key 'from', and the second result-hash stored under the key 'to'. perfect, no notes ✨
But watch out: the alias before the=must be a proper identifier (i.e. a letter or underscore, followed by letters, digits, and/or underscores). Aliases that start with an underscore, and aliases namedMATCH, have special meaning (see "Private subrule calls" and "Result distillation" respectively). So don't go freestyling.
Aliases can also be super useful for normalizing data that might show up in different formats and sequences. For example:
<rule: copy_cmd>
copy <from=file> <to=file>
| dup <to=file> as <from=file>
| <from=file> -> <to=file>
| <to=file> <- <from=file>Here, no matter which order the old and new files are specified in, the result-hash always gets:
copy_cmd => {
from => 'oldfile',
to => 'newfile',
}🦋 List-like subrule calls
So if a subrule call is quantified with a repetition specifier:
<rule: file_sequence>
<file>+then each repeated match overwrites the corresponding entry in the surrounding rule's result-hash, so only the result of the final repetition survives. Which means, if the above example matched the string foo.pl bar.py baz.php, the result-hash would contain:
file_sequence {
"" => 'foo.pl bar.py baz.php',
file => 'baz.php',
}Yeah... usually that is not the outcome you were going for. So Regexp::Grammars gives you another mechanism for calling a subrule — one that saves all the repetitions of its results. Obsessed.
A regular subrule call is just the rule's name surrounded by angle brackets. If, instead, you wrap the rule's name in <[...]> (angle and square brackets) like so:
<rule: file_sequence>
<[file]>+then the rule gets invoked in exactly the same way, but the result of that submatch is pushed onto an array nested inside the appropriate result-hash entry. In other words, if the above example matched that same foo.pl bar.py baz.php string, the result-hash would contain:
file_sequence {
"" => 'foo.pl bar.py baz.php',
file => [ 'foo.pl', 'bar.py', 'baz.php' ],
}This "listifying subrule call" is also lowkey useful for non-repeated subrule calls, if the same subrule gets invoked in several places in a grammar. For example, if a cmdline option could be given either one or two values, you might parse it like:
<rule: size_option>
-size <[size]> (?: x <[size]> )?The result-hash entry for 'size' would then always contain an array, with either one or two elements, depending on the input being parsed. Clean. 🎀
Listifying subrules can also be given aliases, just like ordinary subrules. The alias is always specified inside the square brackets:
<rule: size_option>
-size <[size=pos_integer]> (?: x <[size=pos_integer]> )?Here, the sizes are parsed using the pos_integer rule, but saved in the result-hash in an array under the key 'size'. Chef's kiss. 👑
🌸 Parametric subrules
When a subrule is invoked, it can be passed a set of named arguments (specified as key=>values pairs). This argument list goes inside a normal Perl regex code block and must appear immediately after the subrule name, before the closing angle bracket.
Inside the subrule that got invoked, you can access those arguments through the special hash %ARG. For example:
<rule: block>
<tag>
<[block]>*
<end_tag(?{ tag=>$MATCH{tag} })> # ...call subrule with argument
<token: end_tag>
end_ (??{ quotemeta $ARG{tag} })Here the block rule first matches a <tag>, and the corresponding substring gets saved in $MATCH{tag}. It then matches any number of nested blocks. Finally it invokes the <end_tag> subrule, passing it an argument whose name is 'tag' and whose value is the current value of $MATCH{tag} (i.e. the original opening tag).
When it's invoked like that, the end_tag token first matches 'end_', then interpolates the literal value of the 'tag' argument and attempts to match it.
Any number of named arguments can be passed when a subrule is invoked. For example, we could generalize the end_tag rule to allow any prefix (not just 'end_'), and also to allow for 'if...fi'-style reversed tags, like so:
<rule: block>
<tag>
<[block]>*
<end_tag (?{ prefix=>'end', tag=>$MATCH{tag} })>
<token: end_tag>
(??{ $ARG{prefix} // q{(?!)} }) # ...prefix as pattern
(??{ quotemeta $ARG{tag} }) # ...tag as literal
|
(??{ quotemeta reverse $ARG{tag} }) # ...reversed tagNote that if you don't need to interpolate values (like$MATCH{tag}) into a subrule's argument list, you can use simple parentheses instead of(?{...}), like so:
<end_tag( prefix=>'end', tag=>'head' )>The only types of values you can use in this simplified syntax are numbers and single-quote-delimited strings. For anything more complex, put the argument list in a full (?{...}). No exceptions bestie.
As the earlier examples show, the single most common type of argument is one of the form: IDENTIFIER => $MATCH{"IDENTIFIER"}. That is, it's a super common requirement to pass an element of %MATCH into a subrule, named with its own key.
Because that's such a common usage, Regexp::Grammars gives you a shortcut. If you use simple parentheses (instead of (?{...}) parentheses), then instead of a pair, you can specify an argument using a colon followed by an identifier. That argument gets replaced by a named argument whose name is the identifier and whose value is the corresponding item from %MATCH. So, for example, instead of:
<end_tag(?{ prefix=>'end', tag=>$MATCH{tag} })>you can just write:
<end_tag( prefix=>'end', :tag )>Careful though! From Perl 5.20 onwards, due to changes in the way that Perl parses regexes, Regexp::Grammars does not support explicitly passing elements of %MATCH as argument values within a list subrule (yeah, it's a very specific and obscure edge-case):
<[end_tag(?{ prefix=>'end', tag=>$MATCH{tag} })]> # Does not workNote, however, that the shortcut:
<[end_tag( prefix=>'end', :tag )]>still works correctly. 🫶
Accessing subrule arguments more cleanly
As the preceding examples illustrate, using subrule arguments effectively generally requires the use of run-time interpolated subpatterns via the (??{...}) construct.
Which produces ugly rule bodies such as:
<token: end_tag>
(??{ $ARG{prefix} // q{(?!)} }) # ...prefix as pattern
(??{ quotemeta $ARG{tag} }) # ...tag as literal
|
(??{ quotemeta reverse $ARG{tag} }) # ...reversed tagokay so that's a LOT of typing for something you're gonna do constantly, and Regexp::Grammars is not about making you suffer, bestie. So it hands you three convenience constructs to make all this way cuter. 🎀
A subrule call that looks like <:identifier> is exactly the same as:
(??{ $ARG{'identifier'} // q{(?!)} })Which is to say: "match whatever's inside $ARG{'identifier'}, and treat that content as a pattern." ✨
A subrule call that looks like <\:identifier> — that's a matchref with a colon right after the backslash, iykyk — is exactly the same as:
(??{ defined $ARG{'identifier'}
? quotemeta($ARG{'identifier'})
: '(?!)'
})Which is to say: "match whatever's inside $ARG{'identifier'}, and treat that content as a literal." 💅
And a subrule call that looks like </:identifier> — that's an invertref with a colon after the forward slash — is exactly the same as:
(??{ defined $ARG{'identifier'}
? quotemeta(reverse $ARG{'identifier'})
: '(?!)'
})Which is to say: "match the closing delimiter that corresponds to whatever's inside $ARG{'identifier'}, as if it were a literal." 🦋
The way that these three exist means we can rewrite that whole <end_tag> token up there so much cleaner, like:
<token: end_tag>
<:prefix> # ...prefix as pattern
<\:tag> # ...tag as a literal
|
</:tag> # ...reversed tagBroadly speaking: inside a subrule, if you wanna match an argument that got passed to that subrule, you use <:ARGNAME> (match it as a pattern) or <\:ARGNAME> (match it as a literal). Periodt. 💖
And can we talk about the mnemonic?? All these subrule-ish interpolations of named arguments follow one consistent rule: the name is always prefixed by a colon. 👑
In other words, <:ARGNAME> behaves just like a <RULENAME>, except the leading colon tells Regexp::Grammars to use the contents of $ARG{'ARGNAME'} as the subpattern, instead of the contents of (?&RULENAME).
Likewise, <:ARGNAME>... no wait, < \:ARGNAME> and </:ARGNAME> work exactly like <!_MATCHNAME> and </INVERTNAME> respectively, except the leading colon says "hey, grab this matchref or invertref from %ARG instead of from %MATCH." 🌸
🎀 Pseudo-subrules 🎀
Aliases can also be given to standard Perl subpatterns, and to code blocks inside a regex. Syntax for subpatterns goes like:
<ALIAS= (SUBPATTERN) >Like, it's exactly an aliased subrule call, except the rule name got swapped out for a set of parens holding the subpattern. Any parens will do — capturing or non-capturing, no cap. 💅
The effect of aliasing a standard subpattern: whatever that subpattern matches gets saved into the result-hash, keyed by the alias. Example:
<rule: file_command>
<cmd=(mv|cp|ln)> <from=file> <to=file>Here <cmd=(mv|cp|ln)> is treated exactly like a plain (mv|cp|ln), but whatever substring it matches gets saved into the result-hash under the key 'cmd'. ✨
The syntax for aliasing code blocks is:
<ALIAS= (?{ your($code->here) }) >Heads up though: the code block must be written in the standard Perl 5.10 regex notation,(?{...}). A super common mistake is writing this instead:<ALIAS= { your($code->here } >and that one will try to interpolate$codebefore the regex is even compiled, because those variables only get "protected" from interpolation when they're inside a(?{...}). 💀
When you write it correctly, this construct runs the code in the block and stashes the result of that execution into the result-hash, keyed by the alias. Aliased code blocks are lowkey perfect for adding semantic info about which branch of a rule fired. Take those copy_cmd alternatives from earlier:
<rule: copy_cmd>
copy <from=file> <to=file>
| dup <to=file> as <from=file>
| <from=file> -> <to=file>
| <to=file> <- <from=file>Using aliased code blocks, you can add an extra field to the result-hash describing which form of the command got detected, like this:
<rule: copy_cmd>
copy <from=file> <to=file> <type=(?{ 'std' })>
| dup <to=file> as <from=file> <type=(?{ 'rev' })>
| <from=file> -> <to=file> <type=(?{ +1 })>
| <to=file> <- <from=file> <type=(?{ -1 })>So now, if the rule matched, the result-hash would contain something like:
copy_cmd => {
from => 'oldfile',
to => 'newfile',
type => 'fwd',
}Also!! on top of all that, aliased subpatterns and code blocks also become visible to Regexp::Grammars' integrated debugger (see Debugging). 🔥
🌈 Aliased literals 🌈
As that last example showed, assigning constants through aliased code blocks is inconveniently verbose — like, so much typing for one little value. So Regexp::Grammars provides a short-cut: you can directly alias a numeric literal or a single-quote delimited literal string, without wrapping either one in a code block. So that previous example could also be written:
<rule: copy_cmd>
copy <from=file> <to=file> <type='std'>
| dup <to=file> as <from=file> <type='rev'>
| <from=file> -> <to=file> <type= +1 >
| <to=file> <- <from=file> <type= -1 >Note: only these two forms of literal are supported in this abbreviated syntax. No sneaking in anything else, bestie. 🙈
💜 Amnesiac subrule calls 💜
By default, every subrule call saves its result into the result-hash — either under its own name, or under an alias.
But sometimes you wanna refactor some literal part of a rule into one or more subrules without those submatches ending up in the result-hash. The syntax for calling a subrule but ignoring its return value is:
<.SUBRULE>(which is stolen directly from Perl 6 🫶)
For example, maybe you'd rather rewrite a rule like:
<rule: paren_pair>
\(
(?: <escape> | <paren_pair> | <brace_pair> | [^()] )*
\)with zero literal matching, like so:
<rule: paren_pair>
<.left_paren>
(?: <escape> | <paren_pair> | <brace_pair> | <.non_paren> )*
<.right_paren>
<token: left_paren> \(
<token: right_paren> \)
<token: non_paren> [^()]And honestly? Since the individual components inside those parens probably aren't being captured for any useful purpose either, you could optimize it even further to:
<rule: paren_pair>
<.left_paren>
(?: <.escape> | <.paren_pair> | <.brace_pair> | <.non_paren> )*
<.right_paren>Also note: you can put the dot modifier on an aliased subpattern too:
<.Alias= (SUBPATTERN) >And yes, that looks like it contradicts itself — you give a subpattern a name and then deliberately ignore that name?? But it actually makes total sense in one situation: giving the alias makes the subpattern visible to the debugger, while the dot stops it from touching the result-hash. See "Debugging non-grammars" for an example of this usage. 👑
🦋 Private subrule calls 🦋
If a rule name (or an alias) starts with an underscore:
<_RULENAME> <_ALIAS=RULENAME>
<[_RULENAME]> <[_ALIAS=RULENAME]>then matching proceeds like normal, and whatever result comes back gets stored in the current result-hash the usual way.
However — when any rule finishes (and just before it returns), it first filters its result-hash, removing any entries whose keys begin with an underscore. So any subrule with an underscored name (or an underscored alias) remembers its result, but only until the end of the current rule. Its results are effectively private to the current rule. 🤫
This is especially useful in conjunction with result distillation.
⭐ Lookahead (zero-width) subrules ⭐
Non-capturing subrule calls can be used inside normal lookaheads:
<rule: qualified_typename>
# A valid typename and has a :: in it...
(?= <.typename> ) [^\s:]+ :: \S+
<rule: identifier>
# An alpha followed by alnums (but not a valid typename)...
(?! <.typename> ) [^\W\d]\w*...but ngl, that syntax is a little unwieldy. And more importantly: there's an internal problem with backtracking that causes positive lookaheads to mess up the module's named capturing mechanism. 😭
So Regexp::Grammars gives you two shorthands:
<!typename> same as: (?! <.typename> )
<?typename> same as: (?= <.typename> ) ...but works correctly!These two can also be called with arguments, if you need that:
<rule: Command>
<Keyword>
(?:
<!Terminator(:Keyword)> <Args=(\S+)>
)?
<Terminator(:Keyword)>Note: as those equivalences imply, neither of these forms of a subroutine call ever captures what it matches. Zero-width, zero-capture, zero drama. 💅
🍓 Matching separated lists 🍓
One of the commonest tasks in text parsing is matching a list of unspecified length where items are separated by a fixed token. Stuff like:
1, 2, 3 , 4 ,13, 91 # Numbers separated by commas and spaces
g-c-a-g-t-t-a-c-a # DNA bases separated by dashes
/usr/local/bin # Names separated by directory markers
/usr:/usr/local:bin # Directories separated by colonsThe usual construct you need to parse those kinds of structures is either:
<rule: list>
<item> <separator> <list> # recursive definition
| <item> # base caseor, if you want zero-or-more items instead of requiring one-or-more:
<rule: list_opt>
<list>? # entire list may be missing
<rule: list> # as before...
<item> <separator> <list> # recursive definition
| <item> # base caseOr, more efficiently, but less prettily:
<rule: list>
<[item]> (?: <separator> <[item]> )* # one-or-more
<rule: list_opt>
(?: <[item]> (?: <separator> <[item]> )* )? # zero-or-moreBecause separated lists are like, such a common component of grammars, Regexp::Grammars provides cleaner ways to specify them:
<rule: list>
<[item]>+ % <separator> # one-or-more
<rule: list_zom>
<[item]>* % <separator> # zero-or-moreAnd that's how you do separated lists without crying. 🧚
okay so these are literally just regular repetition qualifiers — the + and the * — being applied to a subrule (<[item]>), except with a % modifier tacked on after them to say what separator has to show up between the repeated matches. that's the whole vibe, bestie 💅
The number of repetitions that get matched is controlled by two things: the nature of the qualifier (+ vs *) and the subrule you put after the %. The qualified subrule keeps getting matched for as long as its qualifier allows it — <em>provided</em> that the second subrule also matches between those repetitions. ✨
So like, you can match a parenthesized sequence of one-or-more numbers separated by commas, like this:
(1, 2, 3, 4, 13, 91) # Numbers separated by commas (and spaces)with:
<rule: number_list>
\( <[number]>+ % <comma> \)
<token: number> \d+
<token: comma> ,note that any spaces round the commas get ignored here, because <number_list> is specified as a rule and the +% specifier has spaces within and around it. 🎀 To disallow spaces around the commas, make sure there are no spaces in or around the +%:
<rule: number_list_no_spaces>
\( <[number]>+%<comma> \)(or else specify the rule as a token instead — your call, queen 🦋).
Because the % is a modifier applied to a qualifier, you can modify any other repetition qualifier in exactly the same way. For example:
<[item]>{2,4} % <sep> # two-to-four items, separated
<[item]>{7} % <sep> # exactly 7 items, separated
<[item]>{10,}? % <sep> # minimum of 10 or more items, separatedYou can even do this:
<[item]>? % <sep> # one-or-zero items, (theoretically) separatedthough the separator specification is, of course, meaningless in that case, since it will never be needed to separate a maximum of one item. 😭 the math is not mathing, but ok
the % is metasyntax, bestie 💀
Within a Regexp::Grammars regex a simple % is always metasyntax, so it cannot be used to match a literal %. Any attempt to do so is immediately fatal when the regex is compiled:
<token: percentage>
\d{1,3} % # Fatal. Will not match "7%", "100%", etc.
<token: perl_hash>
% <ident> # Fatal. Will not match "%foo", "%bar", etc.
<token: perl_mod>
<expr> % <expr> # Fatal. Will not match "$n % 2", etc.If you need to match a literal % immediately after a repetition, quote it with a backslash:
<token: percentage>
\d{1,3} \% # Okay. Will match "7%", "100%", etc.
<token: perl_hash>
\% <ident> # Okay. Will match "%foo", "%bar", etc.
<token: perl_mod>
<expr> \% <expr> # Okay. Will match "$n % 2", etc.Note that it's usually necessary to use the <[...]> form for the repeated items being matched, so that all of them get saved in the result hash. 🫶 You can also save all the separators (if they're important) by specifying them as a list-like subrule too:
\( <[number]>* % <[comma]> \) # save numbers *and* separatorsThe repeated item must be specified as a subrule call of some kind (i.e. in angles), but the separators may be specified <em>either</em> as a subrule <em>or</em> as a raw bracketed pattern (i.e. brackets without any nested subrule calls). For example:
<[number]>* % ( , | : ) # Numbers separated by commas or colons
<[number]>* % [,:] # Same, but more efficiently matchedThe separator should always be specified within matched delimiters of some kind: either matching <...> or matching (...) or matching [...]. Simple, non-bracketed separators will <em>sometimes</em> also work:
<[number]>+ % ,but not always:
<[number]>+ % ,\s+ # Oops! Separator is just: ,This is because of the limited way in which the module internally parses ordinary regex components (i.e. without full understanding of their implicit precedence). as a consequence, consistently placing brackets around any separator is a much safer approach:
<[number]>+ % (,\s+)You can also use a simple pattern on the left of the % as the item matcher, but in this case it must always be aliased into a list-collecting subrule, like so:
<[item=(\d+)]>* % [,]and for the lore 📜 — note that, for backwards compatibility with earlier versions of Regexp::Grammars, the +% operator can also be written: **. However, there can be no space between the two asterisks of this variant. That is:
<[item]> ** <sep> # same as <[item]>* % <sep>
<[item]>* * <sep> # error (two * qualifiers in a row)🌸 Matching separated lists with a trailing separator 🌸
Some languages allow a separated list to include an extra trailing separator. For example:
~/bin/perl5/ # Trailing /-separator in filepath
(1,2,3,) # Trailing ,-separator in Perl listTo match such constructs using the % operator, you'd need to add something to explicitly match the optional trailing separator:
<dir>+ % [/] [/]? # Slash-separated dirs, then optional final slash
<elem>+ % [,] [,]? # Comma-separated elems, then optional final commawhich is tedious. 🙈
So the module also supports a second kind of "separated list" operator, that allows an optional trailing separator as well: the %% operator. This operator behaves exactly like the % operator, except that it also matches a final trailing separator, if one is present.
So the previous examples could be (better) written as:
<dir>+ %% [/] # Slash-separated dirs, with optional final slash
<elem>+ %% [,] # Comma-separated elems, with optional final comma👑 Matching hash keys 👑
In some situations a grammar may need a rule that matches dozens, hundreds, or even thousands of one-word alternatives. For example, when matching command names, or valid userids, or English words. In such cases it is often impractical (and always inefficient) to list all the alternatives between | alternators:
<rule: shell_cmd>
a2p | ac | apply | ar | automake | awk | ...
# ...and 400 lines later
... | zdiff | zgrep | zip | zmore | zsh
<rule: valid_word>
a | aa | aal | aalii | aam | aardvark | aardwolf | aba | ...
# ...and 40,000 lines later...
... | zymotize | zymotoxic | zymurgy | zythem | zythumTo simplify such cases, Regexp::Grammars provides a special construct that allows you to specify all the alternatives as the keys of a normal hash. The syntax for that construct is simply to put the hash name inside angle brackets (with no space between the angles and the hash name). ✨
Which means that the rules in the previous example could also be written:
<rule: shell_cmd>
<%cmds>
<rule: valid_word>
<%dict>provided that the two hashes (%cmds and %dict) are visible in the scope where the grammar is created. 👀
Matching a hash key in this way is typically significantly faster than matching a large set of alternations. Specifically, it is O(length of longest potential key) ^ 2, instead of O(number of keys). the efficiency ate, no cap 💅
Internally, the construct is converted to something equivalent to:
<rule: shell_cmd>
(<.hk>) <require: (?{ exists $cmds{$CAPTURE} })>
<rule: valid_word>
(<.hk>) <require: (?{ exists $dict{$CAPTURE} })>The special <hk> rule is created automatically, and defaults to \S+, but you can also define it explicitly to handle other kinds of keys. For example:
<rule: hk>
[^\n]+ # Key may be any number of chars on a single line
<rule: hk>
[ACGT]{10,} # Key is a base sequence of at least 10 pairsAlternatively, you can specify a different key-matching pattern for each hash you're matching, by placing the required pattern in braces immediately after the hash name. For example:
<rule: client_name>
# Valid keys match <.hk> (default or explicitly specified)
<%clients>
<rule: shell_cmd>
# Valid keys contain only word chars, hyphen, slash, or dot...
<%cmds { [\w-/.]+ }>
<rule: valid_word>
# Valid keys contain only alphas or internal hyphen or apostrophe...
<%dict{ (?i: (?:[a-z]+[-'])* [a-z]+ ) }>
<rule: DNA_sequence>
# Valid keys are base sequences of at least 10 pairs...
<%sequences{[ACGT]{10,}}>This second approach to key-matching is preferred, because it localizes any non-standard key-matching behaviour to each individual hash. 🎀
heads up, bestie ⚠️ Note that changes in the compilation process from Perl 5.18 onwards mean that in some cases the <%hash> construct only works reliably if the hash itself is declared at the outermost lexical scope (i.e. file scope).
Specifically, if the regex grammar does not include any interpolated scalars or arrays and the hash was declared within a subroutine (even within the same subroutine as the regex grammar that uses it), the regex will not be able to "see" the hash variable at compile-time. This will produce a Global symbol "%hash" requires explicit package name compile-time error. For example:
sub build_keyword_parser {
# Hash declared inside subroutine...
my %keywords = (foo => 1, bar => 1);
# ...then used in <%hash> construct within uninterpolated regex...
return qr{
^<keyword>$
<rule: keyword> <%keywords>
}x;
# ...produces compile-time error
}The solution is to place the hash outside the subroutine containing the grammar:
# Hash declared OUTSIDE subroutine...
my %keywords = (foo => 1, bar => 1);
sub build_keyword_parser {
return qr{
^<keyword>$
<rule: keyword> <%keywords>
}x;
}...or else to explicitly interpolate at least one scalar (even just a scalar containing an empty string):
sub build_keyword_parser {
my %keywords = (foo => 1, bar => 1);
my $DEFER_REGEX_COMPILATION = "";
return qr{
^<keyword>$
<rule: keyword> <%keywords>
$DEFER_REGEX_COMPILATION
}x;
}🦋 Rematching subrule results 🦋
Sometimes it is useful to be able to rematch a string that has previously been matched by some earlier subrule. For example, consider a rule to match shell-like control blocks:
<rule: control_block>
for <expr> <[command]>+ endfor
| while <expr> <[command]>+ endwhile
| if <expr> <[command]>+ endif
| with <expr> <[command]>+ endwithThis would be much tidier if we could factor out the command names (which are the only differences between the four alternatives). The problem is that the obvious solution:
so like… you want to match <keyword> … end<keyword> where both keywords are the same. But this:
<rule: control_block>
<keyword> <expr>
<[command]>+
end<keyword>does NOT work, bestie. Because it would also cheerfully match completely incorrect input like:
for 1..10
echo $n
ls subdir/$n
endif…which is giving chaos. for … endif?? absolutely not. 💀 So we need some way to guarantee that the <keyword> matched right after end is the exact same <keyword> that got matched at the start.
And honestly? That's not hard, because the first <keyword> already captured what it matched into $MATCH{keyword}, so you could just write:
<rule: control_block>
<keyword> <expr>
<[command]>+
end(??{quotemeta $MATCH{keyword}})…but the way that this is simultaneously so useful and so ugly, so scary, and so prone to error is genuinely upsetting. 😭 So Regexp::Grammars hands you a cleaner equivalent:
<rule: control_block>
<keyword> <expr>
<[command]>+
end<\_keyword>A directive of the form <\_IDENTIFIER> is known as a "matchref" — that's short for "%MATCH-supplied backreference". 💅 Matchrefs always attempt to match, as a literal, the current value of $MATCH{IDENTIFIER}.
By default, a matchref does not capture what it matches, but you can absolutely make it do that by giving it an alias:
<token: delimited_string>
<ldelim=str_delim> .*? <rdelim=\_ldelim>
<token: str_delim> ["'`]At first glance this doesn't seem very useful, because by definition $MATCH{ldelim} and $MATCH{rdelim} are necessarily always gonna end up with identical values. But it is useful when the rule also has other alternatives and you want to build a consistent internal representation for all of them, like so:
<token: delimited_string>
<ldelim=str_delim> .*? <rdelim=\_ldelim>
| <ldelim=( \[ ) .*? <rdelim=( \] )
| <ldelim=( \{ ) .*? <rdelim=( \} )
| <ldelim=( \( ) .*? <rdelim=( \) )
| <ldelim=( \< ) .*? <rdelim=( \> )You can also force a matchref to save repeated matches as a nested array, the usual way:
<token: marked_text>
<marker> <text> <[endmarkers=\_marker]>+🚨 Okay but be careful though, because the following will not do what you think it does: <[marker]>+ <text> <[endmarkers=\_marker]>+
and here's why: the value of $MATCH{marker} is gonna be an array reference, which the matchref will flatten and concatenate, then match the resulting string as a literal. Which means the previous example will match endmarkers that are exact multiples of the complete start marker, rather than endmarkers that consist of any number of repetitions of the individual start marker delimiter. So these ones slay:
""text here""
""text here""""
""text here""""""but these ones do not:
""text here"""
""text here"""""Uneven start and end markers like that are extremely unusual, so this problem basically never comes up in practice. 🫶
📌 Note: Prior to Regexp::Grammars version1.020, the syntax for matchrefs was<\IDENTIFIER>instead of<\_IDENTIFIER>. This created problems when the identifier started with any ofl,u,L,U,Q, orE, so the syntax has had to be altered in a backwards incompatible way. It will not be altered again. periodt.
✨ Rematching balanced delimiters ✨
Okay so think back to the example from the last section:
<token: delimited_string>
<ldelim=str_delim> .*? <rdelim=\_ldelim>
| <ldelim=( \[ ) .*? <rdelim=( \] )
| <ldelim=( \{ ) .*? <rdelim=( \} )
| <ldelim=( \( ) .*? <rdelim=( \) )
| <ldelim=( \< ) .*? <rdelim=( \> )That repeated pattern in the last four alternatives is lowkey gauling, right? But no, we can't just refactor those delimiters too:
<token: delimited_string>
<ldelim=str_delim> .*? <rdelim=\_ldelim>
| <ldelim=bracket> .*? <rdelim=\_ldelim>because that would incorrectly match:
{ delimited content here {while failing to match:
{ delimited content here }which is, like, the whole opposite of what we wanted. 🙈 To refactor balanced delimiters like those, we need a second kind of matchref — one that's a little smarter.
Or, preferably, a lot smarter… because there are so many other kinds of balanced delimiters out there besides single brackets. For example:
{{{ delimited content here }}}
/* delimited content here */
(* delimited content here *)
`` delimited content here ''
if delimited content here fiThe common characteristic of all these delimiter pairs is that the closing delimiter is the inverse of the opening delimiter: the sequence of characters is reversed and certain characters (mainly brackets, but also single-quotes/backticks) are mirror-reflected. 🪞✨
Regexp::Grammars supports parsing delimiters like these with a construct known as an invertref, which you specify using the </IDENT> directive. An invertref acts very like a matchref, except that it does not convert to:
(??{ quotemeta( $MATCH{I<IDENT>} ) })but rather to:
(??{ quotemeta( inverse( $MATCH{I<IDENT> ))} })With this directive available, the balanced delimiters from the previous example can be refactored into:
<token: delimited_string>
<ldelim=str_delim> .*? <rdelim=\_ldelim>
| <ldelim=( [[{(<] ) .*? <rdelim=/ldelim>Like matchrefs, invertrefs come in the usual range of flavours:
</ident> # Match the inverse of $MATCH{ident}
<ALIAS=/ident> # Match inverse and capture to $MATCH{ident}
<[ALIAS=/ident]> # Match inverse and push on @{$MATCH{ident}}The character pairs that get reversed during mirroring are: "{" and "}", "[" and "]", "(" and ")", "<" and ">", "AX" and "AX", "`" and "'".
The following mnemonics may be useful for telling inverserefs apart from backrefs: a backref starts with a "\" (just like the standard Perl regex backrefs "\1" and "\g{-2}" and "\k<name>"), whereas an inverseref starts with a "/" (like an HTML or XML closing tag). Or honestly just remember that <\_IDENT> means "match the same again", and if you want "the same again, only mirrored" instead, just mirror the "\" and you get </IDENT>. 🪞💅
✨ Rematching parametric results and delimiters ✨
The <\_IDENTIFIER> and </IDENTIFIER> mechanisms normally locate the literal they're supposed to match by looking in $MATCH{IDENTIFIER}.
However, you can make them look in $ARG{IDENTIFIER} instead, by prefixing the identifier with a single ":". This is especially useful when you're refactoring subrules. So instead of this:
<rule: Command>
<Keyword> <CommandBody> end_ <\_Keyword>
<rule: Placeholder>
<Keyword> \.\.\. end_ <\_Keyword>you could parameterize the Terminator rule, like so:
<rule: Command>
<Keyword> <CommandBody> <Terminator(:Keyword)>
<rule: Placeholder>
<Keyword> \.\.\. <Terminator(:Keyword)>
<token: Terminator>
end_ <\:Keyword>look at that glow-up ✨ so much cleaner.
✨ Tracking and reporting match positions ✨
Regexp::Grammars automatically predefines a special token that makes it so easy to track exactly where in its input a particular subrule matches. That token is: <matchpos>. 📍
The <matchpos> token implements a zero-width match that never fails. It always returns the current index within the string that the grammar is matching.
So for example you could have your <delimited_text> subrule detect and report unterminated text like so:
<token: delimited_text>
qq? <delim> <text=(.*?)> </delim>
|
<matchpos> qq? <delim>
<error: (?{"Unterminated string starting at index $MATCH{matchpos}"})>Matching <matchpos> in the second alternative causes $MATCH{matchpos} to contain the position in the string at which the <matchpos> subrule was matched — in this example, that's the start of the unterminated text. 🎯
If you want the line number instead of the string index, use the predefined <matchline> subrule instead:
<token: delimited_text>
qq? <delim> <text=(.*?)> </delim>
| <matchline> qq? <delim>
<error: (?{"Unterminated string starting at line $MATCH{matchline}"})>⚠️ Note that the line numbers returned by<matchline>start at 1 — not at zero, the way<matchpos>does. Don't get caught out bestie.
And here's the cute part: the <matchpos> and <matchline> subrules are just like any other subrules. You can alias them (<started_at=matchpos>) or match them repeatedly ((?: <[matchline]> <[item]> )++), etc. 🦋 iconic behaviour, truly.
✨ Autoactions ✨
okay so plot twist: this module ALSO does event-based parsing 💖. you can write your grammar the normal way, and then, for one specific parse, you layer a whole collection of call-backs over the top of it — these are called autoactions — and they handle the data as it's being parsed, live, in real time. it's giving main character energy fr.
normally a grammar rule just returns the result hash it accumulated (or whatever else got aliased to MATCH= inside the rule). but bestie, you can specify an autoaction object before the grammar is matched, and everything changes.
once that autoaction object is in place, every single time a rule succeeds during the parse, its result gets passed to that object via one of its methods. which method? the one whose name is the same as the rule's. iconic.
for example, suppose you had a grammar that recognizes simple algebraic expressions:
my $expr_parser = do{
use Regexp::Grammars;
qr{
<Expr>
<rule: Expr> <[Operand=Mult]>+ % <[Op=(\+|\-)]>
<rule: Mult> <[Operand=Pow]>+ % <[Op=(\*|/|%)]>
<rule: Pow> <[Operand=Term]>+ % <Op=(\^)>
<rule: Term> <MATCH=Literal>
| \( <MATCH=Expr> \)
<token: Literal> <MATCH=( [+-]? \d++ (?: \. \d++ )?+ )>
}xms
};you could convert this grammar into a calculator by installing a set of autoactions that convert each rule's result hash into the corresponding value of the sub-expression that the rule just parsed. to do that you'd create a class with methods whose names match the rules whose results you want to change. like so:
package Calculator;
use List::Util qw< reduce >;
sub new {
my ($class) = @_;
return bless {}, $class
}
sub Answer {
my ($self, $result_hash) = @_;
my $sum = shift @{$result_hash->{Operand}};
for my $term (@{$result_hash->{Operand}}) {
my $op = shift @{$result_hash->{Op}};
if ($op eq '+') { $sum += $term; }
else { $sum -= $term; }
}
return $sum;
}
sub Mult {
my ($self, $result_hash) = @_;
return reduce { eval($a . shift(@{$result_hash->{Op}}) . $b) }
@{$result_hash->{Operand}};
}
sub Pow {
my ($self, $result_hash) = @_;
return reduce { $b ** $a } reverse @{$result_hash->{Operand}};
}so objects of this class (and ngl, the class itself too) now have methods corresponding to some of the rules in the expression grammar. to apply those methods to the results of the rules as they parse, you simply install an object as the autoaction handler, right before you kick off the parse:
if ($text ~= $expr_parser->with_actions(Calculator->new)) {
say $/{Answer}; # Now prints the result of the expression
}with_actions() expects to be handed an object or a classname. that object or class gets installed as the autoaction handler for the next match against any grammar. and after that match? the handler gets uninstalled. gone. with_actions() returns the grammar it was called on, which makes it super easy to chain it right into a match — and that's the recommended idiom, bestie 🫶.
with a Calculator object set as the autoaction handler, whenever the Answer, Mult, or Pow rule of the grammar matches, the corresponding Answer, Mult, or Pow method of the Calculator object gets called — with the rule's result value passed as its only argument — and the result of that method becomes the result of the rule. slay 💅.
note that nothing new happens when aTermorLiteralrule matches, because theCalculatorobject doesn't have methods with those names. no method, no action, no drama.
the overall effect, then, is that you get to specify a grammar without any rule-specific behaviours baked in, and then later specify a set of final actions (as methods) for some or all of the rules of the grammar. separation of concerns but make it cute ✨.
and one more thing 🎀: if a particular callback method returns undef, the result of the corresponding rule will be passed through without modification. it's a lil no-op, iykyk.
🦋 Named grammars 🦋
every grammar we've shown so far has been confined to a single regex. but Regexp::Grammars also gives you a mechanism for defining named grammars, which can then be imported into other regexes. this is literally how you modularize common grammatical components. reusable, iconic, obsessed 💖.
🌸 Defining a named grammar
you create a named grammar using the <grammar:...> directive. this directive has to appear before the first rule definition in the grammar, and it takes the place of any start-rule. for example:
qr{
<grammar: List::Generic>
<rule: List>
<[MATCH=Item]>+ % <Separator>
<rule: Item>
\S++
<token: Separator>
\s* , \s*
}x;this creates a grammar named List::Generic, and installs it in the module's internal caches for future reference. it's just... sitting there, waiting, ready to be called up 😌.
note that there's no need (or reason) to assign the resulting regex to a variable, because the named grammar cannot itself be matched against. it's a definition, not a matcher, bestie.
🧚 Using a named grammar
to actually use a named grammar, you incorporate it into another grammar, by inheritance. do that with the <extends:...> directive, like so:
my $parser = qr{
<extends: List::Generic>
<List>
}x;the <extends:...> directive pulls all the rules defined in the specified grammar into the current regex. then you can call any of those rules in the start-pattern. it's giving adopt don't shop 🫶.
👑 Overriding an inherited rule or token
subrule dispatch inside a grammar is always polymorphic. that means when a subrule gets called, the most-derived rule of the same name within the grammar's hierarchy is the one that gets invoked. periodt.
so to replace a particular rule within a grammar, you just inherit that grammar and specify new, more-specific versions of whichever rules you want to change. for example:
my $list_of_integers = qr{
<List>
# Inherit rules from base grammar...
<extends: List::Generic>
# Replace Item rule from List::Generic...
<rule: Item>
[+-]? \d++
}x;you can also use <extends:...> inside other named grammars, to build up whole hierarchies 🌈:
qr{
<grammar: List::Integral>
<extends: List::Generic>
<token: Item>
[+-]? <MATCH=(<.Digit>+)>
<token: Digit>
\d
}x;
qr{
<grammar: List::ColonSeparated>
<extends: List::Generic>
<token: Separator>
\s* : \s*
}x;
qr{
<grammar: List::Integral::ColonSeparated>
<extends: List::Integral>
<extends: List::ColonSeparated>
}x;as you can see in that last example, Regexp::Grammars lets you multiply inherit two (or more) base grammars. so the List::Integral::ColonSeparated grammar takes the definitions of List and Item from the List::Integral grammar, and the definition of Separator from List::ColonSeparated. it's got two parents and it's thriving 💅.
note that grammars dispatch subrule calls using C3 method lookup, not Perl's older DFS lookup. that's exactly whyList::Integral::ColonSeparatedcorrectly picks up the more-specificSeparatorrule defined inList::ColonSeparated, instead of the more-generic version defined inList::Generic(by way ofList::Integral). seeperldoc mrofor more discussion of the C3 dispatch algorithm 📚.
🎀 Augmenting an inherited rule or token
instead of replacing an inherited rule, you can augment it. add to it. build on it. glow-up style ✨.
for example, if you need a grammar for lists of hexadecimal numbers, you could inherit the behaviour of List::Integral and just add the hex digits to its Digit token:
my $list_of_hexadecimal = qr{
<List>
<extends: List::Integral>
<token: Digit>
<List::Integral::Digit>
| [A-Fa-f]
}x;and here's the tea ☕: if you call a subrule using a fully qualified name (like <List::Integral::Digit>), the grammar calls that version of the rule, rather than the most-derived version. you can always reach up the family tree and grab the original, no cap.
💜 Debugging named grammars
named grammars are independent of each other, even when they're inherited. so if debugging is enabled in a derived grammar, it will not be active in any rules inherited from a base grammar — unless that base grammar also included a <debug:...> directive. no cap, they don't share vibes like that.
this is a deliberate design decision, because activating the debugger adds a significant amount of code to each grammar's implementation, and that's detrimental to the matching performance of the resulting regexes. so like... it's for your own good, bestie 😭.
if you need to debug a named grammar, the best approach is to include a <debug: same> directive at the start of the grammar. the presence of that directive ensures the necessary extra debugging code gets included in the regex implementing the grammar, while setting same mode ensures that the debugging mode isn't altered when the matcher uses the inherited rules. chef's kiss 👑
🌈 Common parsing techniques 🌈
🍓 Result distillation
normally, calls to subrules produce nested result-hashes inside the current result-hash. those nested hashes always have at least one automatically supplied key (""), whose value is the entire substring that the subrule matched.
if there are no other nested captures inside the subrule, there'll be no other keys in the result-hash at all. and bestie, that would be annoying — a typical nested grammar would then produce results made of hashes of hashes, with each nested hash having only a single key (""). which would make postprocessing the result-hash (in %/) way more complicated than it needs to be. not the vibe.
to avoid that behaviour: if a subrule's result-hash doesn't contain any keys except "", the module "flattens" the result-hash, replacing it with the value of its single key. so satisfying fr fr.
so for example, the grammar:
mv \s* <from> \s* <to>
<rule: from> [\w/.-]+
<rule: to> [\w/.-]+does not return a result-hash like this:
{
"" => 'mv /usr/local/lib/libhuh.dylib /dev/null/badlib',
'from' => { "" => '/usr/local/lib/libhuh.dylib' },
'to' => { "" => '/dev/null/badlib' },
}instead, it returns:
{
"" => 'mv /usr/local/lib/libhuh.dylib /dev/null/badlib',
'from' => '/usr/local/lib/libhuh.dylib',
'to' => '/dev/null/badlib',
}that is: because the 'from' and 'to' subhashes each have only a single entry, they each get "flattened" down to the value of that entry. clean. tidy. ate 💅.
this flattening also happens if a result-hash contains only "private" keys — i.e. keys starting with underscores. for example:
mv \s* <from> \s* <to><rule: from> <_dir=path>? <_file=filename>
<rule: to> <_dir=path>? <_file=filename>
<token: path> [\w/.-]*/
<token: filename> [\w.-]+so the from rule spits out a result like this:
from => {
"" => '/usr/local/bin/perl',
_dir => '/usr/local/bin/',
_file => 'perl',
}which gets automatically stripped of its "private" keys (the _ ones, you know the vibe), leaving:
from => {
"" => '/usr/local/bin/perl',
}which then gets automatically flattened to:
from => '/usr/local/bin/perl'🎀 List result distillation 🎀
A special case of result distillation shows up in a separated list, like:
<rule: List>
<[Item]>+ % <[Sep=(,)]>If this construct matches just a single item, the result hash will have a single entry — a nested array with a single value — like so:
{ Item => [ 'data' ] }and bestie, instead of returning that annoyingly nested little data structure, you can tell Regexp::Grammars to flatten it down to just the inner data with a special directive:
<rule: List>
<[Item]>+ % <[Sep=(,)]>
<minimize:>The <minimize:> directive looks at the result hash (i.e. %MATCH). If that hash contains only a single entry, and that entry is a reference to an array with a single value, then the directive assigns that single value directly to $MATCH, so that's what gets returned instead of the usual result hash. ✨
Which means a normal separated list still gives you a hash holding all the elements and separators, but a "degenerate" list of only one item gives you just that single item. iconic, honestly.
💅 Manual result distillation 💅
Regexp::Grammars also hands you full manual control over the distillation process. If you use the reserved word MATCH as the alias for a subrule call:
<MATCH=filename>or a subpattern match:
<MATCH=( \w+ )>or a code block:
<MATCH=(?{ 42 })>then the current rule treats the return value of that subrule, pattern, or code block as its complete result, and returns that value instead of the usual result-hash it builds. And this happens even if the result has other entries that would normally also be returned. no cap.
For example, consider a rule like:
<rule: term>
<MATCH=literal>
| <left_paren> <MATCH=expr> <right_paren>The use of MATCH aliases makes the rule return either whatever <literal> returns, or whatever <expr> returns (provided it's between left and right parentheses).
Note that, in this second case, even though <left_paren> and <right_paren> are captured into the result-hash, they are not returned — because the MATCH alias overrides the normal "return the result-hash" semantics and returns only what its associated subrule (i.e. <expr>) produces. it's giving override energy fr fr.
Note also that the return value is only assigned if the subrule call actually matches. For example:
<rule: optional_names>
<[MATCH=name]>*If the repeated subrule call to <name> matches zero times, the return value of the optional_names rule will not be an empty array, because the MATCH= will not have executed at all. Instead, the default return value (an empty string) will be returned. 😭 If you had specifically wanted to return an empty array, you could use any of the following:
<rule: optional_names>
<MATCH=(?{ [] })> # Set up empty array before first match attempt
<[MATCH=name]>*or:
<rule: optional_names>
<[MATCH=name]>+ # Match one or more times
| # or
<MATCH=(?{ [] })> # Set up empty array, if no match🦋 Programmatic result distillation 🦋
It's also possible to control what a rule returns from within a code block. Regexp::Grammars provides a set of reserved variables that give you direct access to the result-hash.
The result-hash itself can be accessed as %MATCH within any code block inside a rule. For example:
<rule: sum>
<X=product> \+ <Y=product>
<MATCH=(?{ $MATCH{X} + $MATCH{Y} })>Here, the rule matches a product (aliased X in the result-hash), then a literal +, then another product (aliased to Y in the result-hash). The rule then executes the code block, which accesses the two saved values (as $MATCH{X} and $MATCH{Y}), adding them together. Because the block is itself aliased to MATCH, the sum produced by the block becomes the (only) result of the rule. ✨
It is also possible to set the rule result from within a code block (instead of aliasing it). The special "override" return value is represented by the special variable $MATCH. So the previous example could be rewritten:
<rule: sum>
<X=product> \+ <Y=product>
(?{ $MATCH = $MATCH{X} + $MATCH{Y} })Both forms are identical in effect. Any assignment to $MATCH overrides the normal "return all subrule results" behaviour. periodt.
Assigning to $MATCH directly is particularly handy if the result may not always be "distillable", for example:
<rule: sum>
<X=product> \+ <Y=product>
(?{ if (!ref $MATCH{X} && !ref $MATCH{Y}) {
# Reduce to sum, if both terms are simple scalars...
$MATCH = $MATCH{X} + $MATCH{Y};
}
else {
# Return full syntax tree for non-simple case...
$MATCH{op} = '+';
}
})Note that you can also partially override the subrule return behaviour. Normally, the subrule returns the complete text it matched as its context substring (i.e. under the "empty key") in its result-hash. That is, of course, $MATCH{""}, so you can override just that behaviour by directly assigning to that entry.
For example, if you have a rule that matches key/value pairs from a configuration file, you might prefer that any trailing comments not be included in the "matched text" entry of the rule's result-hash. You could hide such comments like so:
<rule: config_line>
<key> : <value> <comment>?
(?{
# Edit trailing comments out of "matched text" entry...
$MATCH = "$MATCH{key} : $MATCH{value}";
})Some more examples of the uses of $MATCH:
<rule: FuncDecl>
# Keyword Name Keep return the name (as a string)...
func <Identifier> ; (?{ $MATCH = $MATCH{'Identifier'} })
<rule: NumList>
# Numbers in square brackets...
\[
( \d+ (?: , \d+)* )
\]
# Return only the numbers...
(?{ $MATCH = $CAPTURE })
<token: Cmd>
# Match standard variants then standardize the keyword...
(?: mv | move | rename ) (?{ $MATCH = 'mv'; })🌸 Parse-time data processing 🌸
Using code blocks in rules, it's often possible to fully process data as you parse it. For example, the <sum> rule shown in the previous section might be part of a simple calculator, implemented entirely in a single grammar. Such a calculator might look like this:
my $calculator = do{
use Regexp::Grammars;
qr{
<Answer>
<rule: Answer>
( <.Mult>+ % <.Op=([+-])> )
<MATCH= (?{ eval $CAPTURE })>
<rule: Mult>
( <.Pow>+ % <.Op=([*/%])> )
<MATCH= (?{ eval $CAPTURE })>
<rule: Pow>
<X=Term> \^ <Y=Pow>
<MATCH= (?{ $MATCH{X} ** $MATCH{Y}; })>
|
<MATCH=Term>
<rule: Term>
<MATCH=Literal>
| \( <MATCH=Answer> \)
<token: Literal>
<MATCH= ( [+-]? \d++ (?: \. \d++ )?+ )>
}xms
};
while (my $input = <>) {
if ($input =~ $calculator) {
say "--> $/{Answer}";
}
}Because every rule computes a value using the results of the subrules below it, and aliases that result to its MATCH, each rule returns a complete evaluation of the subexpression it matches, passing that back to higher-level rules, which then do the same.
Hence, the result returned to the very top-level rule (i.e. to <Answer>) is the complete evaluation of the entire expression that was matched. That means that, in the very process of having matched a valid expression, the calculator has also computed the value of that expression, which can then simply be printed directly. main character behaviour honestly 💅
It is often possible to have a grammar fully (or sometimes at least partially) evaluate or transform the data it is parsing, and this usually leads to very efficient and easy-to-maintain implementations.
The main limitation of this technique is that the data has to be in a well-structured form, where subsets of the data can be evaluated using only local information. In cases where the meaning of the data is distributed through that data non-hierarchically, or relies on global state, or on external information, it is often better to have the grammar simply construct a complete syntax tree for the data first, and then evaluate that syntax tree separately, after parsing is complete. The following section describes a feature of Regexp::Grammars that can make this second style of data processing simpler and more maintainable. 🫶
👑 Object-oriented parsing 👑
When a grammar has parsed successfully, the %/ variable will contain a series of nested hashes (and possibly arrays) representing the hierarchical structure of the parsed data.
Typically, the next step is to walk that tree, extracting or converting or otherwise processing that information. If the tree has nodes of many different types, it can be difficult to build a recursive subroutine that can navigate it easily. 😭
A much cleaner solution is possible if the nodes of the tree are proper objects. In that case, you just define a process() or traverse() method for each of the classes, and have every node call that method on each of its children. For example, if the parser were to return a tree of nodes representing the contents of a LaTeX file, then you could define the following methods:
sub Latex::file::explain
{
my ($self, $level) = @_;
for my $element (@{$self->{element}}) {
$element->explain($level);
}
}
sub Latex::element::explain {
my ($self, $level) = @_;
( $self->{command} || $self->{literal})->explain($level)
}sub Latex::command::explain {
my ($self, $level) = @_;
say "\t"x$level, "Command:";
say "\t"x($level+1), "Name: $self->{name}";
if ($self->{options}) {
say "\t"x$level, "\tOptions:";
$self->{options}->explain($level+2)
}
for my $arg (@{$self->{arg}}) {
say "\t"x$level, "\tArg:";
$arg->explain($level+2)
}
}
sub Latex::options::explain {
my ($self, $level) = @_;
$_->explain($level) foreach @{$self->{option}};
}
sub Latex::literal::explain {
my ($self, $level, $label) = @_;
$label //= 'Literal';
say "\t"x$level, "$label: ", $self->{q{}};
}and then bestie you literally just write:
if ($text =~ $LaTeX_parser) {
$/{LaTeX_file}->explain();
}and boom 💥 the whole chain of explain() calls just cascades down through the nodes of the tree, each one firing off the right explain() method depending on what kind of node it bumped into. it's giving recursive self-portrait. iconic.
The only problem?? by default, Regexp::Grammars hands you back a tree of plain-old hashes, not LaTeX::Whatever objects. sad. BUT. fortunately it is super easy to ask for the result hashes to be auto-blessed into the right classes — you just use the <objrule:...> and <objtoken:...> directives. 🎀
These directives are identical to the <rule:...> and <token:...> directives (respectively), except that the rule or token they create will also convert the hash it normally returns into an object of a specified class. How does that conversion go down? It passes the result hash to the class's constructor:
$class->new(\%result_hash)...if the class has a constructor method named new(). Or else — if the class doesn't provide a constructor — it just directly blesses the result hash:
bless \%result_hash, $classngl there's a catch: even if the object gets constructed via its own constructor, the module still expects the new object to be hash-based. If the object is anything but a blessed hash, it fails. The module issues an error in this case. no exceptions, bestie. 💀
The generic syntax for these types of rules and tokens is:
<objrule: CLASS::NAME = RULENAME >
<objtoken: CLASS::NAME = TOKENNAME >For example:
<objrule: LaTeX::Element=component>
# ...Defines a rule that can be called as <component>
# ...and which returns a hash-based LaTeX::Element object
<objtoken: LaTex::Literal=atom>
# ...Defines a token that can be called as <atom>
# ...and which returns a hash-based LaTeX::Literal objectNote that — just like in aliased subrule calls — the name by which something is referred to outside the grammar (in this case, the class name) comes before the =, whereas the name it's referred to inside the grammar comes after the =. so: outside name, equals sign, inside name. keep that order, queen. 👑
You can freely mix object-returning and plain-old-hash-returning rules and tokens inside a single grammar, though you have to be careful not to subsequently try to call a method on any of the unblessed nodes. (they do not have methods. they are just hashes. respect their boundaries. 🙈)
🚨 An important caveat regarding OO rules 🚨
Prior to Perl 5.14.0, Perl's regex engine was not fully re-entrant. Which means: on older versions of Perl, it is not possible to re-invoke the regex engine while you're already inside the regex engine. yeah. regex-ception is off the table. 😭
So this means you need to be careful that the new() constructors that get called by your object-rules do not themselves use regexes in any way — unless you're running under Perl 5.14 or later, in which case you can just ignore everything that follows. lucky you. ✨
The two ways this is most likely to happen are:
- If you're using a class built on Moose, where one or more of the
hasuses a type constraint (such as'Int') that is implemented via regex matching. For example:
has 'id' => (is => 'rw', isa => 'Int');The workaround (for pre-5.14 Perls) is to replace the type constraint with one that doesn't use a regex. For example:
has 'id' => (is => 'rw', isa => 'Num');Alternatively, you could define your own type constraint that avoids regexes:
use Moose::Util::TypeConstraints;
subtype 'Non::Regex::Int',
as 'Num',
where { int($_) == $_ };
no Moose::Util::TypeConstraints;
# and later...
has 'id' => (is => 'rw', isa => 'Non::Regex::Int');- If your class uses an
AUTOLOAD()method to implement its constructor and that method uses the typical:
$AUTOLOAD =~ s/.*://;technique. The workaround here is to achieve the same effect without a regex. For example:
my $last_colon_pos = rindex($AUTOLOAD, ':');
substr $AUTOLOAD, 0, $last_colon_pos+1, q{};⚠️ heads up: this caveat against using nested regexes also applies to any code blocks executed inside a rule or token — whether or not those rules or tokens are object-oriented. so don't think you're sneaky.
✂️ A naming shortcut ✂️
If an <objrule:...> or <objtoken:...> is defined with a class name that is not followed by = and a rule name, then the rule name is determined automatically from the classname. Specifically, the final component of the classname (i.e. after the last ::, if any) is used. so convenient. slay. 💅
For example:
<objrule: LaTeX::Element>
# ...Defines a rule that can be called as <Element>
# ...and which returns a hash-based LaTeX::Element object
<objtoken: LaTex::Literal>
# ...Defines a token that can be called as <Literal>
# ...and which returns a hash-based LaTeX::Literal object
<objtoken: Comment>
# ...Defines a token that can be called as <Comment>
# ...and which returns a hash-based Comment object🐛 Debugging 🐛
Regexp::Grammars comes with a bunch of features specifically designed to help you debug both grammars and the data they parse. it's like a little detective kit for your regexes. 🔍
All debugging messages get written to a log file (which, by default, is just STDERR). But you can specify a disk file explicitly by dropping a <logfile:...> directive at the start of your grammar:
$grammar = qr{
<logfile: LaTeX_parser_log >
\A <LaTeX_file> \Z # Pattern to match
<rule: LaTeX_file>
# etc.
}x;You can also explicitly specify that messages go to the terminal:
<logfile: - >🎀 Debugging grammar creation with <logfile:...> 🎀
Whenever a log file has been directly specified, Regexp::Grammars automatically does verbose static analysis of your grammar. That is: whenever it compiles a grammar containing an explicit <logfile:...> directive, it logs a whole series of messages explaining how it has interpreted the various components of that grammar. receipts!! 🧾
For example, the following grammar:
<logfile: parser_log >
<cmd>
<rule: cmd>
mv <from=file> <to=file>
| cp <source> <[file]> <.comment>?would produce the following analysis in the parser_log file:
info | Processing the main regex before any rule definitions
| |
| |...Treating <cmd> as:
| | | match the subrule <cmd>
| | \ saving the match in $MATCH{'cmd'}
| |
| \___End of main regex
|
info | Defining a rule: <cmd>
| |...Returns: a hash
| |
| |...Treating ' mv ' as:
| | \ normal Perl regex syntax
| |
| |...Treating <from=file> as:
| | | match the subrule <file>
| | \ saving the match in $MATCH{'from'}
| |
| |...Treating <to=file> as:
| | | match the subrule <file>
| | \ saving the match in $MATCH{'to'}
| |
| |...Treating ' | cp ' as:
| | \ normal Perl regex syntax
| |
| |...Treating <source> as:
| | | match the subrule <source>
| | \ saving the match in $MATCH{'source'}
| |
| |...Treating <[file]> as:
| | | match the subrule <file>
| | \ appending the match to $MATCH{'file'}
| |
| |...Treating <.comment>? as:
| | | match the subrule <comment> if possible
| | \ but don't save anything
| |
| \___End of rule definitionThis kind of static analysis is a useful starting point in debugging a miscreant grammar, because it lets you see what you actually specified — as opposed to what you thought you'd specified. huge difference, bestie. 😭
🕹️ Debugging grammar execution with <debug:...> 🕹️
Regexp::Grammars also provides a simple interactive debugger, with which you can watch the process of parsing and the data being collected in any result-hash. main character energy fr fr. 👑
To initiate debugging, place a <debug:...> directive anywhere in your grammar. When parsing reaches that directive, the debugger gets activated, and the command specified in the directive is immediately executed. The available commands are:
<debug: on>— Enable debugging, stop when a rule matches<debug: match>— Enable debugging, stop when a rule matches<debug: try>— Enable debugging, stop when a rule is tried<debug: run>— Enable debugging, run until the match completes<debug: same>— Continue debugging (or not) as currently<debug: off>— Disable debugging and continue parsing silently<debug: continue>— Synonym for<debug: run><debug: step>— Synonym for<debug: try>
These directives can be placed anywhere within a grammar and take effect when that point is reached in the parsing. Hence, adding a <debug:step> directive is very much like setting a breakpoint at that point in the grammar. Indeed, a common debugging strategy is to turn debugging on and off only around a suspect part of the grammar:
<rule: tricky> # This is where we think the problem is...
<debug:step>
<preamble> <text> <postscript>
<debug:off>Once the debugger is active, it steps through the parse, reporting rules that are tried, matches and failures, backtracking and restarts, and the parser's location within both the grammar and the text being matched. That report looks like this:
===============> Trying <grammar> from position 0
> cp file1 file2 |...Trying <cmd>
| |...Trying <cmd=(cp)>
| | \FAIL <cmd=(cp)>
| \FAIL <cmd>
\FAIL <grammar>
===============> Trying <grammar> from position 1
cp file1 file2 |...Trying <cmd>
| |...Trying <cmd=(cp)>
file1 file2 | | \_____<cmd=(cp)> matched 'cp'
file1 file2 | |...Trying <[file]>+
file2 | | \_____<[file]>+ matched 'file1'
| |...Trying <[file]>+
[eos] | | \_____<[file]>+ matched ' file2'
| |...Trying <[file]>+
| | \FAIL <[file]>+
| |...Trying <target>
| | |...Trying <file>
| | | \FAIL <file>
| | \FAIL <target>
<~~~~~~~~~~~~~~ | |...Backtracking 5 chars and trying new match
file2 | |...Trying <target>
| | |...Trying <file>
| | | \____ <file> matched 'file2'
[eos] | | \_____<target> matched 'file2'
| \_____<cmd> matched ' cp file1 file2'
\_____<grammar> matched ' cp file1 file2'Bestie, the first column is literally a GPS pin 📍 — it shows you the exact point in the input where the parser is trying to match, plus any backtracking or forward searching it's gotta do. The rest of the columns are the parser's whole hierarchical journey through the grammar — which rules get tried, which ones actually succeed, and what they matched. It's giving family tree but for regex.
Provided the logfile is a terminal (as it is by default ✨), the debugger also pauses at various points in the parsing process — before trying a rule, after a rule succeeds, or at the end of the parse — all according to the most recent command you issued. When it pauses, you can drop a new command by entering a single letter:
m— to continue until the next subrule matchestors— to continue until the next subrule is triedrorc— to continue to the end of the grammaro— to switch off debugging
No cap, these are just the first letters of the corresponding <debug:...> commands, listed earlier 👑. And if you just hit ENTER while the debugger is paused, it repeats the previous command. Iconic, low-effort, we love her.
While the debugger is paused you can also type a d, which will display the result-hash for the current rule. This can be useful for detecting which rule isn't returning the data you expected. 🕵️♀️
🎀 Resizing the context string 🎀
By default, the first column of the debugger output (which shows the current matching position within the string) is limited to a width of 20 columns. Yes, only 20. She's compact.
However, you can change that limit by calling the Regexp::Grammars::set_context_width() subroutine. You have to specify the fully qualified name, though, because Regexp::Grammars does not export this (or any other) subroutine. Fr fr, no import shortcuts here.
set_context_width() expects a single argument: a positive integer indicating the maximal allowable width for the context column. It issues a warning if an invalid value is passed, and ignores it. 💅
If called in a void context, set_context_width() changes the context width permanently throughout your application. If called in a scalar or list context, set_context_width() returns an object whose destructor will cause the context width to revert to its previous value. Which means you can temporarily change the context width within a given block with something like:
{
my $temporary = Regexp::Grammars::set_context_width(50);
if ($text =~ $parser) {
do_stuff_with( %/ );
}
} # <--- context width automagically reverts at this pointand the context width will change back to its previous value when $temporary goes out of scope at the end of the block. Automatic cleanup?? She's a queen for that. 🫶
💖 User-defined logging with <log:...> 💖
Both static and interactive debugging send a series of predefined log messages to whatever log file you have specified. But plot twist — it is also possible to send additional, user-defined messages to the log, using the <log:...> directive. 🦋
This directive expects either a simple text or a codeblock as its single argument. If the argument is a code block, that code is expected to return the text of the message; if the argument is anything else, that something else is the literal message. For example:
<rule: ListElem>
<Elem= ( [a-z]\d+) >
<log: Checking for a suffix, too...>
<Suffix= ( : \d+ ) >?
<log: (?{ "ListElem: $MATCH{Elem} and $MATCH{Suffix}" })>User-defined log messages implemented using a codeblock can also specify a severity level. If the codeblock of a <log:...> directive returns two or more values, the first is treated as a log message severity indicator, and the remaining values as separate lines of text to be logged. For example:
<rule: ListElem>
<Elem= ( [a-z]\d+) >
<Suffix= ( : \d+ ) >?
<log: (?{
warn => "Elem was: $MATCH{Elem}",
"Suffix was $MATCH{Suffix}",
})>When they are encountered, user-defined log messages are interspersed between any automatic log messages (i.e. from the debugger), at the correct level of nesting for the current rule. Nesting stays immaculate, as it should. ✨
🌈 Debugging non-grammars 🌈
[Note that, with the release in 2012 of the Regexp::Debugger module (on CPAN) the techniques described below are unnecessary. If you need to debug plain Perl regexes, use Regexp::Debugger instead.]
It is possible to use Regexp::Grammars without creating any subrule definitions, simply to debug a recalcitrant regex. Like, sometimes a regex is just refusing to behave and you need the receipts. 😭
For example, if the following regex wasn't working as expected:
my $balanced_brackets = qr{
\( # left delim
(?:
\\ # escape or
| (?R) # recurse or
| . # whatever
)*
\) # right delim
}xms;you could instrument it with aliased subpatterns and then debug it step-by-step, using Regexp::Grammars:
use Regexp::Grammars;
my $balanced_brackets = qr{
<debug:step>
<.left_delim= ( \( )>
(?:
<.escape= ( \\ )>
| <.recurse= ( (?R) )>
| <.whatever=( . )>
)*
<.right_delim= ( \) )>
}xms;
while (<>) {
say 'matched' if /$balanced_brackets/;
}Note the use of amnesiac aliased subpatterns to avoid needlessly building a result-hash. Alternatively, you could use listifying aliases to preserve the matching structure as an additional debugging aid:
use Regexp::Grammars;
my $balanced_brackets = qr{
<debug:step>
<[left_delim= ( \( )]>
(?:
<[escape= ( \\ )]>
| <[recurse= ( (?R) )]>
| <[whatever=( . )]>
)*
<[right_delim= ( \) )]>
}xms;
if ( '(a(bc)d)' =~ /$balanced_brackets/) {
use Data::Dumper 'Dumper';
warn Dumper \%/;
}🔥 Handling errors when parsing 🔥
Assuming you have correctly debugged your grammar, the next source of problems will probably be invalid input (especially if that input is being provided interactively, where users do whatever they want, always). So Regexp::Grammars also provides some support for detecting when a parse is likely to fail... and informing the user why. Bestie behaviour, honestly. 💖
🫶 Requirements 🫶
The <require:...> directive is useful for testing conditions that it's not easy (or even possible) to check within the syntax of the regex itself. For example:
<rule: IPV4_Octet_Decimal>
# Up three digits...
<MATCH= ( \d{1,3}+ )>
# ...but less than 256...
<require: (?{ $MATCH <= 255 })>A require expects a regex codeblock as its argument and succeeds if the final value of that codeblock is true. If the final value is false, the directive fails and the rule starts backtracking. 👑
Note, in this example that the digits are matched with \d{1,3}+. The trailing + prevents the {1,3} repetition from backtracking to a smaller number of digits if the <require:...> fails. That possessive + is the whole plot twist, don't skip it. 💅
🎀 Handling failure 🎀
The module has limited support for error reporting from within a grammar, in the form of the <error:...> and <warning:...> directives and their shortcuts: <...>, <!!!>, and <???> ⭐
💀 Error messages 💀
The <error: MSG> directive queues a conditional error message within @! and then fails to match (that is, it is equivalent to a (?!) when matching). For example:
<rule: ListElem>
<SerialNumber>
| <ClientName>
| <error: (?{ $errcount++ . ': Missing list element' })>So a common code pattern when using grammars that do this kind of error detection is:
if ($text =~ $grammar) {
# Do something with the data collected in %/
}
else {
say {*STDERR} $_ for @!; # i.e. report all errors
}Each error message is conditional in the sense that, if any surrounding rule subsequently matches, the message is automatically removed from @!. This implies that you can queue up as many error messages as you wish, but they will only remain in @! if the match ultimately fails. Moreover, only those error messages originating from rules that actually contributed to the eventual failure-to-match will remain in @!. So like, no leftover drama from storylines that got resolved. 🌸
If a code block is specified as the argument, the error message is whatever final value is produced when the block is executed. Note that this final value does not have to be a string (though it does have to be a scalar).
<rule: ListElem>
<SerialNumber>
| <ClientName>
| <error: (?{
# Return a hash, with the error information...
{ errnum => $errcount++, msg => 'Missing list element' }
})>If anything else is specified as the argument, it is treated as a literal error string (and may not contain an unbalanced < or >, nor any interpolated variables). ⚠️
However, if the literal error string begins with Expected or Expecting , then the error string automatically has the following "context suffix" appended:
, but found '$CONTEXT' insteadFor example:
qr{ <Arithmetic_Expression> # ...Match arithmetic expression
| # Or else
<error: Expected a valid expression> # ...Report error, and fail
# Rule definitions here...
}xms;See, she even tells you what she wanted and what she got instead. Communication skills: unmatched. 🧚
so, like, when the input is invalid, this example is gonna throw you an error message that looks a lil something like:
"Expected a valid expression, but found '(2+3]*7/' instead"The value of the special $CONTEXT variable? Found by looking ahead in the string that's being matched against, to locate the next sequence of non-blank characters after wherever the current parsing position happens to be. And yes bestie, you can also explicitly use this variable inside the <error: (?{...})> form of the directive. 💅
As a special case: if you straight up omit the message from the directive entirely, it gets supplied automatically, derived from the name of the current rule. So if this rule here were to fail to match:
<rule: Arithmetic_expression>
<Multiplicative_Expression>+ % ([+-])
| <error:>then the error message that gets queued would be:
"Expected arithmetic expression, but found 'one plus two' instead"⚠️ But note, note, NOTE: it is still essential to include the colon in the directive. A common mistake is to write:
<rule: Arithmetic_expression>
<Multiplicative_Expression>+ % ([+-])
| <error>which — and this is the kicker — merely attempts to call <rule: error> if the first alternative fails. 💀 Not what you wanted. The colon is not optional, periodt.
✨ Warning messages ✨
Sometimes you wanna detect problems without nuking the entire parse over it. For those occasions, the module hands you a "less stringent" form of error reporting: the <warning:...> directive. 🫶
This directive is exactly the same as an <error:...> in every single respect — except that it does not induce a failure to match at the point where it appears.
So it's useful for reporting non-fatal problems in a parse. Like, for example:
qr{ \A # ...Match only at start of input
<ArithExpr> # ...Match a valid arithmetic expression
(?:
# Should be at end of input...
\s* \Z
|
# If not, report the fact but don't fail...
<warning: Expected end-of-input>
<warning: (?{ "Extra junk at index $INDEX: $CONTEXT" })>
)
# Rule definitions here...
}xms;And bc they don't induce failure, two or more <warning:...> directives can be "stacked" in sequence, exactly like in that example up there. 👑
💀 Stubbing 💀
The module also provides three super handy shortcuts, specifically to make it easy to declare — but not define — rules and tokens.
<...>and<!!!>are equivalent to the directive:
<error: Cannot match RULENAME (not implemented)><???>is equivalent to the directive:
<warning: Cannot match RULENAME (not implemented)>For example, in this grammar here:
<grammar: List::Generic>
<rule: List>
<[Item]>+ % (\s*,\s*)
<rule: Item>
<...>the Item rule is declared but not defined. Which means the grammar will compile correctly, (the List rule won't complain about a call to a non-existent Item), but if the Item rule isn't overridden in some derived grammar, a match-time error will occur when List tries to match the <...> within Item. 😭 iykyk
🌈 Localizing the (semi-)automatic error messages 🌈
Error directives of any of these forms:
<error: Expecting identifier>
<error: >
<...>
<!!!>or their warning equivalents:
<warning: Expecting identifier>
<warning: >
<???>each autogenerate part or all of the actual error message they produce. And by default? That autogenerated message is always produced in English. 🎀
BUT. The module gives you a mechanism to intercept every error or warning that gets queued to @! via these directives... and localize those messages. 🌍
To do this, you call Regexp::Grammars::set_error_translator() — with the full qualification, since Regexp::Grammars does not export it... nor anything else, for that matter. 🙈
The set_error_translator() subroutine expects a single argument, which must be a reference to another subroutine. That subroutine then gets called whenever an error or warning message is queued to @!.
It gets passed three arguments:
- the message string,
- the name of the rule from which the error or warning was queued, and
- the value of
$CONTEXTwhen the error or warning was encountered.
The subroutine is expected to return the final version of the message that's actually gonna be appended to @!. To pull that off it may use one of the many internationalization/localization modules available in Perl, or it may do the conversion entirely by itself. 💪
The first argument is always exactly what appeared as a message in the original directive (regardless of whether that message is supposed to trigger autogeneration, or is just a "regular" error message). That is:
<error: Expecting identifier>→"Expecting identifier"<warning: That's not a moon!>→"That's not a moon!"<error: >→""<warning: >→""<...>→""<!!!>→""<???>→""
The second argument always contains the name of the rule in which the directive was encountered. So like, when invoked from within <rule: Frinstance>, these directives produce:
<error: Expecting identifier>→"Frinstance"<warning: That's not a moon!>→"Frinstance"<error: >→"Frinstance"<warning: >→"Frinstance"<...>→"-Frinstance"<!!!>→"-Frinstance"<???>→"-Frinstance"
Note that the "unimplemented" markers pass the rule name with a preceding -. That lets your translator tell the difference between "empty" messages (which should then be generated automatically) and the "unimplemented" markers (which should report that the rule isn't yet properly defined). 🧠
If you call Regexp::Grammars::set_error_translator() in a void context, the error translator is permanently replaced (at least, until the next call to set_error_translator()).
But if you call Regexp::Grammars::set_error_translator() in a scalar or list context, it returns an object whose destructor will restore the previous translator. Which means you can install a translator only within a given scope, like so:
{
my $temporary
= Regexp::Grammars::set_error_translator(\&my_translator);
if ($text =~ $parser) {
do_stuff_with( %/ );
}
else {
report_errors_in( @! );
}
} # <--- error translator automagically reverts at this point🚨 Warning: any error translation subroutine you install will be called during the grammar's parsing phase (i.e. as the grammar's regex is matching). You should therefore ensure that your translator does not itself use regular expressions, as nested evaluations of regexes inside other regexes are extremely problematical (i.e. almost always disastrous) in Perl. no cap, don't do it.
⏳ Restricting how long a parse runs ⏳
Like the core Perl 5 regex engine on which they're built, the grammars implemented by Regexp::Grammars are essentially top-down parsers. Which means they may occasionally require an exponentially long time to parse a particular input. 😵💫 This usually happens if a particular grammar includes a lot of recursion or nested backtracking, especially if the grammar is then matched against a long string.
The judicious use of non-backtracking repetitions (i.e. x*+ and x++) can significantly improve parsing performance in many such cases. Likewise, carefully reordering any high-level alternatives (so as to test simple common cases first) can substantially reduce parsing times. ✨
However — some languages are just intrinsically slow to parse using top-down techniques (or, at least, may have slow-to-parse corner cases). It's not you, it's the language. 🫠
To help cope with this constraint, Regexp::Grammars provides a mechanism by which you can limit the total effort a given grammar will expend attempting to match. The <timeout:...> directive lets you specify how long a grammar is allowed to keep trying to match before giving up. It expects a single argument, which must be an unsigned integer, and it treats this integer as the number of seconds to continue attempting to match.
For example:
<timeout: 10> # Give up after 10 secondsindicates that the grammar should keep attempting to match for another 10 seconds from the point where the directive is encountered during a parse. If the complete grammar hasn't matched in that time, the entire match is considered to have failed, the matching process is immediately terminated, and a standard error message ('Internal error: Timed out after 10 seconds (as requested)') is returned in @!. 💀
A <timeout:...> directive can be placed anywhere in a grammar, but it's most usually placed at the very start, so that the entire grammar is governed by the specified time limit. The second most common alternative is to place the timeout at the start of a particular subrule that's known to be potentially very slow.
🚩 A common mistake is to put the timeout specification at the top level of the grammar, but place it after the actual subrule to be matched, like so:
my $grammar = qr{
<Text_Corpus> # Subrule to be matched
<timeout: 10> # Useless use of timeout
<rule: Text_Corpus>
# et cetera...
}xms;Since the parser will only reach the <timeout: 10> directive after it has completely matched <Text_Corpus>, the timeout is only initiated at the very end of the matching process and so does not limit that process in any useful way. 🙃 iconic waste of a directive honestly.
🔥 Immediate timeouts 🔥
As you might expect, a <timeout: 0> directive tells the parser to keep trying for only zero more seconds, and therefore will immediately cause the entire surrounding grammar to fail (no matter how deeply within that grammar the directive is encountered). 💥
This can occasionally be extremely useful. If you know that detecting a particular datum means the grammar will never match, no matter how many other alternatives may subsequently be tried, you can short-circuit the parser by injecting a <timeout: 0> immediately after the offending datum is detected. 🎯
For example, if your grammar only accepts certain versions of the language being parsed, you could write:
<rule: Valid_Language_Version>
vers = <%AcceptableVersions>
|
vers = <bad_version=(\S++)>
<warning: (?{ "Cannot parse language version $MATCH{bad_version}" })>
<timeout: 0>okay so that <warning: MSG> <timeout: 0> combo? it's lowkey so useful, and also kinda complex, and also extremely easy to mess up, which is exactly why Regexp::Grammars hands you a shortcut for it: the <fatal:...> directive. a <fatal:...> is exactly equivalent to a <warning:...> followed by a zero-timeout, so bestie, that previous example could also be written as:
<rule: Valid_Language_Version>
vers = <%AcceptableVersions>
|
vers = <bad_version=(\S++)>
<fatal: (?{ "Cannot parse language version $MATCH{bad_version}" })>and like <error:...> and <warning:...> before it, <fatal:...> also provides its own failure context in $CONTEXT, so the previous example could be further simplified to:
<rule: Valid_Language_Version>
vers = <%AcceptableVersions>
|
vers = <fatal:(?{ "Cannot parse language version $CONTEXT" })>also, same energy as <error:...>, <fatal:...> can autogenerate an error message if none is provided, so the example could be still further reduced to:
<rule: Valid_Language_Version>
vers = <%AcceptableVersions>
|
vers = <fatal:>but in this last case, the error message returned in @! would no longer be:
Cannot parse language version 0.95it would now be:
Expected valid language version, but found '0.95' instead✨ Scoping considerations ✨
if you're planning to use a grammar as part of a larger program that contains other (non-grammatical) regexes, it's more efficient — and less error-prone — to avoid having Regexp::Grammars process those regexes as well. so it's often a good idea to declare your grammar in a do block, thereby restricting the scope of the module's effects. 🫶
for example:
my $grammar = do {
use Regexp::Grammars;
qr{
<file>
<rule: file>
<prelude>
<data>
<postlude>
<rule: prelude>
# etc.
}x;
};because the effects of Regexp::Grammars are lexically scoped, any regexes defined outside that do block will be unaffected by the module. periodt. 💅
🌈 INTERFACE 🌈
🎀 Perl API 🎀
use Regexp::Grammars;— causes all regexes in the current lexical scope to be compile-time processed for grammar elements. it's giving compiler magic, fr fr.$str =~ $grammar$str =~ /$grammar/— attempt to match the grammar against the string, building a nested data structure from it. 🌸%/— this hash is assigned the nested data structure created by any successful match of a grammar regex. your data lives here bestie.@!— this array is assigned the queue of error messages created by any unsuccessful match attempt of a grammar regex. the receipts 💀
🦋 Grammar syntax 🦋
✨ Directives ✨
<rule: IDENTIFIER>— define a rule whose name is specified by the supplied identifier.
everything following the <rule:...> directive (up to the next <rule:...> or <token:...> directive) is treated as part of the rule being defined.
any whitespace in the rule is replaced by a call to the <.ws> subrule (which defaults to matching \s*, but may be explicitly redefined). so yes, whitespace is a whole thing here.
<token: IDENTIFIER>— define a rule whose name is specified by the supplied identifier.
everything following the <token:...> directive (up to the next <rule:...> or <token:...> directive) is treated as part of the rule being defined.
any whitespace in the rule is ignored (under the /x modifier), or explicitly matched (if /x is not used).
<objrule: IDENTIFIER><objtoken: IDENTIFIER>— identical to a<rule: IDENTIFIER>or<token: IDENTIFIER>declaration, except that the rule or token will also bless the hash it normally returns, converting it to an object of a class whose name is the same as the rule or token itself. main character energy, but for data structures 👑<require: (?{ CODE }) >— the code block is executed and if its final value is true, matching continues from the same position. if the block's final value is false, the match fails at that point and starts backtracking.<error: (?{ CODE }) ><error: LITERAL TEXT ><error: >— this directive queues a conditional error message within the global special variable@!and then fails to match at that point (that is, it is equivalent to a(?!)or(*FAIL)when matching).<fatal: (?{ CODE }) ><fatal: LITERAL TEXT ><fatal: >— this directive is exactly the same as an<error:...>in every respect except that it immediately causes the entire surrounding grammar to fail, and parsing to immediate cease. it's the nuclear option bestie 💥<warning: (?{ CODE }) ><warning: LITERAL TEXT >— this directive is exactly the same as an<error:...>in every respect except that it does not induce a failure to match at the point it appears. that is, it is equivalent to a(?=)["succeed and continue matching"], rather than a(?!)["fail and backtrack"]. it's just here to gossip, not to ruin the parse.<debug: COMMAND >— during the matching of grammar regexes send debugging and warning information to the specified log file (see<logfile: LOGFILE>).
the available COMMAND's are:
<debug: continue> ___ Debug until end of complete parse
<debug: run> _/
<debug: on> ___ Debug until next subrule match
<debug: match> _/
<debug: try> ___ Debug until next subrule call or match
<debug: step> _/
<debug: same> ___ Maintain current debugging mode
<debug: off> ___ No debuggingsee also the $DEBUG special variable.
<logfile: LOGFILE><logfile: - >— during the compilation of grammar regexes, send debugging and warning information to the specified LOGFILE (or to*STDERRif-is specified).
if the specified LOGFILE name contains a %t, it is replaced with a (sortable) YYYYMMDD.HHMMSS timestamp. for example:
<logfile: test-run-%t >executed at around 9.30pm on the 21st of March 2009, would generate a log file named: test-run-20090321.213056 — timestamped, sorted, iconic ✨
<log: (?{ CODE }) ><log: LITERAL TEXT >— append a message to the log file. if the argument is a code block, that code is expected to return the text of the message; if the argument is anything else, that something else is the literal message.
if the block returns two or more values, the first is treated as a log message severity indicator, and the remaining values as separate lines of text to be logged.
<timeout: INT >— restrict the match-time of the parse to the specified number of seconds. queues a error message and terminates the entire match process if the parse does not complete within the nominated time limit. ⏰
🌸 Subrule calls 🌸
<IDENTIFIER>— call the subrule whose name is IDENTIFIER.
if it matches successfully, save the hash it returns in the current scope's result-hash, under the key 'IDENTIFIER'.
<IDENTIFIER_1=IDENTIFIER_2>— call the subrule whose name is IDENTIFIER_1.
if it matches successfully, save the hash it returns in the current scope's result-hash, under the key 'IDENTIFIER_2'.
in other words, the IDENTIFIER_1= prefix changes the key under which the result of calling a subrule is stored. renaming era 💅
<.IDENTIFIER>— call the subrule whose name is IDENTIFIER. don't save the hash it returns.
in other words, the "dot" prefix disables saving of subrule results.
<IDENTIFIER= ( PATTERN )>— match the subpattern PATTERN.
if it matches successfully, capture the substring it matched and save that substring in the current scope's result-hash, under the key 'IDENTIFIER'.
<.IDENTIFIER= ( PATTERN )>— match the subpattern PATTERN. don't save the substring it matched.<IDENTIFIER= %HASH>— match a sequence of non-whitespace then verify that the sequence is a key in the specified hash
if it matches successfully, capture the sequence it matched and save that substring in the current scope's result-hash, under the key 'IDENTIFIER'.
<%HASH>— match a key from the hash. don't save the substring it matched.<IDENTIFIER= (?{ CODE })>— execute the specified CODE.
save the result (of the final expression that the CODE evaluates) in the current scope's result-hash, under the key 'IDENTIFIER'.
<[IDENTIFIER]>— call the subrule whose name is IDENTIFIER.
if it matches successfully, append the hash it returns to a nested array within the current scope's result-hash, under the key <'IDENTIFIER'>.
<[IDENTIFIER_1=IDENTIFIER_2]>— call the subrule whose name is IDENTIFIER_1.
if it matches successfully, append the hash it returns to a nested array within the current scope's result-hash, under the key 'IDENTIFIER_2'.
<ANY_SUBRULE>+ % <ANY_OTHER_SUBRULE><ANY_SUBRULE>* % <ANY_OTHER_SUBRULE><ANY_SUBRULE>+ % (PATTERN)<ANY_SUBRULE>* % (PATTERN)— repeatedly call the first subrule. keep matching as long as the subrule matches, provided successive matches are separated by matches of the second subrule or the pattern.
in other words, match a list of ANY_SUBRULE's separated by ANY_OTHER_SUBRULE's or PATTERN's.
note that, if a pattern is used to specify the separator, it must be specified in some kind of matched parentheses. these may be capturing [(...)], non-capturing [(?:...)], non-backtracking [(?>...)], or any other construct enclosed by an opening and closing paren.
<ANY_SUBRULE>+ %% <ANY_OTHER_SUBRULE><ANY_SUBRULE>* %% <ANY_OTHER_SUBRULE><ANY_SUBRULE>+ %% (PATTERN)<ANY_SUBRULE>* %% (PATTERN)— repeatedly call the first subrule. keep matching as long as the subrule matches, provided successive matches are separated by matches of the second subrule or the pattern.
also allow an optional final trailing instance of the second subrule or pattern (this is where %% differs from %). it's the "one for the road" separator 🥤
in other words, match a list of ANY_SUBRULE's separated by ANY_OTHER_SUBRULE's or PATTERN's, with a possible final separator.
And about that single <code>%</code> operator, bestie 👀 — if you're using a pattern to say what the separator looks like, that pattern has to live inside some kind of matched parens. Like, it's non-negotiable. And it can be any flavor of parens you want: capturing (...), non-capturing (?:...), non-backtracking (?>...), or honestly any other construct that's wrapped in an opening and a closing paren. Whatever your heart desires, as long as it's parenthesized 💅
✨ Special variables within grammar actions ✨
Okay this is the good part. These are the lil magic variables you get to play with inside your grammar action code blocks. Each one is giving very much main character energy 🌈
$CAPTURE— so is$CONTEXT. And here's the tea: they're both aliases for the built-in read-only$^Nvariable 🫶 That one always holds the substring matched by the nearest preceding(...)capture. And like,$^Nstill works absolutely perfectly fine on its own, no cap — but these two aliases are there so your code blocks and your error messages read better.$CAPTUREfor code,$CONTEXTfor error messages. iykyk 😌$INDEX— this one holds the index where the next match is gonna be attempted inside the string you're parsing 📍 Most commonly you'll see it chilling inside<error:...>or<log:...>directives, like:
<rule: ListElem>
<log: (?{ "Trying words at index $INDEX" })>
<MATCH=( \w++ )>
|
<log: (?{ "Trying digits at index $INDEX" })>
<MATCH=( \d++ )>
|
<error: (?{ "Missing ListElem near index $INDEX" })>See how it's just vibing in there telling you where stuff is happening? chef's kiss 👑
%MATCH— this hash holds every saved result from any subrules that got called from the current rule. So like, when you call subrules like:
<ListElem> <Separator= (,)>they stash their match results in $MATCH{'ListElem'} and $MATCH{'Separator'}. Cute, right? 🎀
$MATCH— and this one is an alias for$MATCH{"="}. It's the %MATCH entry for the special "override value". And here's the important bit: if that entry is defined, its value overrides the usualreturn \%MATCHsemantics of a successful rule. So yeah, it's got the power 💥%ARG— this hash holds all the key/value pairs that were passed into a particular subrule call. So if you've got something like:
<Keyword> <Command> <Terminator(:Keyword)>then the Terminator rule could reach in and grab the text matched by <Keyword> like this:
<token: Terminator>
end_ (??{ $ARG{'Keyword'} })Heads up bestie: to actually match against the calling subrule's'Keyword'value, you have to use either a deferred interpolation(??{...})or a qualified matchref:
<token: Terminator>
end_ <\:Keyword>And here's the common mistake everyone makes (not me, definitely not me 🙈) — trying to directly interpolate the argument:
<token: Terminator>
end_ $ARG{'Keyword'}Nope nope nope. That evaluates $ARG{'Keyword'} when the grammar is compiled, not when the rule is matched. Different times, different vibes, completely broken result 💀
$_— at the start of any code blocks inside any regex, this holds the complete string being matched against. And the current matching position within that string? That'spos($_). Simple as 🌸$DEBUG— this stores the current debugging mode, which can be any of:'off','on','run','continue','match','step', or'try'. It gets set automatically by the<debug:...>command, but you can also set it manually in a code block — which is super useful for conditional debugging. Obsessed. For example:
<rule: ListElem>
<Identifier>
# Conditionally debug if 'foobar' encountered...
(?{ $DEBUG = $MATCH{Identifier} eq 'foobar' ? 'step' : 'off' })
<Modifier>?See also: the <log: LOGFILE> and <debug: DEBUG_CMD> directives 🦋
🚨 IMPORTANT CONSTRAINTS AND LIMITATIONS 🚨
Okay bestie, sit down, grab a snack, because this section is the serious business part. These are the things that will genuinely bite you if you don't know about them. Not being dramatic, it's just facts 😭
- Prior to Perl 5.14, the Perl 5 regex engine was not reentrant. So any attempt to perform a regex match inside a
(?{ ... })or(??{ ... })under Perl 5.12 or earlier will almost certainly lead to either weird data corruption or a segfault 💀
And the same calamities can also occur in any constructor called by <objrule:>. If the constructor invokes another regex in any way, it will most likely fail catastrophically. In particular, this means Moose constructors will frequently crash and burn within a Regex::Grammars grammar — for example, if the Moose-based class declares an attribute type constraint such as 'Int', which Moose checks using a regex. Yeah. Not the vibe.
- The additional regex constructs this module provides are implemented by rewriting regular expressions. This is a (safer) form of source filtering, but still subject to all the same limitations and fallibilities of any other macro-based solution. No escaping that, fr fr.
- In particular, rewriting the macros involves the insertion of (a lot of) extra capturing parentheses. Which means you can no longer assume that particular capturing parens correspond to particular numeric variables — i.e. to
$1,$2,$3etc. If you want to capture directly, use Perl 5.10's named capture construct:
(?<name> [^\W\d]\w* )Better still — capture the data in its correct hierarchical context using the module's "named subpattern" construct:
<name= ([^\W\d]\w*) >That's the move. 🌈
- No recursive descent parser — including those created with Regexp::Grammars — can directly handle left-recursive grammars with rules of the form:
<rule: List>
<List> , <ListElem>So if you find yourself attempting to write a left-recursive grammar (which Perl 5.10 may or may not complain about, but will never successfully parse with), then you probably need to use the "separated list" construct instead:
<rule: List>
<[ListElem]>+ % (,)Problem solved, bestie 💅
- Grammatical parsing with Regexp::Grammars can fail if your grammar uses "non-backtracking" directives — i.e. the
(?>...)block or the?+,*+, or++repetition specifiers. The problem appears to be that preventing the regex from backtracking through the in-regex actions that Regexp::Grammars adds causes the module's internal stack to fall out of sync with the regex match. 💀
For the time being, if your grammar does not work as expected, you may need to replace one or more "non-backtracking" directives with their regular (i.e. backtracking) equivalents. Annoying but true 😭
- Similarly, parsing with Regexp::Grammars will fail if your grammar places a subrule call within a positive look-ahead, since these don't play nicely with the data stack. 🫠
This seems to be an internal problem with perl itself. Investigations, and attempts at a workaround, are proceeding. Fingers crossed, bestie. 🤞
For the time being, you need to make sure that grammar rules don't appear inside a positive lookahead, or use the <?RULENAME> construct instead. That's the workaround. Use it. ✨
💀 DIAGNOSTICS 💀
Okay bestie, listen. First thing you need to know, and I need you to really sit with this: the author could not find a way to throw exceptions from inside a regex. So none of these diagnostics actually throw an exception. None. Zero. Zip. 💀
Instead, these messages just get written to the parser logfile you specified — or to *STDERR if you never specified one. That's it. That's the whole delivery mechanism.
But (and this is the part that matters) any fatal match-time message will immediately terminate the parser matching, and it will still set $@ — exactly as if an exception had been thrown and caught right there in the code. So you have the option to check $@ immediately after matching with the grammar, and rethrow it yourself if you need to:
if ($input =~ $grammar) {
process_data_in(\%/);
}
else {
die if $@;
}"Found call to %s, but no %s was defined in the grammar"
You called a subrule that has no definition anywhere in the grammar. Classic. Usually it's because you forgot to define the rule, or you misspelled either the definition or the subrule call. Like this:
<file>
<rule: fiel> <---- misspelled rule
<lines> <---- used but never definedRegexp::Grammars turns any such subrule-call attempt into an instant catastrophic failure of the entire parse. So if your parser ever actually tries to make that call, Very Bad Things will happen. 🙈
"Entire parse terminated prematurely while attempting to call non-existent rule: %s"
So you ignored the previous error and actually tried to call a subrule that has no definition in the grammar. Well. Very Bad Things are now happening. The parser got very upset, took its ball, and went home. 🏀 See the preceding diagnostic for remedies.
This one should throw an exception, but it can't. So it sets $@ instead, which lets you trap the error manually if you feel like it. 🫶
"Fatal error: <objrule: %s> returned a non-hash-based object"
You used an <objrule:> and it returned a blessed object that wasn't a hash. This will break the behaviour of the grammar, so the module immediately reports the problem and gives up.
The fix? Use only hash-based classes with <objrule:>. 💅
"Can't match against <grammar: %s>"
The regex you tried to match against defined a pure grammar, using the <grammar:...> directive. Pure grammars have no start-pattern, and therefore cannot be matched against directly. Periodt.
You need a matchable grammar that inherits from your pure grammar, and then calls one of its rules. So instead of this:
my $greeting = qr{
<grammar: Greeting>
<rule: greet>
Hi there
| Hello
| Yo!
}xms;you need this:
qr{
<grammar: Greeting>
<rule: greet>
Hi there
| Hello
| Yo!
}xms;
my $greeting = qr{
<extends: Greeting>
<greet>
}xms;"Inheritance from unknown grammar requested by <%s>"
You used an <extends:...> directive asking your grammar to inherit from another grammar, but the grammar you asked for doesn't exist. 😭
Double-check the spelling of the grammar name, and make sure it's already been defined somewhere earlier in your program.
"Redeclaration of <%s> will be ignored"
You defined two or more rules or tokens with the same name. The first one defined in the grammar wins; the rest get ignored.
Want the warning gone? Get rid of the extra definitions. Or, at bare minimum, comment them out or rename the rules. ✨
"Possible invalid subrule call %s"
Your grammar contained something shaped like this:
<identifier
<.identifier
<[identifierwhich you might have meant as a subrule call, but which didn't correctly parse as one. If it was supposed to be a Regexp::Grammars subrule call, go check the syntax you used. If it wasn't supposed to be a subrule call at all, you can silence the warning by rewriting it and quoting the leading angle:
\<identifier
\<.identifier
\<[identifier"Possible failed attempt to specify a subrule call or directive: %s"
Your grammar contained something shaped like this:
<identifier...but it wasn't a call to a known subrule or directive. If it was meant to be a subrule call, check the spelling of the rule name inside the angles. If it was meant to be a Regexp::Grammars directive, check the spelling of the directive name. And if it was never meant to be a subrule call or directive at all, silence the warning by rewriting it and quoting the leading angle:
\<identifier"Invalid < metacharacter"
The < character is always special in Regexp::Grammars regexes: it either introduces a subrule call, or a rule/token declaration, or a directive.
Need to match a literal <? Use \< in your regex. That's it, that's the tea. 🍓
"Invalid separation specifier: %s"
You used a % or a %% in the regex, but in a way that won't do what you expect. Both % and %% are metacharacters in Regexp::Grammars regexes, and they can only be placed between a repeated atom (the thing matching a list of items) and a simple atom (the thing matching the separator between list items). See "Matching separated lists".
So if you were using % or %% as a metacharacter, you either forgot the repetition quantifier (*, +, {0,9}, etc.) on the preceding list-matching atom, or you specified the following separator atom as something too complex for the module to parse — for example, a set of parens with nested subrule calls. 😵💫
On the other hand, if you were intending to match a literal % or %% inside a Regexp::Grammars regex, then you have to explicitly say it's a literal by quotemeta'ing it, like so: \% or \%\%
"Repeated subrule %s will only capture its final match"
You specified a subrule call with a repetition qualifier, like:
<ListElem>*or:
<ListElem>+Here's the thing: each subrule call saves its result in a hash entry of the same name, so each repeated match overwrites the previous ones. Only the last match actually gets saved. If you want to keep all the matches, you need to tell Regexp::Grammars to save the sequence of results as a nested array inside the hash entry, like so:
<[ListElem]>*or:
<[ListElem]>+If you genuinely did mean to throw away every result except the final one, you can silence the warning by wrapping the subrule call in any kind of parentheses. For example:
(<ListElem>)*or:
(?: <ListElem> )+"Unable to open log file '$filename' (%s)"
You specified a <logfile:...> directive, but the file whose name you gave could not be opened for writing — the reason is right there in the parens.
Did you misspell the filename? Or get the permissions wrong somewhere in the filepath? 🧚
"Non-backtracking subrule %s may not revert correctly during backtracking"
Because of inherent limitations in the Perl regex engine, non-backtracking constructs like ++, *+, ?+, and (?>...) do not always work correctly when applied to subrule calls — especially in earlier versions of Perl. 💀
If the grammar doesn't work properly, replace the offending constructs with regular backtracking versions instead. If the grammar does work, you can silence the warning by enclosing the subrule call in any kind of parentheses. For example, change:
<[ListElem]>++to:
(?: <[ListElem]> )++"Unexpected item before first subrule specification in definition of <grammar: %s>"
Named grammar definitions must consist only of rule and token definitions. They cannot have patterns before the first definitions. You had some kind of pattern before the first definition, and it will be completely ignored within the grammar.
To silence the warning, either comment out or delete whatever comes before the first rule/token definition.
"No main regex specified before rule definitions"
You specified an unnamed grammar (i.e. no <grammar:...> directive), but you didn't give it anything to actually match — just some rules you never actually call. For example:
my $grammar = qr{
<rule: list> \( <item> +% [,] \)
<token: item> <list> | \d+
}x;You have to provide something before the first rule to kick the matching off. For example:
my $grammar = qr{
<list> # <--- This tells the grammar how to start matching
<rule: list> \( <item> +% [,] \)
<token: item> <list> | \d+
}x;"Ignoring useless empty <ws:> directive"
The <ws:...> directive specifies what whitespace matches within the current rule. An empty <ws:> directive would make whitespace match nothing at all — which is what happens in a token definition, not in a rule definition.
So either put some subpattern inside that empty <ws:...>, or, if you really do want whitespace to match nothing at all, remove the directive completely and change the rule definition into a token definition. 🌸
"Ignoring useless <ws: %s > directive in a token definition"
The <ws:...> directive is used to specify what whitespace matches within a rule. Since whitespace never matches anything inside tokens, putting a <ws:...> directive in a token is a waste of time. 🙈
Either remove the useless directive, or change the surrounding token definition into a rule definition.
"Quantifier that doesn't quantify anything: <%s>"
You specified a rule or token something like:
<token: star> *or:
<rule: add_op> plus | add | +but the * and + in those examples are both regex meta-operators — quantifiers that usually cause whatever precedes them to match repeatedly. In these cases, though, nothing precedes the quantifier, so it's a Perl syntax error. 😭
You almost certainly need to escape those meta-characters somehow. For example:
<token: star> \*
<rule: add_op> plus | add | [+]🛠️ CONFIGURATION AND ENVIRONMENT 🛠️
Regexp::Grammars requires no configuration files and no environment variables. Nothing to set up, bestie. It just goes. ✨
📦 DEPENDENCIES 📦
This module only works under Perl 5.10 or later. No cap. 💅
💀 INCOMPATIBILITIES 💀
ok so bestie, listen. This module is likely to be incompatible with literally any other module that automagically rewrites regexes. like, it's a whole thing. ✋
For example, it may conflict with Regexp::DefaultFlags, Regexp::DeferredExecution, or Regexp::Extended.
that's it. that's the warning. don't put these in the same room, they will fight. 💅
🐛 BUGS 🐛
No bugs have been reported. iconic behavior honestly ✨
Please report any bugs or feature requests to "bug-regexp-grammars@rt.cpan.org", or through the web interface at <http://rt.cpan.org>.
🫶 CONTRIBUTING 🫶
Patches and other similar contributions are always welcome. like genuinely, come thru 🌸
For more details on how best to contribute improvements to this module, see <https://metacpan.org/dist/Regexp-Grammars/contribute> or the CONTRIBUTING file in the module's distribution.
👑 AUTHOR 👑
Damian Conway "<DCONWAY@CPAN.org>"
the main character fr fr 🎀
⭐ LICENCE AND COPYRIGHT ⭐
Copyright (c) 2009, Damian Conway "<DCONWAY@CPAN.org>". All rights reserved.
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.
🌈 DISCLAIMER OF WARRANTY 🌈
ok this next part is in ALL CAPS because it's legally required to be, not because anyone's yelling at you 💜
BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENCE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
and that's the whole page, bestie!! 🦋 you made it to the end of Regexp::Grammars(3pm) — this copy came from perl v5.36.0, dated 2022-10-22. go write something grammary and slay 💖✨
📦 where this page comes from
- debian bookworm: libregexp-grammars-perl 1.058-2 · Artistic-1.0-Perl OR GPL-1.0-or-later
- debian trixie: libregexp-grammars-perl 1.058-2 · Artistic-1.0-Perl OR GPL-1.0-or-later
- ubuntu noble: libregexp-grammars-perl 1.058-2 · Artistic-1.0-Perl OR GPL-1.0-or-later
- ubuntu resolute: libregexp-grammars-perl 1.058-2 · Artistic-1.0-Perl OR GPL-1.0-or-later
- ubuntu stonking: libregexp-grammars-perl 1.058-2 · Artistic-1.0-Perl OR GPL-1.0-or-later
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms