Parser::MGC(3pm)

💖 Parser::MGC 💖

the manual page · Parser::MGC(3pm) · User Contributed Perl Documentation

---

✨ NAME ✨

Parser::MGC — bestie, it builds simple recursive-descent parsers 🧚

---

🌸 SYNOPSIS 🌸

okay so here's the whole vibe:

package My::Grammar::Parser;
use base qw( Parser::MGC );

sub parse
{
   my $self = shift;

   $self->sequence_of( sub {
      $self->any_of(
         sub { $self->token_int },
         sub { $self->token_string },
         sub { \$self->token_ident },
         sub { $self->scope_of( "(", \&parse, ")" ) }
      );
   } );
}

my $parser = My::Grammar::Parser->new;

my $tree = $parser->from_file( $ARGV[0] );

...

like. that's it. that's the whole parser. iconic behavior tbh 👑

---

💅 DESCRIPTION 💅

So Parser::MGC is a base class, right? It hands you a low-level framework for building recursive-descent parsers that chomp a given input string from left to right and hand back a parse structure. And the name? It comes from the m//gc regexps used to implement all the token-parsing behavior. iykyk 💅

It gives you a whole bunch of token-parsing methods, and each one extracts a grammatical token out of the string. It ALSO gives you wrapping methods you can use to build up a possibly-recursive grammar structure, by applying a structure around other parts of parsing code. cute, right? 🎀

🦋 Backtracking 🦋

Every single method — token ones AND structural ones — is atomic. It either consumes a prefix of the string and returns its result, or it fails and consumes nothing. Zero. Nada. That's the whole reason implementing grammars that need backtracking is so easy with this thing. periodt.

Several of the structure-forming methods have some form of "optional" behavior. Meaning: they can optionally consume some amount of input, or take some particular choice — but if the code you invoked inside that goes and fails, the structure can backtrack and take some different behavior instead. And honestly? That's usually exactly what you want when you're testing whether the structure of the input string matches some part of the grammar that's optional, or that has multiple choices. 💖

BUT — and listen, this is the important part — once the choice of grammar has been made, it's often useful to be able to fix on that one choice, so that subsequent failures propagate up rather than taking that alternative behavior. Control of this backtracking comes from the commit method. And careful use of commit is genuinely one of the key advantages that Parser::MGC has over just doing more simple parsing using single regexps alone. no cap 🧚

👑 Stall Detection 👑

Most of the methods in this class have bounded execution time, but some methods — namely list_of and sequence_of — repeatedly recurse into other code to build up a list of results until some ending condition gets reached. And that's where a whole class of bug can sneak in: whatever they recurse into might successfully match an empty string, and thus make absolutely no progress. 💀

So these methods will automatically detect that situation. How? If they repeatedly encounter the same string position more than a certain number of times (given by the stallcount argument), that's the signal. And if that count is reached, the entire parse attempt will be aborted by the die method. so yeah. don't stall. 🙈

---

🌈 CONSTRUCTOR 🌈

🍓 new

$parser = Parser::MGC->new( %args );

Returns a new instance of a Parser::MGC object. Important: this must be called on a subclass that provides a method of the name provided as toplevel — which by default is called parse. 👑

Takes the following named arguments:

• toplevel => STRING — the name of the toplevel method to use to start the parse from. If you don't supply it, it'll try to use a method called parse. ✨
• patterns => HASH — the keys in this hash should map to quoted regexp (qr//) references, so you can override the default patterns used to match tokens. Peep PATTERNS below for the full tea. 💅
• accept_0o_oct => BOOL — if true, the token_int method will also accept integers with a 0o prefix as octal. yes queen, octal rights 🎀
• stallcount => INT — Since version 0.21. The number of times that the stall-detector would have to see the same position before it aborts the parse attempt. If you don't supply it, a default of 10 applies. 🔥

---

💜 PATTERNS 💜

The following pattern names are recognised. You can pass them to the constructor in the patterns hash, or provide them as a class method under the name pattern_name. 🦋

• ws — pattern used to skip whitespace between tokens. Defaults to /[\s\n\t]+/
• comment — pattern used to skip comments between tokens. Undefined by default.
• int — pattern used to parse an integer by token_int. Defaults to /-?(?:0x[[:xdigit:]]+|[[:digit:]]+)/. If accept_0o_oct is given, then this will be expanded to match /0o[0-7]+/ as well. ⭐
• float — pattern used to parse a floating-point number by token_float. Defaults to /-?(?:\d*\.\d+|\d+\.)(?:e-?\d+)?|-?\d+e-?\d+/i.
• ident — pattern used to parse an identifier by token_ident. Defaults to /[[:alpha:]_]\w*/
• string_delim — pattern used to delimit a string by token_string. Defaults to /["']/.

---

🎀 SUBCLASSING METHODS 🎀

The following optional methods may be defined by subclasses, to customise their parsing. go off, customize ur heart out 💖

🌸 on_parse_start

$parser->on_parse_start;

Since version 0.21.

If defined, it's invoked by the from_* method that begins a new parse operation — just before invoking the toplevel structure method. ✨

🌸 on_parse_end

$result = $parser->on_parse_end( $result );

Since version 0.21.

If defined, it's invoked by the from_* method once it has finished the toplevel structure method. It gets passed the tentative result from the structure method, and whatever it returns becomes the result of the from_* method itself. so like, you're the main character here. shape the destiny of that result 👑

---

✨ METHODS ✨

Okay bestie, this is where the actual doing stuff happens. Grab a snack 🍓, we're going in.

---

🌸 from_string

$result = $parser->from_string( $str );

Hands the parser a literal string and goes "okay babe, parse this". You get back whatever your toplevel method returned. That's it, that's the whole vibe. 💅

---

🦋 from_file

$result = $parser->from_file( $file, %opts );

Same deal but for a file. $file can be a pathname sitting in a string, or an already-opened IO handle — the parser doesn't care, it'll figure it out. Result comes back from the toplevel method, no cap.

Here's the options it recognises:

• binmode => STRING — if you set this, that binmode gets applied to the filehandle before any reading happens. The classic move is using it to set the file's encoding, like this:

$parser->from_file( $file, binmode => ":encoding(UTF-8)" );

---

🎀 filename

$filename = $parser->filename;

Since version 0.20. 👑

Gives you back the name of the file that's currently being parsed — but only if you call it from inside from_file. Otherwise, bestie, there's nothing to report.

---

🧚 from_reader

$result = $parser->from_reader( \&reader );

Since version 0.05.

Parses input that comes from the reader function you hand it. That function gets called in scalar context to spit out chunks of string to parse, and it receives the $parser object as its argument. When it's got no more string left to give? It should return undef. Simple.

$reader->( $parser );

Okay real talk, this one has a catch and you need to hear it. Because it's basically impossible to detect exactly when more input is needed (failed regexp parsing is sneaky like that), the reader function only ever gets invoked while the parser is hunting for skippable whitespace. So it's perfect for reading lines of a file — in the common case where lines count as skippable whitespace — or for reading interactive input from a user line by line. But it cannot be used in all cases. Say you're reading fixed-size buffers from a file: two successive calls can split a single token right across the buffer boundary, and boom, parse failures. So don't do that. 💀

---

🌈 pos

$pos = $parser->pos;

Since version 0.09.

The current parse position, as a character offset from the very beginning of the file or string. Little coordinate check-in, you know? 📍

---

💜 take

$str = $parser->take( $len );

Since version 0.16.

Yoinks the next $len characters straight from the input — before any whitespace or comment skipping gets a say. And it completely ignores any end-of-scope marker that might be pending. This exists for the girlies parsing partially-binary protocols, or any other situation where it would be straight-up wrong for the end-of-scope marker to kick in at this moment.

---

🫶 where

( $lineno, $col, $text ) = $parser->where;

Gives you the current parse position as a line number, a column number, and the entire current line of text. Heads up: the first line is numbered 1, and the first column is numbered 0. Asymmetric? Yes. Iconic? Also yes.

---

🔥 fail / fail_from

$parser->fail( $message );

$parser->fail_from( $pos, $message );

fail_from is since version 0.09.

This aborts the current parse attempt with the message you give it. The failure message will include the line and column position, plus the line of input that failed — at the current parse position for fail, or at a position you grabbed earlier using the pos method for fail_from.

And then it propagates! It travels up to the inner-most structure parsing method that hasn't been committed yet. Or, if there are literally no further options left to take, it takes the whole entire parser down with it. 😭

---

👑 die / die_from

$parser->die( $message );

$parser->die_from( $pos, $message );

Since version 0.20.

Throws an exception that propagates like a normal die — meaning it goes entirely out of the whole parser and straight to whoever called the toplevel from_* method that started all this. It bypasses all of the back-tracking logic. No detours, no second chances. 💅

It's basically core's die, except the message string gets the line and column position appended, plus the line of input the parser was working on — exactly like the fail method does.

Use this one for reporting fatal errors: situations where your grammar did correctly recognise the input, but it's asking for something that just cannot be fulfilled semantically. Recognised, but impossible. Tragic.

---

💖 warn / warn_from

$parser->warn( $message );

$parser->warn_from( $pos, $message );

Since version 0.22.

Issues a warning, exactly like core Perl's warn does — but it appends info to the message string: the current line and column position, and the line of input the parser was working on. Same energy as the fail method's message. Gentle nudge instead of a full crash-out. 🎀

---

⭐ at_eos

$eos = $parser->at_eos;

Returns true if the input string is at the end of the string. Are we at the end? Vibe check: yes or no. 🌸

---

🦄 scope_level

$level = $parser->scope_level;

Since version 0.05.

Returns how many nested scope_of calls have been made so far. How deep in the nesting are we, bestie? This tells you.

---

🍓 include_string

$result = $parser->include_string( $str, %opts );

Since version 0.21.

Parses a given string into the existing parser object — same object, new content.

The current parser state gets moved aside for the duration of this method, and the given string takes its place. Then the toplevel parser method (or a different one, if you specified it) is invoked over that string. Whatever that returns is what this method returns.

The classic use case here is some kind of "include" or "macro expansion" ability — injecting new content in as if the current parse location had stumbled right into it. And here's the cute part: other than the internal parser state, the object's other fields are not altered. So whatever effects your invoked parsing methods have on it, they can keep inspecting and altering it as needed. The object stays live the whole time. 🫶

Options it recognises:

• filename => STRING — if set, gives a filename (or any other descriptive text, honestly) to pretend is the source of this string. It does not need to be a real file on the filesystem — for example, it could just explain where the string came from in some other way. This is the value reported by the filename method and printed in failure messages.
• toplevel => STRING | CODE — if set, gives the toplevel parser method to use within this inclusion, overriding the object's own defined default.

---

✨ STRUCTURE-FORMING METHODS ✨

okay bestie, this is where it gets good. These are the methods that let you build an actual grammatical structure out of all those little basic token-parsing methods we already talked about. The way that they work is: every single one of them takes at least one code reference, and that code reference gets handed the actual $parser object as its first argument. so it can, yk, do parser things. 🫶

And here's the cute part — anywhere a code reference is expected, you can also just pass a plain string with the name of a method to invoke instead. That's literally all you need in tons of simple cases, like:

$self->any_of(
   'token_int',
   'token_string',
   ...
);

see? no sub { } necessary. slay. 💅

---

🦋 maybe

$ret = $parser->maybe( $code );

So maybe tries to run the given $code in scalar context, and just... hands you back whatever it returned. It's fully accepting that it might fail. no judgement here. 🎀 $code can be either a CODE reference or a method name given as a string, your choice bestie.

$ret = $parser->maybe( $code, @args );

Since version 0.22, this method passes any extra arguments into the invoked code. This is especially useful if $code is a method name — like, game changer, honestly.

Now the important part, so listen up: if the code fails (either by calling "fail" itself, or by propagating a failure from another method it invoked) before it has invoked "commit", then none of the input string gets consumed — the current parsing position is restored, no harm done. And "undef" gets returned in that case.

But if it calls "commit"? Then any subsequent failure gets propagated right up to the caller instead of just returning "undef". That's the difference, periodt.

You can lowkey think of this as the ? regexp qualifier. iconic.

sub parse_declaration
{
   my $self = shift;

   [ $self->parse_type,
     $self->token_ident,
     $self->maybe( sub {
        $self->expect( "=" );
        $self->parse_expression
     } ),
   ];
}

---

🧚 scope_of

$ret = $parser->scope_of( $start, $code, $stop );

scope_of expects to find the $start pattern, then attempts to execute the given $code, then expects to find the $stop pattern. Whatever the code returned, you get back. $code can be a CODE reference or a method name given as a string. same deal as always. 🌸

While the code is being executed, that $stop pattern gets used by the token parsing methods as an end-of-scope marker — which means they'll raise a failure if they get called at the end of a scope. it's giving very much "we're done here, don't @ me". 💀

sub parse_block
{
   my $self = shift;

   $self->scope_of( "{", 'parse_statements', "}" );
}

If the $start pattern is "undef", it's presumed the caller already checked for this. This is super useful when the stop pattern needs to be calculated based on the start pattern:

sub parse_bracketed
{
   my $self = shift;

   my $delim = $self->expect( qr/[\(\[\<\{]/ );
   $delim =~ tr/([<{/)]>}/;

   $self->scope_of( undef, 'parse_body', $delim );
}

heads up: this method does not have any optional parts to it. any failures are immediately propagated to the caller. no takebacks, no backsies. 😭

---

👑

💜 committed_scope_of

$ret = $parser->committed_scope_of( $start, $code, $stop );

Since version 0.16. ✨

This is a variant of "scope_of" that calls "commit" right after a successful match of the start pattern. Basically: this is usually what you want if you're using "scope_of" from inside an "any_of" choice, and no other alternative following this one could possibly match if the start pattern already has. so like, don't even bother trying the others. main character energy fr fr. 👑

---

🍓 list_of

$ret = $parser->list_of( $sep, $code );

list_of expects to find a list of instances of something parsed by $code, all separated by the $sep pattern. What you get back is an ARRAY ref containing a list of the return values from the $code. A single trailing delimiter is allowed, and it does not affect the return value — generous queen. 💅

$code may either be a CODE reference or a method name given as a string. It gets called in list context, and whatever values it returns get appended to the eventual result — similar to perl's "map". 🦋

Also very important: this method does not consider it an error if the returned list is empty. That is, the scope ended before any item instances were parsed from it. empty is valid, bestie.

sub parse_numbers
{
   my $self = shift;

   $self->list_of( ",", 'token_int' );
}

If the code fails (either by invoking "fail" itself, or by propagating a failure from another method it invoked) before it has invoked "commit" on a particular item, then that item is aborted and the parsing position gets restored to the beginning of that failed item. The list of results from previous successful attempts will still be returned. so you keep the wins, you just drop the L. 🫶

But if it calls "commit" within an item? Then any subsequent failure for that item causes the entire "list_of" to fail, and that gets propagated to the caller. whole thing goes down together. 😭

---

🌈 sequence_of

$ret = $parser->sequence_of( $code );

This is just a shortcut for calling "list_of" with an empty string as the separator. It expects to find at least one instance of something parsed by $code, separated only by skipped whitespace. chill, effortless, minimal. ✨

You can think of this as being similar to the + or * regexp qualifiers.

sub parse_statements
{
   my $self = shift;

   $self->sequence_of( 'parse_statement' );
}

The interaction of failures in the code and the "commit" method is identical to that of "list_of". so go read that one again if you forgot. 💜

---

🎀 any_of

$ret = $parser->any_of( @codes );

Since version 0.06. 💖

any_of expects that one of the given code instances can parse something from the input, and it returns what that one returned. Each code instance can indicate a failure to parse by calling the "fail" method or otherwise propagating a failure. And yes, each code instance may either be a CODE reference or a method name given as a string. we love flexibility.

This is similar to the | regexp operator for forming alternations of possible parse trees. choose-your-own-adventure vibes. 🌸

sub parse_statement
{
   my $self = shift;

   $self->any_of(
      sub { $self->parse_declaration; $self->expect(";") },
      sub { $self->parse_expression; $self->expect(";") },
      sub { $self->parse_block },
   );
}

If the code for a given choice fails (either by invoking "fail" itself, or by propagating a failure from another method it invoked) before it has invoked "commit" itself, then the parsing position is restored and the next choice gets attempted. next!! 💅

If it calls "commit", then any subsequent failure for that choice causes the entire "any_of" to fail, propagating that to the caller, and no further choices will be attempted. that's it, we're locked in.

Now, if none of the choices match, a simple failure message gets printed:

Found nothing parseable

ngl bestie, that message is not gonna be helpful to your users. so like, do better: provide a better message via the final choice instead. But don't forget to "commit" before printing the failure message, or it won't count. crucial. 🚨

$self->any_of(
   'token_int',
   'token_string',
   ...,

   sub { $self->commit; $self->fail( "Expected an int or string" ) }
);

---

💅 commit

$parser->commit;

Calling this method cancels the backtracking behaviour of the innermost "maybe", "list_of", "sequence_of", or "any_of" structure-forming method. Like, it locks in the vibe. 🔥

Which means: if later code then calls "fail", the exception will be propagated out of "maybe", no further list items will be attempted by "list_of" or "sequence_of", and no further code blocks will be attempted by "any_of". the whole thing commits, no takebacks.

Typically you call this once the grammatical structure alter has been determined, so that any further failures get raised as real exceptions rather than by trying other alternatives. it's giving decisive. 💅

sub parse_statement
{
   my $self = shift;

   $self->any_of(
      ...
      sub {
         $self->scope_of( "{",
            sub { $self->commit; $self->parse_statements; },
         "}" ),
      },
   );
}

Though honestly, in this super common pattern, "committed_scope_of" may be used instead. save yourself the typing, bestie. 🫶

✨ expect ✨

$str = $parser->expect( $literal );

$str = $parser->expect( qr/pattern/ );

@groups = $parser->expect( qr/pattern/ );

Okay so this one is like, the bread and butter 💅 it's gonna find you a literal string or a regexp pattern match, and then it consumes it. In scalar context you get back the string it captured. In list context you get the matching substring plus the contents of any subgroups inside the pattern. Bestie, the subgroups just show up, no cap.

If the regexp fails to match it will raise a parse error by calling fail. And heads up, if your pattern could match an empty string (like, say, qr/\d*/) then it will always match, even when it has to match an empty string to do it. A zero-width match is not considered a failure by this method. So like, be aware 🫶

---

✨ maybe_expect ✨

$str = $parser->maybe_expect( ... );

@groups = $parser->maybe_expect( ... );

Since version 0.10.</purl>

It's literally just a cute shortcut for calling expect inside of maybe, except it's implemented more efficiently because it skips all the exception-handling setup that maybe does. If the match fails you get undef back, or an empty list in list context. Obsessed with the efficiency tbh ✨

---

✨ substring_before ✨

$str = $parser->substring_before( $literal );

$str = $parser->substring_before( qr/pattern/ );

Since version 0.06.

So this one is like, maybe finds a literal string or regexp pattern match. If it finds one, it consumes all the input text before that match — not including the match itself — and hands it back to you. If it can't find a match before the end of the current scope, it consumes everything up to the end of scope and gives you that instead.

Important vibe check: it does not consume the part of the input that matches. Only the text before it. And it's not considered a failure if the substring before the match is empty. If you need a non-empty match, that's a job for fail, bestie:

sub token_nonempty_part
{
   my $self = shift;

   my $str = $parser->substring_before( "," );
   length $str or $self->fail( "Expected a string fragment before ," );

   return $str;
}

Also, and this is the part that's different from basically every other token parsing method, it does not consume leading or trailing whitespace around the substring. The whole point is you'd use this in a parser reading quoted strings, or similar situations where whitespace needs to be preserved. Periodt. 🎀

---

✨ nonempty_substring_before ✨

$str = $parser->nonempty_substring_before( $literal );

$str = $parser->nonempty_substring_before( qr/pattern/ );

Since version 0.20.

It's a variant of substring_before that fails if the matched part is empty. So like, the example from before could have just been written as:

sub token_nonempty_part
{
   my $self = shift;

   return $parser->nonempty_substring_before( "," );
}

Honestly this is so useful for breaking out of repeating loops, iykyk. Like:

sub token_escaped_string
{
   my $self = shift;
   $self->expect( '"' );

   my $ret = "";
   1 while $self->any_of(
     sub { $ret .= $self->nonempty_substring_before( qr/%|$/m ); 1 }
     sub { my $escape = ( $self->expect( qr/%(.)/ ) )[1];
           $ret .= _handle_escape( $escape );
           1 },
     sub { 0 },
   )

   return $ret;
}

---

✨ generic_token ✨

$val = $parser->generic_token( $name, $re, $convert );

Since version 0.08.

Expects to find a token matching the precompiled regexp $re. If you pass one, the $convert CODE reference can turn that string into something more convenient for you. And $name is what shows up in the failure message when the pattern doesn't match.

If you give it a $convert function, that function gets handed the parser and the matching substring, and whatever it returns is what comes back out of generic_token:

$convert->( $parser, $substr )

If you don't give it one, you just get the substring as it stands. Simple 💜

This method is mostly there so subclasses can define their own token types. Like, for example:

sub token_hex
{
   my $self = shift;
   $self->generic_token( hex => qr/[0-9A-F]{2}h/, sub { hex $_[1] } );
}

---

✨ token_int ✨

$int = $parser->token_int;

Expects to find an integer written in decimal, octal or hexadecimal notation, and consumes it. Negative integers with a - in front are recognised too. 💅

---

✨ token_float ✨

$float = $parser->token_float;

Since version 0.04.

Expects to find a number in floating-point notation: a sequence of digits, possibly prefixed by -, possibly containing a decimal point, possibly followed by an exponent written as e followed by an integer. Then it gives you the numerical value. Math girlie hours 🦋

---

✨ token_number ✨

$number = $parser->token_number;

Since version 0.09.

Expects to find a number in either of the two forms above. Slay.

---

✨ token_string ✨

$str = $parser->token_string;

Expects to find a quoted string, and consumes it. The string should be quoted with " or ' quote marks.

The content of the quoted string can contain character escapes similar to the ones C or Perl accept. Specifically, these forms are recognised:

\a               Bell ("alert")
\b               Backspace
\e               Escape
\f               Form feed
\n               Newline
\r               Return
\t               Horizontal Tab
\0, \012         Octal character
\x34, \x{5678}   Hexadecimal character

Now for the not-supported list, bestie, because this matters: C's \v for vertical tab is not supported, since it's rarely used in practice and it collides with Perl's \v regexp escape. Perl's \c for forming other control characters is also not supported. Okay? Noted 📝

---

✨ token_ident ✨

$ident = $parser->token_ident;

Expects to find an identifier, and consumes it. That's it, that's the method 💖

---

✨ token_kw ✨

$keyword = $parser->token_kw( @keywords );

Expects to find a keyword, and consumes it. A keyword is defined as an identifier which is exactly one of the literal values you passed in. So like, it's gotta match match, no fuzzy business 🌸

---

🌈 EXAMPLES 🌈

💖 Accumulating Results Using Variables 💖

So all the structure-forming methods return a value that comes from their nested parsing code, right? But sometimes it's just more convenient to accumulate your result in a variable instead. Main character energy, honestly.

Like, consider this parser method — it's designed to parse a set of name: "value" assignments, the kind you'd find in a config file or a YAML/JSON-style mapping value:

sub parse_dict
{
   my $self = shift;

   my %ret;
   $self->list_of( ",", sub {
      my $key = $self->token_ident;
      exists $ret{$key} and $self->fail( "Already have a mapping for '$key'" );

      $self->expect( ":" );

      $ret{$key} = $self->parse_value;
   } );

   return \%ret
}

See how instead of using the return value from list_of, this method accumulates everything in the %ret hash and eventually returns a reference to it as its result? And because of that, it can do some error checking while it parses — namely, rejecting duplicate keys. Iconic behaviour tbh 👑

---

💀 TODO 💀

• Make unescaping of string constants more customisable. Possibly consider instead a parse_string_generic using a loop over substring_before.
• Easy ability for subclasses to define more token types as methods. Perhaps provide a class method such as

__PACKAGE__->has_token( hex => qr/[0-9A-F]+/i, sub { hex $_[1] } );

• Investigate how well from_reader can cope with buffer splitting across other tokens than simply skippable whitespace

---

🫶 AUTHOR 🫶

Paul Evans <leonerd@leonerd.org.uk>

---

And that's the whole page, bestie, we made it to the end 🎉💖 this was perl v5.40.1, dated 2025-09-28, and it's the Parser::MGC(3pm) manpage. Go parse something iconic with it. Periodt ✨👑🌸