Parser...utorial(3pm)

✨ NAME ✨

Parser::MGC::Tutorial — the tutorial for building simple recursive-descent parsers, no cap 🦋

---

💖 INTRODUCTION 💖

okay so bestie, here's the whole vibe: Parser::MGC is an abstract base class that hands you a bunch of genuinely useful features to help you write a parser. It's giving scaffolding, but make it glam. 💅

A parser you write with this module is a subclass of Parser::MGC, and that subclass gets a parse method for free. Like, for example, this lil baby parser here only accepts input that matches the given regexp:

package ExampleParser;
use base qw( Parser::MGC );

sub parse
{
   my $self = shift;

   $self->expect( qr/hello world/i );

   return 1;
}

Then a program using this parser makes an instance of it and uses that instance to actually parse stuff. You feed content in either as a string value to the from_string method, or from a named file or filehandle to the from_file method. 🎀

use feature qw( say );
use ExampleParser;

my $parser = ExampleParser->new;

say $parser->from_string( "Hello World" );

When you run it, the program outputs the return value of the from_string method — which is itself whatever the parse method returned. cute, right?

$ perl tut01.pl
1

And you can also hand over the content with from_file instead:

use feature qw( say );
use ExampleParser;

my $parser = ExampleParser->new;

say $parser->from_file( \*STDIN );

🍓 Returning Values 🍓

ngl, a parser that just returns 1 to say "yeah it matched" is not exactly the most useful thing in the world. Almost every time you're using a parser to match some input, it's because you actually want values or structure out of that input, and you want those handed back to whoever called you. main character data, periodt. 👑

The expect method tries to match the next part of the input string against the given regexp, and it returns the matching substring. A follow-up expect call keeps parsing right where the previous one left off. And Parser::MGC automatically skips over whitespace between these calls, so most grammars that don't care about whitespace shouldn't have to think about it explicitly at all. ✨

By using two expect calls, a parser can first check that some literal is present, and then return a matched substring from the input:

sub parse
{
   my $self = shift;

   $self->expect( qr/hello/i );
   my $name = $self->expect( qr/\w+/ );

   return $name;
}

say $parser->from_string( "Hello World" );

$ perl tut02.pl
World

Token methods make this even easier, because they're convenient shortcuts to commonly-used matching patterns. 🫶

sub parse
{
   my $self = shift;

   $self->expect( qr/hello/i );
   return $self->token_ident;
}

But if we pass in a value that does not match the regexp? An exception gets thrown, and it comes with details of how the parse failed. the drama 💀

say $parser->from_string( "Hello, world!" );

$ perl tut03.pl
Expected (?i-xsm:hello world) on line 1 at:
Hello, world!
^

A super typical use of a parser is to form an Abstract Syntax Tree (AST) out of some given input. In that case the return value from the parse method is probably gonna be some object the application can use to inspect the syntax. 🌸

sub parse
{
   my $self = shift;

   my $num = $self->token_number;
   return MyGrammar::Expression::Number->new( $num );
}

🙈 Indicating Failure 🙈

So like, the basic methods such as expect and the various token methods will flag a failure automatically, no stress. But sometimes your grammar needs more logic than that from the parser. And if that logic wants to indicate a failure in the input and cause back-tracking to happen, it can use the fail method. 🌈

sub parse
{
   my $self = shift;

   my $num = $self->token_number;
   $num >= 0 or $self->fail( "Expected a non-negative number" );

   return $num;
}

---

🔥 STRUCTURE 🔥

So far we've only managed to parse simple patterns that honestly could've just been a plain regular expression. But any parser for a nontrivial grammar is gonna need other abilities too — it needs to choose from a list of alternatives, repeat patterns, and form nested scopes so it can match other content inside them. iconic behavior only. 💅

Parser::MGC gives you a set of methods that take one or more CODE references (each of which performs some parsing step) and form a higher-level construction out of them. You use those to build more complex parsers out of simple ones. It is exactly this recursive structure that gives Parser::MGC its main power over simple one-shot regexp matching. 🔥

Any nontrivial grammar is likely to be formed from multiple named rules. So it's only natural to split the parser for such a grammar into methods whose names reflect the structure of the grammar you're parsing. Every one of the structure-forming methods that takes CODE references invokes each of them by passing in the parser object itself as the first argument. Which makes it simple to invoke sub-rules by passing references to method subs themselves, because the parser object will already be passed as the invocant. 🎀

The following examples all build together into a parser for a simple C-like expression language. let's gooo 🦋

⭐ Optional Rules ⭐

The simplest of the structure-forming methods is maybe. It tries to run the parser step you gave it, and if that succeeds, it returns the value that step returned. If it fails by throwing an exception, then the maybe call just returns undef and resets the current parse position back to where it was before it started. That's how you write a grammar with an optional element — similar to the ? quantifier in a regular expression. 😌

sub parse_type
{
   my $self = shift;

   my $storage = $self->maybe( sub {
      $self->token_kw(qw( static auto typedef ));
   } );

   return MyGrammar::Type->new( $self->parse_ident, $storage );
}

💜 Repeated Rules 💜

Next up: sequence_of. It tries to run the parser step you gave it multiple times until it fails, and returns an ARRAY reference collecting up all the return values from each iteration that succeeded. And fun fact — by itself, sequence_of can never fail. If the body never matches at all, it just yields an empty array and consumes nothing from the input. That's how you write a grammar with a repeating element, similar to the * quantifier in a regular expression. ✨

sub parse_statements
{
   my $self = shift;

   my $statements = $self->sequence_of( sub {
      $self->parse_statement;
   } );

   return MyGrammar::Statements->new( $statements );
}

A lot of the time though, the grammar needs at least one item to be present, and it should not accept an empty parse of zero elements. You can do that in code by testing the size of the returned array and using the fail method. Think of it like the + quantifier in a regular expression. 🫶

sub parse_statements
{
   my $self = shift;

   my $statements = $self->sequence_of( sub {
      $self->parse_statement;
   } );

   @$statements > 0 or $self->fail( "Expected at least one statement" );

   return MyGrammar::Statements->new( $statements );
}

Another thing that happens constantly: the grammar needs some simple separation pattern between each parsed item, like a comma. The list_of method is your bestie here, because it automatically handles those separating patterns between the items and returns a reference to an array containing only the actual parsed items — no separators invited. 🍓

sub parse_expression_list
{
   my $self = shift;

   my $exprs = $self->list_of( ",", sub {
      $self->parse_expression;
   } );

   return MyGrammar::ExpressionList->new( $exprs );
}

🌸 Alternate Rules 🌸

To handle a choice between multiple different alternatives in the grammar, any_of takes an ordered list of parser steps and tries to invoke each one in turn. It yields, as its result, the result of the first one of these that didn't fail. That's how you write a grammar that allows a choice of multiple different rules at some point — similar to the | alternation in a regular expression. 💅

sub parse_statement
{
   my $self = shift;

   $self->any_of(
      sub { $self->parse_declaration },
      sub { $self->parse_expression; $self->expect( ';' ); },
      sub { $self->parse_block_statement },
   );
}

🧚 Scoping Rules 🧚

The final structure-forming method has no direct analogy to a regular expression, though usually you can find similar structures out there. For the case where some nested structure has to be handled between an opening and a closing marker, you use the scope_of method. It takes three arguments: the opening marker, a parser step to handle the contents of the body, and the closing marker. It expects to find each of these in sequence, and it returns the value that the inner parsing step returned. 🌟

But here's what makes it genuinely interesting — during execution of the inner parsing step, the basic token functions all take the closing marker into account. No token function will return a result if the stream now looks like the scope closing marker. Instead, they'll all fail, claiming to be at the end of the scope. Which makes it so much simpler to parse things like lists of values surrounded by braces. 👑

sub parse_array_initialiser
{
   my $self = shift;

   $self->scope_of( "{", sub { $self->parse_expression_list }, "}" );
}

During execution of that inner call to parse_expression_list, any occurrence in the stream of the "}" marker will appear to be the end of the stream — which causes the inner call to stop at (hopefully) exactly the right place, barring other syntax errors, and to terminate correctly. 💖

---

💌 AUTHOR 💌

Paul Evans <leonerd@leonerd.org.uk>

---

okay that's the whole thing bestie, slay 💅✨ this was Parser::MGC::Tutorial, manual page Parser...utorial(3pm), from perl v5.40.1, dated 2025-09-28 — go build something iconic with it 🎀🦋💖