Marpa:...ognizer(3pm)

📦

version 2.086000 · 4 versions

💖 NAME 💖

okay so this is Marpa:...ognizer(3pm) — like, the User Contributed Perl Documentation for Marpa::R2::NAIF::Recognizer — and it's all about NAIF recognizers. NAIF = Named Argument InterFace, iykyk 🎀

✨ SYNOPSIS ✨

bestie. look at this little guy go 🦋

my $recce = Marpa::R2::Recognizer->new( { grammar => $grammar } );
$recce->read( 'Number', 42 );
$recce->read('Multiply');
$recce->read( 'Number', 1 );
$recce->read('Add');
$recce->read( 'Number', 7 );

🌸 DESCRIPTION 🌸

So this doc is about recognizers for Marpa's named argument interface (that's the NAIF, bestie). BUT — and this is important — if you're a beginner, or you're not sure which interface you even want, or you literally do not know what the NAIF interface is, then you are probably looking for the document on recognizers for the SLIF interface. Go find that one. Go go go 🏃‍♀️💨

To make a recognizer object → use the new method.

To read input → use the read method.

To evaluate a parse tree, based on the input → use the value method. periodt 💅

🎀 Token streams 🎀

By default, Marpa uses the token-stream model of input. And the token-stream model is standard — like, so standard that most documents about parsing don't even bother describing it 💀. In the token-stream model, each read adds a token at the current location, then advances the current location by one. The location before any input is numbered 0, and if N tokens are parsed, they fill the locations from 1 to N. slay.

This document describes only the token-stream model of input. Marpa does allow other models of the input, but using them requires special method calls, and those are described in the document on alternative input models. (not here, bestie 🙈)

🧚 Constructor 🧚

`new()`

my $recce = Marpa::R2::Recognizer->new( { grammar => $grammar } );

The new method creates a recognizer object. It either returns you a brand new recognizer object 🌟 or it throws an exception. no in-between, it's giving commitment.

The arguments to new are references to hashes of named arguments. In each key/value pair of these hashes: the key is the argument name, and the hash value is the value of the argument. The named arguments are described below 👇

✨ Accessors ✨

`check_terminal()`

my $is_symbol_a_terminal = $recce->check_terminal('Document');

Returns a Perl true when its argument is the name of a terminal symbol. Otherwise returns a Perl false. Not often needed, ngl. 💅

`events()`

my @expected_symbols =
    map { $_->[1]; }
    grep { $_->[0] eq 'SYMBOL_EXPECTED' } @{ $recce->events() };

Returns a reference to an array of the events from the last read() method call. Each element of the array is a subarray of 1 or 2 elements. The first element of the subarray is the name of an event type, as described in "Recognizer events". The second element is the event value of the event, where that is applicable. For more detail, see "Recognizer events". 🫶

`exhausted()`

$recce->exhausted() and die 'Recognizer exhausted';

The exhausted method returns a Perl true if parsing in a recognizer is exhausted, and a Perl false otherwise. Parsing is exhausted when the recognizer will not accept any further input.

By default, a recognizer event occurs if parsing is exhausted. And an attempt to read input into an exhausted parser causes an exception to be thrown 💀. The recognizer event and the exception are all that many applications require, but this method lets you discover the recognizer's exhaustion status directly. main character energy, honestly.

`latest_earley_set()`

my $latest_earley_set = $recce->latest_earley_set();

Returns the location of the latest (in other words, the most recent) Earley set. In the places where it's most often needed, the latest Earley set is the default, so there's usually no need to request the explicit value of the latest Earley set. iconic behavior ✨

`progress()`

Given the location (Earley set ID) as its argument, this returns an array that describes the parse progress at that location. Details on progress reports live in their own document, bestie — go peek 👀

`terminals_expected()`

my $terminals_expected = $recce->terminals_expected();

Returns a reference to a list of strings, where the strings are the names of the terminals acceptable at the current location. In the default input model, the presence of a terminal in this list means that terminal will be acceptable in the next read method call. This is highly useful for Ruby Slippers parsing 👑💖

💅 Mutators 💅

`expected_symbol_event_set()`

$recce->expected_symbol_event_set( 'endmark', 1 );

Marpa can generate a recognizer event when a symbol is expected at the current earleme. This method takes a symbol name as its first argument, and turns the expected-symbol event for that symbol on or off, according to whether its second argument is 1 or 0. Always succeeds or throws an exception — no middle ground, bestie.

⚠️ okay so listen: events can occur at location 0 — that's when the recognizer is first created. BUT the event setting of expected_symbol_event_set() cannot have an effect until after the first token is read — like, after location 0. In cases where this is an issue, the event_if_expected named argument of the new() method can be used to set an expected-symbol event. don't say I didn't warn you 🙈

`read()`

$recce->read( 'Number', 42 );
$recce->read('Multiply');
$recce->read( 'Number', 1 );
$recce->read('Add');
$recce->read( 'Number', 7 );

The read method reads one token at the current parse location. It then advances the current location by 1. so efficient, we love to see it 🦋

read takes two arguments: a token name and a token value.

The token name is required. It must be the name of a valid terminal symbol.
The token value is optional. It defaults to a Perl undef.

For details about terminal symbols, see "Terminals" in Marpa::R2::NAIF::Grammar.

The parser may accept or reject the token. If the parser accepted the token, the read method returns the number of recognizer events that occurred during the read. For more about events, see "Recognizer events".

Marpa may reject a token because it is not one of those acceptable at the current location. When that happens, read returns a Perl undef. And here's the thing bestie — a rejected token need not end parsing! It is perfectly possible to retry the read call with another token. This is, in fact, an important technique in Ruby Slippers parsing. For details, see the section on Ruby Slippers parsing. ✨

For other failures — including an attempt to read a token into an exhausted parser — Marpa throws an exception. 💀

`set()`

$recce->set( { max_parses => 10, } );

The set method's arguments are references to hashes of named arguments. You can use set to set or change named arguments after the recognizer has been created. Details of the named arguments are below 👇

`value()`

my $value_ref = $recce->value;
my $value = $value_ref ? ${$value_ref} : 'No Parse';

Because Marpa parses ambiguous grammars, every parse is a series of zero or more parse trees. There are zero parse trees if there was no valid parse of the input according to the grammar. (sad girl hours 😭)

The value method call evaluates the next parse tree in the parse series, and returns a reference to the parse result for that parse tree. If there are no more parse trees, the value method returns undef. that's it, that's the whole vibe.

`reset_evaluation()`

$recce->reset_evaluation();
$recce->set( { end => $loc, max_parses => 999, } );

The reset_evaluation() method ends a parse series, and starts another. It can be used to "restart" the parse series 🎀

Restarting the parse series with reset_evaluation() lets the application specify new values for the closures, end and ranking_method named arguments. Once a parse series is underway, these values cannot be changed — so restart if you wanna switch it up.

The most common use for the reset_evaluation() method is to parse a single input stream at different end points. You could also do this by creating a new recognizer and re-reading the input from the beginning, but it is much more efficient to evaluate a single recognizer run several times, using different parse end locations 👑. After the parse is restarted using reset_evaluation(), the recognizer's set() method and its end named argument can be used to change the parse end location. ate. 💅

🌈 Trace accessors 🌈

`show_progress()`

print $recce->show_progress()
    or die "print failed: $ERRNO";

Returns a string describing the progress of the parse. Here's the tea on the arguments:

With no arguments, the string contains reports for the current location.
With a single integer argument N, the string contains reports for location N.
With two numeric arguments, N and M, the arguments are interpreted as a range of locations, and the returned string contains reports for all locations in the range.

("Location" as referred to in this section, and elsewhere in this document, is what is also called the Earley set ID. keeping the lore consistent ✨)

If an argument is negative, -N, it indicates the Nth location counting backward from the furthest location of the parse. For example, if 42 was the furthest location, -1 would be location 42 and -2 would be location 41. For example, the method call $recce->show_progress(-3, -1) returns reports for the last three locations of the parse. And the method call $recce->show_progress(0, -1) will print progress reports for the entire parse. 🦋

show_progress is Marpa's most powerful tool for debugging application grammars. It can also be used to track the progress of a parse or to investigate how a parse works. A much fuller description, with an example, is in the document on debugging Marpa grammars. go read it, bestie, it's giving essential 📖💖

✨ Named arguments ✨

So the recognizer's named args — you can hand them over through its new or its set method, whichever vibe you're on. 💖

🎀 closures

The value of closures has to be a reference to a hash, bestie. In every key/value pair of that hash, the key has to be an action name, and the value has to be a CODE ref. And heads up: closures is not allowed once evaluation has begun. 🙅

When an action name is a key in closures, the usual action resolution mechanism of the semantics gets completely bypassed — like, skipped, ignored, not invited. One super common use of closures is letting anonymous subroutines act as semantic actions. For the full tea, go read the document on semantics. 📖

🎀 end

end sets the parse end location. Default behaviour? The parse ends wherever the input ended, so what you get back is a parse of the entire input. And nope — end is not allowed once evaluation has begun.

Oh and quick vocab moment: "Location" here and everywhere else in this doc is the thing also called an Earley set ID. Slay, terminology. ✨

🎀 event_if_expected

The value of event_if_expected must be a reference to an array of symbol names. Expected-symbol events get turned on for those symbol names. You can flip them off (or back on) later with the expected_symbol_event_set() method.

The glow-up of using event_if_expected instead of that method? It takes effect the instant the recognizer is created, whereas events set with expected_symbol_event_set() can't happen until after the first token is read. Timing is everything, fr fr. ⏰

🎀 grammar

The new method is required to have a grammar named argument, and its value must be a precomputed Marpa grammar object. Also, grammar is not allowed anywhere else. Nowhere. Not even a little. 👑

🎀 max_parses

If it's non-zero, it causes a fatal error when that number of parse results is exceeded. max_parses is your bestie for limiting CPU usage and output length while testing and debugging. Stable, production apps might prefer to just count the parses themselves and take a less Draconian response when the count goes over. 💀

The value must be an integer. If it's zero, there's no limit on the number of parse results returned. Default? No limit. 🦋

🎀 ranking_method

The value must be a string, and it's one of "none", "rule", or "high_rule_only". When it's "none", Marpa returns the parse results in arbitrary order — and that's the default. ranking_method is not allowed once evaluation has begun.

The "rule" and "high_rule_only" ranking methods let you control the order parse results come back in from the value method, and even exclude some parse results from the parse series entirely. Details live in the document on parse order. 📚

🎀 too_many_earley_items

too_many_earley_items is optional. If you specify it, it sets the Earley item warning threshold. If an Earley set grows bigger than that threshold, a recognizer event is generated and a warning gets printed to the trace file handle. 📢

Marpa parses from any BNF, so it can absolutely handle grammars and inputs that produce large Earley sets. But — plot twist — parsing with large Earley sets can be slow. Large Earley sets are something most applications can avoid and will want to avoid.

By default, Marpa calculates an Earley item warning threshold based on the size of the grammar. That default threshold will never be less than 100. And if you set the threshold to 0, no recognizer event is generated and warnings about large Earley sets are turned off completely. Silence. 🤫

🎀 trace_actions

trace_actions is a boolean. If it's true, Marpa prints tracing information as it resolves action names to Perl closures. False turns tracing off — and false is the default. Traces get written to the trace file handle. 🕵️

🎀 trace_file_handle

The value is a file handle. Traces and warning messages go to the trace file handle. By default, the trace file handle is inherited from the grammar used to create the recognizer. 🧬

🎀 trace_terminals

Very handy for debugging, and honestly often useful even when the problem isn't in the lexing at all. The value is a trace level. Trace level 0 means terminal tracing is off, and that's the default. ✨

At trace level 1 or higher, Marpa produces a trace message for each terminal as it's accepted or rejected by the recognizer. At trace level 2 or higher, the trace messages include, for every location, a list of the terminals expected. Warning though: in practical grammars, output from trace level 2 can be voluminous. Like, unhinged amounts of text. 😭

🎀 trace_values

trace_values is a numeric trace level. If it's 1, Marpa prints tracing information as values are computed in the evaluation stack. Trace level 0 turns value tracing off, which is the default. Traces get written to the trace file handle. 🫶

🎀 warnings

The value is a boolean. Warnings are written to the trace file handle. By default, the recognizer's warnings are on. Usually your app will want to leave them on, so like... don't touch. 💅

🌈 Recognizer events 🌈

The recognizer's read() method can generate events. To get at those events, use the recognizer's events() method. 🎀

The EARLEY_ITEM_THRESHOLD and The EXHAUSTED events are enabled by default. Events optionally have an "event value", as specified in the description of each event. Here's everything that can pop off:

⭐ EARLEY_ITEM_THRESHOLD

The Earley item threshold was exceeded. Want more on the Earley item warning threshold? See too_many_earley_items. No event value is defined for this event. This event is enabled by default. 🔥

⭐ EXHAUSTED

"Exhaustion" means the next read call must fail, because there's no token that will be acceptable to it. More details on "exhaustion" are in a section below. No event value is defined for this event. This event is enabled by default. 😮

⭐ SYMBOL_EXPECTED

A "symbol expected" event means a symbol is expected at that point. The event value of this event is the symbol whose expectation caused the event. This event is disabled by default. For details, see expected_symbol_event_set(). 👀

💜 Parse exhaustion 💜

A parse is exhausted when it will accept no more input. But plot twist: an exhausted parse is not necessarily a failed parse. Grammars are often written so that once they "find what they are looking for", no further input is acceptable. Grammars of that kind become exhausted when they succeed. 🌸

By default, a recognizer event occurs whenever the parse is exhausted. An application can also check for exhaustion explicitly, using the recognizer's exhausted method. 🦋

👑 Ruby Slippers parsing 👑

$recce = Marpa::R2::Recognizer->new( { grammar => $grammar } );

my @tokens = (
    [ 'Number', 42 ],
    ['Multiply'], [ 'Number', 1 ],
    ['Add'],      [ 'Number', 7 ],
);

TOKEN: for ( my $token_ix = 0; $token_ix <= $#tokens; $token_ix++ ) {
    defined $recce->read( @{ $tokens[$token_ix] } )
        or fix_things( $recce, $token_ix, \@tokens )
        or die q{Don't know how to fix things};
}

Marpa can tell the application which symbols are acceptable as tokens at the next location in the parse. The terminals_expected method returns the list of tokens that will be accepted by the next read. Your application can use that info to change the input "on the fly" so it becomes acceptable to the parser. ✨

You can also take the "try it and see" approach. If your app isn't sure whether a token is acceptable, just try to read the dubious token with the read method. If the token gets rejected, the read method call returns a Perl undef. At that point, you can retry the read with a different token. Easy. 💅

🎀 An example

Marpa's HTML parser, Marpa::HTML, is an example of how Ruby Slippers parsing can help with a non-trivial, real-life application. When a token is rejected in Marpa::HTML, it changes the input to match the parser's expectations by

Modifying existing tokens, and
Creating new tokens.

That second technique — creating new "virtual" tokens — is what Marpa::HTML uses to deal with omitted start and end tags. The actual HTML grammar that Marpa::HTML uses takes an oversimplified view of HTML: it assumes that start and end tags are always present, even when the HTML standards don't require it. And for most HTML files of interest, that assumption will be contrary to fact. 🙈

Ruby Slippers parsing is used to make the grammar's over-simplistic view of the world come true for it. Whenever a token is rejected, Marpa::HTML looks at the expected tokens list. If it sees that a start or end tag is expected, Marpa::HTML creates a token for it — a completely new "virtual" token that gives the parser exactly what it expects. Then Marpa::HTML resumes input at the point in the original input stream where it left off. 🧚

🎀 Copyright and License 🎀

Copyright 2014 Jeffrey Kegler
This file is part of Marpa::R2.  Marpa::R2 is free software: you can
redistribute it and/or modify it under the terms of the GNU Lesser
General Public License as published by the Free Software Foundation,
either version 3 of the License, or (at your option) any later version.

Marpa::R2 is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser
General Public License along with Marpa::R2.  If not, see
http://www.gnu.org/licenses/.

And that's a wrap on Marpa:...ognizer(3pm), bestie — this one shipped as part of perl v5.34.0 and was dated 2022-02-06. Go forth and parse iconically. 💖✨👑

📦 where this page comes from

ubuntu jammy: libmarpa-r2-perl 2.086000~dfsg-7build1 · LGPL-3+~Marpa
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-14: a language model's rewrite of the original, under its terms