Marpa:...ess::R(3pm)

📦

version 12.000000 · 6 versions

12.000000 · ubuntu stonking (latest)
2.086000 · ubuntu jammy
2.086000 · ubuntu noble
2.086000 · debian bookworm
12.000000 · ubuntu resolute
2.086000 · debian trixie

💖 Marpa::R2::Scanless::R 💖

ok so bestie, this is the Marpa:...ess::R(3pm) page — User Contributed Perl Documentation. It's giving "the girl who knows exactly what a recognizer is and wants YOU to know too". Let's go 🦋

✨ Name ✨

Marpa::R2::Scanless::R — Scanless interface recognizers 💅

Like, they recognize stuff. Idk, they're iconic, that's the point.

🌸 Synopsis 🌸

lowkey just copy this vibe:

my $recce = Marpa::R2::Scanless::R->new( { grammar => $grammar } );
my $self = bless { grammar => $grammar }, 'My_Actions';
$self->{recce} = $recce;

if ( not defined eval { $recce->read($p_input_string); 1 }
    )
{
    ## Add last expression found, and rethrow
    my $eval_error = $EVAL_ERROR;
    chomp $eval_error;
    die $self->show_last_expression(), "\n", $eval_error, "\n";
} ## end if ( not defined eval { $event_count = $recce->read...})

my $value_ref = $recce->value( $self );
if ( not defined $value_ref ) {
    die $self->show_last_expression(), "\n",
        "No parse was found, after reading the entire input\n";
}

package My_Actions;
sub do_parens    { shift; return $_[1] }
sub do_add       { shift; return $_[0] + $_[2] }
sub do_subtract  { shift; return $_[0] - $_[2] }
sub do_multiply  { shift; return $_[0] * $_[2] }
sub do_divide    { shift; return $_[0] / $_[2] }
sub do_pow       { shift; return $_[0]**$_[2] }
sub do_first_arg { shift; return shift; }
sub do_script    { shift; return join q{ }, @_ }

periodt. 💫

🎀 About this document 🎀

okay so this page is THE reference doc for the recognizer objects of Marpa's SLIF (that's the Scanless interface, bestie, keep up 🫶).

It's called "Scanless" because it does not require the application to supply a scanner (lexer). The SLIF contains its own lexer, and its use is integrated into its syntax. In this document, using the SLIF's internal scanner is called internal scanning. ✨

But also — the SLIF lets applications do their own scanning if they find that useful. When an application bypasses the SLIF's internal scanner and does its own scanning, this document calls it external scanning. An application can use external scanning to supplement internal scanning, or to replace the SLIF's internal scanner entirely. Both valid, both slay. 👑

🌈 The input stream 🌈

The recognizer reads a virtual input stream. By default, this is identical to a physical input stream. The physical input stream is a Perl string passed as the first argument to the $recce->read() method. Once it's set by the read() method, the physical input stream cannot be changed. Not negotiable, no take-backs, no editing the group chat after sending. 😭

Physical input stream location is just the Perl pos() location in the physical input string. Physical input stream location may be zero, but it is never negative.

In this document, the phrase "input stream" and the word "stream", unless otherwise specified, refer to the physical input stream. The phrase "input stream location" and the word "location", unless otherwise specified, refer to physical input stream location.

Now — virtual input streams complicate the idea of parse location, but they are essential for some applications, fr fr. Implementing the C language's pre-processor directives requires either two passes, or a virtual approach to the input. And Perl here-documents cannot be parsed correctly by an application which insists on moving forward serially in the input. The SLIF allows applications to skip backward and forward in the physical input stream, and to read sections of the stream repeatedly. 🦋

Input streams are ordered sets of characters, and the locations in them are represented as the integers from 0 to N, where N+1 is the size of the set. In this document, we will refer to ordered subsets of contiguous locations as either ranges or spans.

🔥 Ranges 🔥

A range is an ordered set of contiguous locations specified by start location and end location: [S ... E]. A range is a subset of a "universe" — some larger ordered set of locations 0 to N. In this document the larger sets, or universes, will be either physical input streams or G1 location streams. 💅

The start and end locations of the range refer to locations in its universe. Negative locations refer to locations relative to the end of the range's universe, so that -1 refers to the last location of the universe, -2 refers to the second-to-last location of the universe, etc. Yes bestie, we're counting backwards, it's giving mirror dimension 🪞

🧚 Spans 🧚

A span is an ordered set of contiguous locations specified by start location and length: [S, L]. A span is a subset of a universe of locations, as was described above for ranges.

The range corresponding to the span [S, L] is [S ... (S+L)-1]. The span corresponding to the range [S ... E] is [S, (E-S)+1]. A span with a negative length is interpreted as if it was the range with that same pair of values.

In general, spans are more convenient for programming. But when fencepost issues are important, spans require a lot of mental arithmetic, and a discussion that uses ranges is easier to follow. The math is mathing either way, choose your fighter. ⚔️

As examples,

The entire input stream is the range [0 ... -1] and the span [0, -1].
The first 42 characters of the input stream are the range [0 ... 41] and the span [0, 42].
The entire input stream, except for the last character, is the range [0 ... -2] and the span [0, -2].
The substring consisting only of the last character is the range [-1 ... -1] and the span [-1, 1].
The substring which consists of the last 3 characters is the range [-3 ... -1] and the span [-3, 3].
The substring which consists of only the third-to-last character is the range [-3 ... -3] and the span [-3, 1].

no cap, those examples ate. 🍓

👑 Internal scanning 👑

The virtual input stream is a series of input strings. An input string is a substring of the physical input stream. By default the virtual input stream consists of exactly one input string, one which begins at location 0 in the physical input stream and whose length is the length of the physical input stream. Minimal effort, maximum results, main character energy ✨

The SLIF always starts scanning using the read() method, and the first input string is specified, implicitly or explicitly, by the read() method. When not specified, the input string for read() defaults to the range [0 ... -1].

read() will return success when it reaches the end of its input string, or when a SLIF parse event triggers. (Parse events are described in a separate document, so don't @ me.) In many cases there are no parse events declared, or none trigger. If no parse event triggers and the parse does not fail, then read() will read to the end of string.

The SLIF tracks a "current location" in the physical input stream. On return from the read() method, current location will depend on the reason for the return. If a SLIF parse event triggered, the current location will be the trigger location; otherwise the current location will be at the end of the input string. 📍

⚠️ heads up bestie: the read() method may only be called once for a recognizer. But internal scanning can be resumed with the resume() method.

The resume() method, as the name suggests, resumes the internal scanning with a new input string. This input string must always be a substring of the physical input stream that was specified to the read() method. By default, the new input string runs from the current location to the end of the physical input stream.

On successful return from the resume() method, the current location is set in the same way as it for the read() method: the trigger location, if an event triggered; otherwise, the end of string. The resume() method may be called repeatedly, until the application considers the virtual input stream complete. More details are in the reference descriptions of the read() and resume() methods, below.

When the application considers input complete, and is ready to produce a parse value, the $recce->value() method is used. In most cases, this is all that is needed. But Marpa also allows repeated passes over the same input with different settings. More details on the semantics are provided in a separate document. 💜

💀 External scanning 💀

External scanning is usually performed by reading lexemes using the $recce->lexeme_read() method, which allows the reading of unambiguous lexemes. If ambiguous lexemes are needed, then the $recce->lexeme_alternative() and $recce->lexeme_complete() methods can be used. Pick your method, they're all valid. 🙈

Scanning must always begin with a call to the read() method, so that, in a pedantic sense, scanning always begins with internal scanning. But the first input string may be zero length:

$recce->read( \$string, 0, 0 );

and there is no requirement that internal scanning ever be resumed. Zero length input string? Iconic. Doing the bare minimum and thriving. 💅

🦋 External lexemes and the input stream 🦋

For error message and other purposes, even externally scanned lexemes are required to correspond to a span of the input stream. An external scanner must set up a relationship to the input stream, even if that relationship is completely artificial. Fake it till you make it, bestie. 😌

Here is one very general way to deal with external lexemes which have no natural mapping into the physical input stream. We will call what would ordinarily be the input string, the "natural input". To form the physical input stream, we append these 7 characters: "NO TEXT". For example, if the natural input is "Hi! I am the real input", then the physical input stream will be

"Hi! I am the real inputNO TEXT"

To read the natural input, we will use an initial call to the read() method of the form $recce->read($input_string, -8). If we want to read a lexeme which has no real relationship to the natural input, we can read it externally, using a method call similar to $recce->lexeme_read($symbol_name, -7, -1, $value).

The above approach allows the application, essentially, to ignore the natural input. External scanning also allows a wide variety of alternative input models. Alternative input models are an advanced topic and are discussed in a separate document. 🧚

🌸 okay pausing here because this page is LONG and we're only on part 1 of 7. more to come, don't go anywhere 💖

💫 G1 locations 💫

okay so bestie, heads up — the SLIF doesn't only keep track of where you are in the input stream. It also tracks G1 location, which is a whole other thing. 👀

G1 locations run from 0 to N, where N+1 is the length of the input stream. And the conventions and notation for numbering G1 locations, and for describing G1 spans and ranges, are the same as for input stream locations. (so if you got comfy with those, you're already set 🫶)

ngl, you can literally ignore G1 location like 95% of the time. It only becomes relevant when you're tracing the G1 grammar, and when you're dealing with ambiguous terminals. (and for the girlies familiar with Marpa's internals: the G1 location is the G1 Earley set index. periodt.)

Now here's the twist 💅 — because lexemes may be ambiguous, more than one lexeme may be read at a single G1 location. So we think of the lexemes read at a single G1 location as a set — call it the G1 lexeme set, or, for brevity, the G1 set. If a lexeme is unambiguous, its G1 set will contain exactly one lexeme. slay.

G1 location can be thought of as location in terms of boundaries of G1 sets, so that the first G1 set starts at G1 location 0 and ends at G1 location 1. When we speak of a G1 set at G1 location L, we mean the G1 set ending at G1 location L. Which means... there is no G1 set at G1 location 0. 🙈

As each G1 set is read, G1 location increases by one. And G1 length is length calculated in terms of G1 locations. For example, if a span of G1 locations begins at G1 location 42 and has length 2, it will contain a pair of G1 locations: G1 location 42 and G1 location 43. math is mathing ✨

Sometimes it's convenient to think of a G1 location as corresponding to a single input stream location. When that's the case, what's meant is the location at the end of the physical input stream span: "$span_start+$span_length". iykyk.

🌸 Literals and G1 spans 🌸

So sometimes you want to find the literal substring of the physical input stream that corresponds to a span of G1 locations. And if your application reads the physical stream in sequence within the G1 span, Marpa "does what you mean". 🧚 For the more complicated cases, the exact rules are right here bestie, no gatekeeping:

Except for G1 location zero, every G1 location X corresponds to one or more characters in the physical input stream. Let [s(X) ... e(X)] be the physical input stream range that corresponds to G1 location X. And only two things are guaranteed about s(X) and e(X) as a function of X:

s(X) and e(X) are not defined when X is zero.
It will always be the case that s(X) <= e(X).

Now, mapping ranges of G1 locations to ranges of physical input stream locations comes with ✨complications✨:

the fencepost versus interval issue: physical input stream locations correspond to characters, but G1 locations are locations before and after characters. it's giving off-by-one energy 😭
Both kinds of locations are zero-based, BUT G1 location 0 does not correspond to a range in the physical input stream.
Scanning is allowed to skip backward and forward, so the mapping of G1 location to physical stream locations is not necessarily monotonic. For example, if X and Y are G1 locations such that X < Y, it's possible that s(X) > e(Y).
Repeated scanning of the same physical input stream locations is allowed, and so are overlaps. For example, if X and Y are G1 locations, it's possible that s(X) < s(Y) < e(X) < e(Y).
Even when there is a monotonic function from G1 location to physical input stream span, there will usually be gaps. For example, applications typically discard whitespace. Which means if W is the physical input stream location of a whitespace character, there will be no G1 location X such that s(X) <= W <= e(X).

To cope with all of this, here are the rules used when translating G1 locations into literal substrings of the physical input stream:

If [X ... Y] is a G1 range, and s(X) < e(Y), the literal will be the substring made of the characters in the physical input stream range [s(X) ... e(Y)].
If s(X) >= e(Y), the literal will be the empty string.

So for applications that read the physical input stream in lexical order, without skipping forward — the above rules work as expected. 💖 For other applications, the above may be "close enough". But some applications might want to use custom logic to reassemble the input from the physical input stream, and the literal() method can assist in that process. you got this bestie 🫶

👑 The life cycle of a recognizer 👑

okay so this is describing the life cycle of a recognizer that has only one parse series. Your recognizer has only one parse series unless it calls the series_restart() method. Use of multiple parse series is an advanced technique — the kind most applications will literally never need. Full details about parse series live in a separate document. 📄

🎀 The Initial Phase 🎀

The Initial Phase begins when the recognizer is created with the calls the new() method. It ends when the read() method is called. It will also end, of course, if the recognizer is destroyed — but most applications will want to continue into the next phase. Very little can happen in this phase. It is possible to change some recognizer settings using the set() method. lowkey chill phase fr fr

🦋 The Reading Phase 🦋

The Reading Phase of a recognizer begins when it calls the read() method. It ends when it first calls the value() method. (and yes, it will also end if the recognizer is destroyed, but most applications will want to continue into the next phase.) During this phase, it's possible to add other input strings to the virtual input, by calling the resume() method. main character phase tbh 🔥

💜 The Evaluation Phase 💜

The Evaluation Phase of a SLIF recognizer begins when it first calls the value() method, which returns the result of the first parse tree. If there were no parses, the value() method will return a Perl undef. 💀

The value() method may be called more than once during the Evaluation Phase. The second and later calls of the value() method will return the result of the next parse tree. When there are no more parse trees, the value() method will return a Perl undef.

⚠️ bestie listen — the resume() method should not be called during the Evaluation Phase. no cap.

🌈 For more details 🌈

So, in all of the above, we described the life cycle for recognizers which have only one parse series. And a recognizer will have only one parse series unless it calls the series_restart() method.

Using multiple parse series, an application can run the SLIF recognizer several times on the same virtual input stream. More detail about the recognizer's life cycle — including a full treatment of parse series — is in a separate document. go read it if you're feeling iconic ✨

✨ Recognizer settings ✨

Okay bestie, so the recognizer settings are literally just the named arguments that the recognizer setting-aware methods accept. And those methods are new(), set() and series_restart(). Lowkey important: not every one of those methods takes every setting. The deets are right below, one setting at a time. Slay. 💅

🎀 end

Most of y'all will never need this one, fr fr. The end setting says where the parse ends, and it does it as a G1 location. The default? The parse just ends wherever the input ended — so the parse you get back is the parse of the whole entire virtual input stream. Also, heads up: end is only allowed in the new() and series_restart() methods.

🌸 event_is_active

    $slr = Marpa::R2::Scanless::R->new(
        {   grammar           => $grammar,
            semantics_package => 'My_Actions',
            event_is_active   => { 'before c' => 1, 'after b' => 0 }
        }
    );

The event_is_active recognizer setting changes the activation setting of events, bestie. Its value should be a reference to a hash where every entry's key is an event name and its value is either 0 or 1. Value 1? That event named in the hash key gets activated when the recognizer starts. Value 0? That event named in the hash key is inactive when the recognizer starts. And the event_is_active setting is only allowed with the recognizer's new() method. Periodt.

The setting in that event_is_active hash overrides the activation setting in the grammar. It's in effect before events at earleme 0 are triggered, and before any of the input stream is read. The activate() method can also change an event's activation setting — but only for events that trigger after earleme 0. The thing is, events at earleme 0 trigger during the recognizer's new() method, so calls to the activate() method literally cannot touch them. 😭

So if an event is initialized to inactive in the grammar, the event_is_active recognizer setting is the only way for a recognizer to let that event be active at earleme 0. Same energy in reverse: if an event is initialized to active in the grammar, event_is_active is the only way for a recognizer to set that event to be inactive at earleme 0. iykyk 🦋

💀 exhaustion

The exhaustion recognizer setting decides what goes down when asynchronous parse exhaustion happens. Intuitively, "asynchronous" parse exhaustion is parse exhaustion at a point when control would not normally return to the application. The exhaustion setting is allowed in any call of any of the recognizer setting-aware methods. For the full breakdown, go look at the description of exhaustion parse events.

The value of exhaustion has to be either "fatal" or "event". "fatal" is the default. If it's "fatal", asynchronous parse exhaustion gets treated as an error and an exception is thrown. If it's "event", then an event happens, exactly as described in the section on exhaustion parse events. ✨

👑

The value of the grammar setting must be a SLIF grammar object. The new() method is required to have a grammar setting. And grammar is only allowed by the new() method. Once the recognizer is created, the grammar cannot be changed. Not even a little. No takebacks. 🙈

🔥 max_parses

If it's non-zero, it causes a fatal error when that number of parse results gets exceeded. max_parses is useful for limiting CPU usage and output length when you're testing and debugging. That said — stable and production applications might prefer to just count the number of parses themselves and take a less Draconian response when the count goes over. 😅

The value must be an integer. If it's zero, there's no limit on the number of parse results returned. The default is no limit at all. The max_parses setting is valid in all calls of the recognizer setting-aware methods.

💜 ranking_method

The ranking_method is only allowed in calls of the new() method. The value must be a string, and it's one of "none", "rule", or "high_rule_only". When the value is "none", Marpa returns the parse results in arbitrary order. That's the default, btw.

The "rule" and "high_rule_only" ranking methods let you control the order in which parse results come back from the value method, and also let you exclude some parse results from the parse series. For the details, check the document on parse order. 📚

🍓 rejection

The rejection recognizer setting determines what happens when all tokens are rejected by the G1 parser. It's allowed in any call of any of the recognizer setting-aware methods. The value must be either "fatal" or "event". "fatal" is the default.

If the value is "fatal", rejection of all tokens is treated as an error and an exception is thrown. If the value is "event", an event occurs, as described in the section on rejection parse events.

🌈 semantics_package

Sets the semantic package for the recognizer. This setting takes precedence over any package implied by the blessing of the per-parse arguments to the SLIF recognizer's value() method. The semantics_package recognizer setting is what gets used when resolving action names to fully qualified Perl names. For more details on SLIF semantics, see the document on SLIF semantics.

The semantics_package setting is only allowed when a new parse series is begun — meaning it's only allowed in calls of the new() and series_restart() methods. And queen, please don't confuse the semantics_package recognizer setting with the SLIF's bless_package grammar setting. The two are genuinely not closely related. Two different things living two different lives. 💅

🌸 too_many_earley_items

The too_many_earley_items setting is optional, and like, very few applications will ever need it. If you specify it, it sets the Earley item warning threshold to a value other than its default. And if an Earley set gets bigger than the Earley item warning threshold, a recognizer event is generated, and a warning gets printed to the trace file handle. 👀

Marpa parses from any BNF, and it can handle grammars and inputs that produce very large Earley sets. But bestie, parsing that involves very large Earley sets can be slow.

By default, Marpa calculates an Earley item warning threshold for the G1 recognizer based on the size of the G1 grammar, and for each L0 recognizer based on the size of the L0 grammar. Those default thresholds will never be less than 100. The default is the result of considerable experience, and almost all users will be happy with it. Trust. 🫶

If the Earley item warning threshold gets changed from its default, the change applies to both L0 and G1 — currently there is no way to set them separately. And if the Earley item warning threshold is set to 0, no recognizer event is generated and warnings about large Earley sets are turned off. An Earley item threshold warning almost always indicates a serious issue, so turning these warnings off will rarely be something an application actually wants to do. 🚩

The too_many_earley_items setting is allowed in any call of any of the recognizer setting-aware methods.

🎀 trace_terminals

If it's non-zero, it traces the lexemes — those tokens passed from the L0 parser to the G1 parser. This recognizer setting is the best way to follow what the L0 parser is doing, and it's also super helpful for tracing the G1 parser. The trace_terminals setting is allowed in any call of any of the recognizer setting-aware methods. ✨

🦋 trace_values

The value of the trace_values setting is a numeric trace level. If that numeric trace level is 1, Marpa prints tracing information as values are computed in the evaluation stack. A trace level of 0 turns value tracing off, and that's the default. Traces get written to the trace file handle. The trace_values setting is allowed in any call of any of the recognizer setting-aware methods.

💖 trace_file_handle

The value is a file handle. Trace output and warning messages go to the trace file handle. By default, the trace file handle is inherited from the grammar. The trace_file_handle setting is allowed in any call of any of the recognizer setting-aware methods.

👑 Constructor 👑

    my $recce = Marpa::R2::Scanless::R->new( { grammar => $grammar } );

The new() method is the constructor for SLIF recognizers. The arguments to the new() constructor must be one or more hashes of named arguments, where each hash key is a recognizer setting. The grammar recognizer setting is required. All the other recognizer settings are optional. And if you want more on recognizer settings, go read the section that describes them. 💅

💅 Basic mutators 💅

okay so these are the methods that let you poke the recognizer after it’s already been constructed, bestie. mutator era, recognizer glow-up, whatever you wanna call it. let’s go. ✨

🌸 ambiguous() 🌸

if ( my $ambiguous_status = $recce->ambiguous() ) {
    chomp $ambiguous_status;
    die "Parse is ambiguous\n", $ambiguous_status;
}

Call this one after the read() method. If there’s exactly one parse, it hands you back the empty string. If there’s no parse, you get a non-empty string that says so. If there are two or more parses, you get a non-empty string describing the ambiguity. 💀

Applications should only test the returned string for empty vs non-empty. The non-empty strings are meant for humans to read only — their exact format is subject to change. No parsing the error text, bestie, iykyk.

When ambiguous() detects an ambiguous parse, it puts the recognizer into "forest mode" so it can examine the parse. As long as the recognizer is in forest mode, calls to the value() method will produce fatal errors. Forest mode can be cleared using the series_restart() method. That starts a new parse series in "tree mode", which lets calls to value() succeed. 🌳

🌸 read() 🌸

$recce->read($p_input_string);

$recce->read( \$string, 0, 0 );

Given a pointer to a physical input stream and, optionally, a span specifying an input string within it, read() parses the input string according to the grammar. read() returns success if it parses to the end of the input string, or if it triggers a SLIF parse event. Only a single call to read() is allowed for a SLIF recognizer. one and done, periodt. 💅

The first argument of read() is a pointer to the physical input stream which, by default, will be exactly the same as the virtual input stream. To specify the input string, read() recognizes optional second and third arguments and treats them as the start and length of a span of the physical input stream. The default start location is zero. The default length is -1. Negative locations and lengths are interpreted as described above. 📍

If a SLIF parse event occurs during the read() method, the current location is set to the trigger location. SLIF parse events are described in detail in a separate document. If no SLIF parse event triggers, and the parse reaches the end of the input string without a failure, the current location is set to the end of the input string. 🦋

On success, read() returns the current physical input stream location. This value may be zero. The call is considered successful if it reaches the end of input string, or if a SLIF parse event triggers. On failure, read() throws an exception. 😭

🌸 series_restart() 🌸

$recce->series_restart( { end => $i } );

The series_restart() method ends the current parse series, and starts another. Parse series are described in another document. The series_restart() method allows, as optional arguments, hashes whose key-value pairs are recognizer settings. 🔄

The series_restart() method cannot change the "grammar" recognizer setting. If any other recognizer setting is not specified explicitly, it is reset to its default. If an application wants an explicit recognizer setting to persist into a new parse series, it must specify that setting explicitly in the new parse series. series_restart() is particularly useful with the "end" and "semantics_package" named arguments. ✨

The series_restart() method must be called before value() when ambiguous() detects an ambiguous parse and the application needs to get the parse values. no skipping, bestie. 🚨

🌸 set() 🌸

$recce->set( { max_parses => 42 } );

This method allows recognizer settings to be changed after a SLIF grammar is created. The arguments to set() must be one or more hashes whose key-value pairs are recognizer settings and their values. The allowed recognizer settings are described above. 🎀

🌸 value() 🌸

my $value_ref = $recce->value( $self );

The "value" method call evaluates the next parse tree in the parse series, and returns a reference to the parse result for that parse tree. If there are no more parse trees, the "value" method returns "undef". There are zero parse trees if there was no valid parse of the input according to the grammar. There will be more than one parse tree if the parse was ambiguous. 🌈

The value() method allows one optional argument. If provided, the argument explicitly specifies the per-parse argument for the parse tree. This per-parse argument can be a Perl scalar of any type, but the most useful type for a per-parse argument is a reference (blessed or unblessed) to a hash or to an array. The per-parse argument, if provided, will be the first argument of all Perl semantics closures. When data does not conveniently fit into the bottom-up flow of parse tree evaluation, the per-parse argument is useful for sharing it within the tree. Symbol tables are one example of the kind of data which parses often require, but which it is not convenient to accumulate bottom-up. 🧚

If the "semantics_package" setting of the SLIF recognizer was not specified, Marpa will use the package into which the per-parse argument was blessed as the semantics package. (As a reminder, the semantics package is the package in which Marpa looks for the parse's Perl semantic closures.) 💖

When the per-parse argument of the value() method is the source of the semantics package, all calls to the value() method in the same parse series must have a per-parse argument that specifies the same semantics package. More precisely, if the per-parse argument of the first call of the value() method in a parse series is the source of the semantics package, it will be a fatal error if any subsequent value() call in that parse series:

does not have a per-parse argument;
if that per-parse argument is not blessed; or
if that per-parse argument is blessed into a different package.

✨ MUTATORS FOR EXTERNAL SCANNING ✨

`activate()` 💅

$recce->activate($_, 0) for @events;

okay so bestie, activate() is how your recognizer deactivates and reactivates SLIF parse events. SLIF parse events? They're described in a whole separate document, iykyk.

It takes two arguments. First one is the event's name. Second one is optional and is either 0 or 1. If you pass 0, the event gets deactivated. If you pass 1, it gets activated. Passing 1 is the default. Since an SLIF recognizer always starts with all defined events activated, 0 will probably be the more common second argument. periodt.

But wait — location 0 events are triggered in the SLIF recognizer's constructor, before you can even call activate(). They just don't get reported until you call read(). Currently there's no way to deactivate location zero events. no cap.

You can reduce event overhead with activate(). But making tons of activate() calls just for efficiency? Counter-productive, bestie. Also, deactivated events still impose some overhead. So if an event is never used, comment it out in the SLIF DSL. 💅

`lexeme_alternative()` 🦋

if ( not defined $recce->lexeme_alternative($token_name) ) {
    die
        qq{Parser rejected token "$long_name" at position $start_of_lexeme, before "},
        substr( $string, $start_of_lexeme, 40 ), q{"};
}

lexeme_alternative() lets an external scanner read ambiguous tokens. Most apps are gonna prefer the simpler lexeme_read() though.

It takes one or two arguments. The first is required: the name of a symbol to be read at the current location. The second is optional: the value of the symbol. That value argument is interpreted exactly as described for lexeme_read().

You can read any number of tokens with lexeme_alternative() without advancing the current location. That's the whole point — ambiguous tokens, bestie. To finish reading at a G1 location and advance the current G1 location to the next one, use lexeme_complete().

On success, it returns a non-negative number, which may be zero. Returns undef if the token was rejected. Failures are thrown as exceptions. 😭

`lexeme_complete()` ✨

next TOKEN
    if $recce->lexeme_complete( $start_of_lexeme,
            ( length $lexeme ) );

lexeme_complete() lets an external scanner read ambiguous tokens too. It completes the reading of a set of tokens that you specified with one or more calls to lexeme_alternative() at a G1 location. Most apps will still prefer the simpler lexeme_read().

It requires two arguments, which are the start and length parameters of a span in the physical input stream. That span is interpreted, and the G1 location and current input stream location are adjusted, just like described for lexeme_read().

SLIF parse events may occur during lexeme_complete(), as described for lexeme_read().

Return value: on success, lexeme_complete() returns the new current location. That will never be location zero, because a successful call always advances the location. Failure is thrown as an exception. 🙈

`lexeme_priority_set()` 👑

$recce->lexeme_priority_set( 'prefix lexeme', -1 );

First argument: the name of a lexeme. Second argument: the new priority for that lexeme. Both arguments are required. No skipping.

Changing lexeme priority is a very flexible technique. It can, in effect, let an application switch lexers. iconic.

On success, returns the old priority value. Failure is thrown. 💀

`lexeme_read()` 🌸

$re->lexeme_read( 'lstring', $start, $length, $value ) // die;

lexeme_read() reads a single, unambiguous, lexeme. It takes four arguments, but only the first is required. The first argument is the lexeme's symbol name. The second and third arguments specify the span in the physical input stream. The last argument specifies the value of the lexeme.

In the span from the second and third arguments, the start location defaults to the current location. If the pause span is defined, and the start of the pause lexeme is the same as the current location, length defaults to the length of the pause span. Otherwise length defaults to -1. Negative values are allowed and are interpreted as described above. (iykyk)

The span gets interpreted as the section of the physical input stream that corresponds to the current G1 set. Quick reminder: the G1 set is made of the tokens read at a single G1 location. That correspondence between the span and the token may be artificial, but a span is defined for every token, even if only by default.

The fourth argument specifies the lexeme value. The lexeme value plays an important role in the SLIF's semantics. More details on SLIF semantics live in a document dedicated to them. If the fourth argument is omitted, the lexeme value will be a string containing the corresponding substring of the input stream. Omitting the value argument does not have the same effect as passing an explicit Perl undef. If the value argument is an explicit Perl undef, the lexeme value will be a Perl undef. periodt.

$recce->lexeme_read($symbol, $start, $length, $value)

is roughly equivalent to

$recce->lexeme_alternative($symbol, $value)
$recce->lexeme_complete($start, $length)

Non-lexeme SLIF parse events may trigger during the lexeme_read() method. Lexeme SLIF parse events are ignored because they're designed to let you switch over to external scanning, and they make no sense when external scanning is already in progress. SLIF parse events are described in detail in a separate document.

Current input stream location will be set to $start+$length. If a SLIF parse event triggers, current input stream location will be set to the trigger location. Currently the trigger location and $start+$length will always be the same, but that may change. no cap.

When successful, lexeme_read() advances the current G1 location by one. The token read by lexeme_read() will start at the previous G1 location and end at the new current G1 location. The new current location in the input stream will be at the end location of the new lexeme.

On success, lexeme_read() returns the new current physical input stream location. This will never be location zero, because lexemes cannot be zero length. If the token was rejected, lexeme_read() returns a Perl undef. Failure is thrown as an exception. 😭

`resume()` 🧚

my $re = Marpa::R2::Scanless::R->new(
    {   grammar           => $parser->{grammar},
        semantics_package => 'MarpaX::JSON::Actions'
    }
);
my $length = length $string;
for (
    my $pos = $re->read( \$string );
    $pos < $length;
    $pos = $re->resume()
    )
{
    my ( $start, $length ) = $re->pause_span();
    my $value = substr $string, $start + 1, $length - 2;
    $value = decode_string($value) if -1 != index $value, '\\';
    $re->lexeme_read( 'lstring', $start, $length, $value ) // die;
} ## end for ( my $pos = $re->read( \$string ); $pos < $length...)
my $per_parse_arg = bless {}, 'MarpaX::JSON::Actions';
my $value_ref = $re->value($per_parse_arg);
return ${$value_ref};

The resume() method resumes the SLIF's internal scanning, as described above. A physical input stream must already have been specified using the $recce->read() method. resume() should only be called during the Reading Phase.

It takes two optional arguments, which represent the start and length parameters of a span in the physical input stream. The default start location is the current location. The default length is -1. Negative arguments are interpreted as described above.

resume() is considered successful if it reads input to the end of the input string, or if a SLIF parse event triggers. On success, resume() returns the new current location. On unthrown failure, resume() returns a Perl undef. Currently, all failures are thrown. 💅

🔮 ACCESSORS 🔮

`ambiguity_metric()` 📊

my $ambiguity_metric = $recce->ambiguity_metric();

Succeeds and returns 1 if there was an unambiguous parse — in other words, exactly one parse tree. Succeeds and returns 2 or greater if the parse was ambiguous — in other words, more than one parse tree. Succeeds and returns 0 if there are no parse trees, because parsing failed. Currently, all other failures are thrown.

When the return value is 2 or greater, that return value is not necessarily the parse count. Instead, it's a value subject to change, and an application should not rely on it. The intent was that, someday, return values of 2 or greater would represent a "metric" that was cheap to compute, but which estimated the degree of ambiguity in some useful way. The best metric would, of course, be the exact parse count, but determining that is expensive. 🙈

`current_g1_location()` 📍

my $current_g1_location = $recce->current_g1_location();

Returns the current G1 location. simple as that, bestie. ✨

`events()` 🎀

EVENT:
for my $event ( @{ $recce->events() } ) {
    my ($name) = @{$event};
    push @actual_events, $name;
}

The events() method takes no arguments, and returns an array of SLIF parse event descriptors. It returns the empty array if there were no events.

SLIF parse events are described in detail in a separate document. Each SLIF parse event descriptor is a reference to an array of one or more elements. The first element of every named event descriptor is a string containing the name of the event. Typically the name of the event is the only element. Other elements will be as described for each type of parse event.

Any other SLIF recognizer mutator may clear the events. It's expected that an application interested in events will call the events() method immediately after the event-triggering event. iykyk.

Named events are returned in order by type:

Lexeme events
Completion events
Nulling events
Prediction events

Within each type, the order of events is arbitrary. fr fr — don't build your whole personality around an ordering that isn't promised. 💅

🥱 exhausted()

my $exhausted_status = $recce->exhausted();

okay so exhausted() takes literally nothing, and hands you back a Perl true if parsing in a SLIF recognizer is exhausted, and a Perl false if it's not. And "exhausted" here means exactly what it sounds like: the recognizer is done-done, it will not accept any more input. 💀

Marpa usually just "does what you mean" when parse exhaustion happens, so you don't have to think about it — but this method is the way to directly peek at the recognizer's exhaustion status if you want the receipts. Parse exhaustion is discussed in detail in a separate document. iykyk 📄

📍 g1_location_to_span()

my ( $span_start, $span_length ) =
    $recce->g1_location_to_span($g1_location);

G1 locations don't map onto a single input stream location — they map onto a span of them, bestie. That's the whole vibe. g1_location_to_span() gives you back an array of two elements representing a span in the physical input stream. 🦋

Special case alert 🚨: G1 location 0 doesn't correspond to an input stream span at all, so as a special case the input stream span for G1 location 0 comes back as (0,0).

📏 input_length()

my $input_length = $recce->input_length();

Takes zero arguments. Returns the length of the physical input stream. That's it, that's the whole method. iconic, honestly. ✨

🏁 last_completed()

sub show_last_expression {
    my ($self) = @_;
    my $recce = $self->{recce};
    my ( $g1_start, $g1_len ) = $recce->last_completed('Expression');
    return 'No expression was successfully parsed' if not defined $g1_start;
    my $last_expression = $recce->substring( $g1_start, $g1_len );
    return "Last expression successfully parsed was: $last_expression";
} ## end sub show_last_expression

my ( $g1_start, $g1_len ) = $recce->last_completed('Expression');

You hand it the name of a symbol, and last_completed() slides back the 2-element array that is the G1 location span of the most recent match. 🎀

If there was more than one most recent match? It returns the longest. Longest wins, periodt.
If there was no match at all? It returns the empty array in array context, and a Perl false in scalar context. <em>no notes.</em>

🎯 last_completed_span()

my @longest_span = $recce->last_completed_span('target');
diag( "Actual target at $pos: ", $recce->literal(@longest_span) ) if $verbose;

Returns the most recent input stream span for a completed instance of the symbol name that is its first and only argument. That argument is required — you can't skip it, bestie. 🙈

And heads up: the search for a completed instance of a symbol can only succeed if that first argument is the name of the LHS symbol of some rule in the grammar. If you want the full deets on how the input stream span gets determined, go read "Literals and G1 spans".

If more than one instance of the symbol ends at the same location, last_completed_span() returns the longest span. If there's no symbol instance for the argument symbol, it returns the empty array. And any other failures? Those get thrown. Not returned — thrown. 💀

🧭 line_column()

my ( $start, $span_length ) = $re->pause_span();
my ( $line,  $column )      = $re->line_column($start);

line_column() takes one optional argument: a location in the input stream. No argument? It defaults to the current location. 🌸

It returns the corresponding line and column position as a 2-element array. First element = the line position, second element = the column position. Easy.

Numbering of lines and columns is 1-based, following UNIX editor tradition. 🫶 Except at EOVS (the end of the virtual input stream), the line and column will be that of an actual character. At EOVS, the line number will be that of the last line, and the column number will be that of the last column plus one. If your app wants to treat EOVS as a special case, you can test for it using the pos() method and the input_length() method. chef's kiss.

A line is considered to end with any newline sequence as defined in the Unicode Specification 4.0.0, Section 5.8. Specifically, a line ends with one of the following:

a LF (line feed U+000A);
a CR (carriage return, U+000D), when it is not followed by a LF;
a CRLF sequence (U+000D,U+000A);
a NEL (next line, U+0085);
a VT (vertical tab, U+000B);
a FF (form feed, U+000C);
a LS (line separator, U+2028) or
a PS (paragraph separator, U+2029).

that's EIGHT whole newline types btw. unicode said "what if lines ended in every direction" and honestly? slay. 🌈

📝 literal()

my $literal_string = $re->literal( $start, $span_length );

literal() takes two arguments: the start location and the length of a span in the physical input stream. It returns the substring of the input stream corresponding to that span. short, sweet, does exactly one thing. ✨

🚫 pause_lexeme()

my $lexeme = $re->pause_lexeme();

Use of this method is discouraged. New applications should avoid it. Instead, the lexeme event should be declared as a named event — the named lexeme event can be set up in such a way that it uniquely identifies the lexeme that triggered it. SLIF parse events are described in detail in a separate document. 📄

pause_lexeme() accepts no arguments, and returns the current pause lexeme. More than one lexeme may trigger at the same location — and in that case, the choice of pause lexeme is made arbitrarily. 😭 That right there is one reason the use of pause_lexeme() is discouraged, bestie. It returns a Perl "undef" when the pause lexeme is undefined.

⏸️ pause_span()

my ( $start, $length ) = $re->pause_span();

No arguments. Returns the pause span as a 2-element array: start and length. The "pause span" is described in detail in another document. 📄 pause_span() returns a Perl "undef" if the pause span is undefined. 🦋

📌 pos()

my $pos = $recce->pos();

No arguments. Returns the current physical input stream location. that's it. that's the method. obsessed. 💖

📊 progress()

my $progress_output = $recce->progress();

Returns a reference to an array that describes the progress of a parse at a location. 🎀

With no argument, progress() reports progress at the current location.
If a G1 location is given as its argument, progress() reports progress at that G1 location. Negative G1 locations are interpreted as described above.

The progress reports returned by progress() identify rules by their G1 rule ID. And G1 rule IDs can be converted to a list of the rule's symbols using the rule() method of the SLIF grammar. 🌟 Details on progress reports can be found in their own document.

🖨️ show_progress()

my $show_progress_output = $recce->show_progress();

Returns a string showing the progress of the G1 parse. For a description of its output, see Marpa::R2::Progress. With no arguments, the string contains reports for the current location.

Locations can be specified as arguments to show_progress():

With a single integer argument N, the string contains reports for G1 location N.
Two numeric arguments are interpreted as a span of G1 locations, and the returned string contains reports for all locations in the span. For example, the method call $recce->show_progress(0, -1) will print progress reports for the entire parse. 🎉

okay but why G1 locations and not physical input stream locations? Good question bestie 💅 — because G1 locations represent a unique point in the parse. By contrast, a single physical input stream location might be visited many times by a SLIF recognizer. So a physical location just isn't a unique address the way a G1 location is.

⚠️ warning bestie: The output is intended only for reading by humans. The exact format is subject to change and should not be relied on by applications. Do not parse the parse-printer. 🙈

✂️ substring()

my $last_expression = $recce->substring( $g1_start, $g1_len );

Give it a G1 span — that's a G1 start location and a length in G1 locations — and substring() returns a substring of the input stream. A G1 length of zero will produce the zero-length string. (Yes, an empty string. That's the correct answer. 🎀)

The substring of the input stream is determined on the assumption that the application reads the input in lexical order and without gaps except for whitespace and other normal discards. When that is not the case, the substring is determined as described above. 🦋

🔮 terminals_expected()

my @terminals_expected = @{$recce->terminals_expected()};

Returns a reference to a list of strings, where the strings are the names of the lexemes acceptable at the current location. 🌈

The presence of a lexeme in this list means that lexeme will be acceptable in the next call of the resume() method. This is highly useful for Ruby Slippers parsing. 👑 A more fine-tuned approach is to identify the lexemes of interest and create "predicted symbol" events for them.

Now here's the tea ☕: some lexemes are specified in the G1 rules of the DSL as quoted strings or as character classes. That's convenient, but the lexemes created in this way do not have real names. Instead, internal names, like "[Lex-1]", are created for them, and these are what appear in the list of strings returned by terminals_expected(). 🙈 If an application wants a quoted string or a character class to have a mnemonic name, the application must provide that name explicitly, by specifying the character class or quoted string in an L0 rule.

💀 Discouraged methods 💀

Methods in this section continue to be supported, but their use is discouraged in favor of other, better solutions. New applications should avoid using discouraged methods. they're not broken, bestie, they're just not the move anymore. 🫶

📇 event()

my $event    = $recce->event($event_ix);

Use of this method is discouraged in favor of the more efficient events() method. ⚡

The event() method requires one argument, an event index. It returns a descriptor of the named event with that index, or a Perl "undef" if there is no such event. For more details on events, see the description of the events() method. 📄

🏁 last_completed_range()

Use of this method is discouraged in favor of "last_completed()". 🫶

Given the name of a symbol, last_completed_range() returns the G1 start and G1 end locations of the most recent match. If there was more than one most recent match, last_completed_range() returns the longest. If there was no match, last_completed_range() returns the empty array in array context and a Perl false in scalar context. same energy, older API. ✨

🔤 range_to_string()

Use of this method is discouraged in favor of "substring()". 🎀

Given a G1 start and a G1 end location, range_to_string() returns the substring of the input stream that is between the two. The range_to_string() method assumes that the application read the physical input stream in lexical order and without gaps except for whitespace and other normal discards. When that is not the case, range_to_string() behaves in much the same way as described above for "substring()". 🦋

💖 COPYRIGHT AND LICENSE 💖

okay bestie, final section, and yes it's the legal one, but it's short and it matters so let's go ✨

This file is part of Marpa::R2 💅

🌈 the actual license tea 🌈

Marpa::R2 is free software, and here's the deal, fr fr: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, and you get to pick — either version 3 of the License, or (at your option) any later version 👑

Marpa::R2 is distributed in the hope that it will be useful, but — and this is the important part, so listen — WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. 💀 yeah, the all-caps is doing a lot but it's literally how it's written, that's the vibe

You should have received a copy of the GNU Lesser General Public License along with Marpa::R2. If you did not — no worries, it happens — see <http://www.gnu.org/licenses/> and you're all set 🌸

not me reminding you that "the software might be broken and nobody promised you anything" is, like, a legally required mood, but here we are 🙈

aaand that's the whole page, bestie 🎀 you read a lot of manpage, lowkey iconic. this one came out of Marpa:...ess::R(3pm) — the perl manual section, running under perl v5.42.3, dated 2026-08-20 ✨ go forth and parse literally everything, you absolute main character 👑💜

📦 where this page comes from

ubuntu stonking: libmarpa-r2-perl 12.000000-1build1 · LGPL-3+~Marpa
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms