Marpa:...rogress(3pm)

📦

version 12.000000 · 2 versions

12.000000 · ubuntu stonking (latest)
12.000000 · ubuntu resolute

💖 Marpa::R2::Deprecated::NAIF::Progress 💖

okay bestie, this is Marpa::R2::Deprecated::NAIF::Progress(3pm) from the User Contributed Perl Documentation, and it's literally all about progress reports for the NAIF 🦋 buckle up.

✨ NAME ✨

Marpa::R2::Deprecated::NAIF::Progress — Progress reports for the NAIF, no cap 💅

🚨 THE NAIF INTERFACE IS DEPRECATED 🚨

This doc is describing the NAIF interface, and it is deprecated. So like... PLEASE DO NOT USE IT FOR NEW DEVELOPMENT. i'm so serious 😭

🌸 About this document 🌸

So this whole thing is about Marpa's progress reports for its named argument interface, aka NAIF. If you're a beginner, or you're not even sure which interface you care about, or you literally do not know what the NAIF interface is, then bestie you are probably looking for the document on progress reports for the SLIF interface instead 🫶

Here's the vibe of progress reports: they let an application know exactly where it is in the parse at any point. For parse locations of the user's choosing, progress reports list all the rules in play, and they say where the rule started and how far into the rule parsing has progressed. iconic behaviour honestly ✨

Now, if you're new to parsing, this "situational awareness" might seem like something you'd expect from literally any halfway-decent parser. But no. Situation awareness in most parsers is extremely limited, and it's lowkey more limited in the parsers people consider production quality 💀 Marpa isn't just unusual in how much feedback it gives the application — it breaks new ground. periodt 👑

Progress reports are super duper useful for debugging grammars. And because debugging creates a clear "narrative", the detailed example in this document is a debugging situation. So if you're here specifically to debug a grammar, go read the document on tracing problems before you read this one, k? 💖

🐣 Introduction to Earley parsing 🐣

To read the show_progress output, you really gotta have a basic idea of what Earley items are and what the info inside them means. Everything you need to know is right here in this section, bestie, i got you 🌈

🌼 Dotted rules 🌼

The whole idea behind Earley's algorithm is that you can parse by building a table of rules and where you are in those rules. And "where" means two things: location in the rule relative to the rule's symbols, and location relative to the parse's input stream. two things. it's giving duality 💫

Let's look at an example of a rule in a context-free grammar. Here's the rule for assignment from the Perl distribution's perly.y:

"    termbinop -> term ASSIGNOP term"

ASSIGNOP is perly.y's internal name for the assignment operator. In plain Perl terms, that's the "=" character 👑

So while parsing this rule, we could be at any of four possible locations. One location is at the beginning, before all of the symbols. The other three locations are immediately after each of the rule's three symbols.

Within a rule, the position relative to the symbols of the rule is traditionally shown with a dot. In fact, the symbol-relative rule position is very often called the dot location. Taken as a pair, a rule and a dot location are called a dotted rule. slay 💅

Here's our rule with a dot location indicated:

"    termbinop -> · term ASSIGNOP term"

The dot location in this dotted rule is at the beginning. A dot location at the beginning of a dotted rule means we haven't recognized any symbols in the rule yet. All we're doing is predicting that the rule will occur. And a dotted rule with the dot before all of its symbols is called a prediction or a predicted rule. it's manifesting ✨

Here's another dotted rule:

"    termbinop -> term · ASSIGNOP term"

In this one, we're saying we've seen a term, but we haven't recognized an ASSIGNOP yet. not yet bestie, we're getting there 😌

There's another special kind of dotted rule too: a completion. A completion (also called a completed rule) is a dotted rule with the dot after all of the symbols. Here's the completion for the rule we've been using as our example:

"    termbinop -> term ASSIGNOP term ·"

A completion means a rule has been fully recognized. ate and left no crumbs 👑

🦋 Earley items 🦋

So the dotted rules contain all but one piece of the information Earley's algorithm needs to track. The missing piece is the second of the two "wheres": where in the input stream. To link up input stream location and dotted rules, Earley's algorithm uses what are now called Earley items.

A convenient way to think of an Earley item is as a triple, or 3-tuple, consisting of dotted rule, origin and current location. The origin is the location in the input stream where the dotted rule starts. The current location (also called the dot location) is the location in the input stream that corresponds to the dot position. three little pieces, bestie, a whole trio 💖

Two noteworthy consequences follow from how origin and current location are defined:

If a dotted rule is a prediction, then origin and current location will always be the same.
The input stream location where a rule ends is not tracked unless the dotted rule is a completion. In other cases, an Earley item does not tell us if a rule will ever be completed, much less at which location. 💀

👀 The example 👀

For this example of debugging, I've taken a very common example of a grammar and deliberately introduced a problem. (All the code and the full trace outputs for this example are in the Appendix, so go look if you want the receipts.) I've commented out the correct start rule:

        ## { lhs => 'Expression', rhs => [qw/Term/] },

and replaced it with another start rule, one which will cause problems:

        { lhs => 'Expression', rhs => [qw/Factor/] },

In what follows, we're gonna pretend we don't already know where the problem is, and use the Marpa diagnostics and tracing facilities to "discover" it. 🕵️‍♀️ main character detective energy

⚠️ First warning ⚠️

Right off the bat, we get two warning messages:

    Inaccessible symbol: Add
    Inaccessible symbol: Term

If we were alert, these would already be enough to tell us there's a serious problem. "Inaccessible" symbols are symbols which cannot be reached from the start symbol. Which means the grammar will never produce them, and parses will never find them in the input. so like... that's bad 😭

Since Add and Term are both important symbols in our application, that should tell us our grammar has a serious problem. In fact, these warning messages would often be enough to point us right at the error. But — so we can look at more of Marpa's tracing facilities — let's pretend we have not had our morning coffee and that we miss the significance of these warning messages entirely. relatable honestly ☕🙈

🔎 Output from trace_terminals() 🔎

Before looking at Marpa's progress reports, it's usually best to orient yourself by looking at the output from trace_terminals. Typically you'll be interested in the last tokens to be accepted, and maybe also in the tokens the recognizer was looking for when it didn't find what it wanted. Sometimes that information alone is enough to make it crystal clear where the problem is. ✨

The full trace_terminals output for this example is in the Appendix. We see that the recognizer seems to accept "42*1" but it fails when it confronts the plus sign ("+"). The last two lines are:

    Accepted "Number" at 2-3
    Expecting "Multiply" at 3

A note in passing, bestie: Marpa shows the location of the tokens it accepts as a range of locations. For Number, the range is "2-3", which means the token starts at location 2 and ends at location 3. In its default input model, all tokens have length 1, so this is very clearly information overkill 💀 But Marpa allows other input models, and in those models the info about start and end location of the token actually matters. iykyk 🌸

Returning to the problem at hand: we notice that at location 3 we're expecting a Multiply operator, but not an Add operator. That should strike us as strange and send us right back to the grammar. But for the sake of our example we're gonna assume we're slow on the uptake today and that this doesn't clue us in at all. We move on. 🚶‍♀️

🌸 Output from show_progress() 🌸

Okay bestie, so Marpa's most powerful tool for debugging grammars is its progress report 💅. It shows you the Earley items that are being worked on. In the Appendix, progress reports for the entire parse are shown. Our example in this document is a very small one, so producing progress reports for the entire parse is a reasonable thing to do in this case. But if a parse is at all large, you'll usually need to be selective. No cap.

The progress report that's usually of most interest is the one for the Earley set that you were working on when the error occurred. That's called the current location ✨. In our example, the current location is location 3. By default, show_progress prints out only the progress reports for the current location. Periodt.

Here are the progress reports for the current location, location 3, from our example.

F0 @0-3 Expression -> Factor .
F2 @2-3 Factor -> Number .
R4:1 x2 @0,2-3 Factor -> Factor . Multiply Factor
F4 @0-3 Factor -> Factor Multiply Factor .

✨ Progress report lines ✨

The last field of each Progress Report line shows, in fully expanded form, the dotted rule we were working on. Since that's the most important info, it may be tempting to skip the rest of this section and move directly forward with the debugging.

In fact, you might want to do exactly that — skip to the beginning of the next section. What follows talks about the details of the format of the first few fields in each progress report line. These first few fields, while helpful, are also usually one or more of: obvious in their meaning, not relevant to our example, and repetitive of information which can be deduced from other fields. So like, iykyk.

F0 @0-3 Expression -> Factor .

Prefixed to the dotted rule are two fields: F0 @0-3. The F0 says that this is a completed or final rule, and that it is rule number 0. The rule number is used in other tracing and debugging output, when displaying the whole rule would take too much space. In what follows we won't need the rule number.

The @0-3 describes the location of the dotted rule in the parse. In its simplest form, the location field is two location numbers, separated by a hyphen. The first location number is the origin, the place where Marpa first started recognizing the rule. The last location number is the dot location, the location of the dot in a dotted rule. @0-3 says that this rule began at location 0, and that the dot is at location 3.

The current location is location 3, and this is no coincidence. Whenever we are displaying the progress report for a location, all the progress report lines will have their dot location at that location. Slay.

As an aside, notice that the left hand side symbol is Expression. That is the start symbol. The presence of a completed start rule in our progress report indicates that if our input ended at location 3, it would be a valid sentence in the language of our grammar. Iconic.

Let's look at another progress report line:

R4:1 x2 @0,2-3 Factor -> Factor . Multiply Factor

Here the R4:1 indicates that this is rule number 4 (the R stands for rule number) and that its dot position is after the first symbol on the right hand side. Symbol positions are numbered using the ordinal of the symbol just before the position. Symbols are numbered starting with 1, and symbol position 1 is the position immediately after symbol 1.

The next field (x2) is new. It is a count. A progress report can contain multiple instances of the same dotted rule, and when there is more than one, a count field is included in the progress report line. Here the x2 indicates that there are two instances of Factor -> Factor . Multiply Factor at this location.

Multiple instances of a dotted rule will differ in their origin, and where they do, this is shown in the location field of the progress report line. Here the location field is @0,2-3, which indicates that one instance of this dotted rule has its origin at location 0, and the other has its origin at location 2. All instances reported on a single progress report line will always have the same dot location, and in this case it is location 3.

Predicted rules also appear in progress reports:

P2 @2-2 Factor -> . Number

Here the P in the summary field means predicted. As with much of the information in the summary field, this only repeats what is obvious from the full expansion of the dotted rule later in the line. But final (or completed) and predicted rules can be important and the initial F and P make these lines easy to spot. 👀

Notice that in the predicted rule, the origin is the same as the dot location. This will always be the case with predicted rules.

For any given location, no predicted rule has more than one instance. For other dotted rules, there may be many instances of the dotted rule at a single location. In grammars with right recursion, the number of instances is limited only by the length of the recursion. The length of a recursion is limited primarily by the available memory. So like, don't go infinite, bestie. 💀

When there are many instances of a dotted rule at a single location, it is inconvenient to show all the origins in a comma-separated list. In that case the origins in the location field are shown as a range, with the earliest separated from the most recent by a .... The example in this document contains no lines with a large number of instances, but here is an example from another grammar. This is the progress report line for the completed rule in a right recursion of length 20.

F1 x19 @0...18-20 Top_sequence -> Top Top_sequence .

🔍 OK! Now to find the bug 🔍

Here again are progress reports at the location where things went wrong:

F0 @0-3 Expression -> Factor .
F2 @2-3 Factor -> Number .
R4:1 x2 @0,2-3 Factor -> Factor . Multiply Factor
F4 @0-3 Factor -> Factor Multiply Factor .

We see that we have completed rules for Expression and Factor, as expected. We also see two Earley items that show that we are in the process of building another Factor, and that it is expecting a Multiply symbol. This is not the rule we want, but it explains why the trace_terminals output showed that the recognizer was expecting a Multiply symbol.

What we want to know is, why is the recognizer not expecting an Add symbol? Looking back at the grammar, we see that only one rule uses the Add symbol: the rule Term -> Term Add Term. The next step is to look at the Earley items for this rule. But there is a problem. We don't find any. 😭

Next, we ask ourselves, what is the earliest place the Term -> Term Add Term rule should be appearing? The answer is that there should be a prediction of Term -> Term Add Term at location 0. So we look at the predictions at location 0.

P0 @0-0 Expression -> . Factor
P2 @0-0 Factor -> . Number
P4 @0-0 Factor -> . Factor Multiply Factor

No Term -> Term Add Term rule. We are never even predicting a Term -> Term Add Term rule. We look back at the grammar, and start from the beginning.

{ lhs     => 'Expression', rhs => [qw/Factor/] },
{ lhs => 'Term',       rhs => [qw/Factor/] },
{ lhs => 'Factor',     rhs => [qw/Number/] },
{   lhs    => 'Term',
    rhs    => [qw/Term Add Term/],
    action => 'do_add'
},
{   lhs    => 'Factor',
    rhs    => [qw/Factor Multiply Factor/],
    action => 'do_multiply'
},

The start symbol is Expression and we do see a rule with Expression on the left hand side. Expression in turn produces a Factor symbol, and there are two rules with Factor on the left hand side.

But none of these rules ever produce a Term. In fact, however far we follow the productions, no rule ever produces a Term. At this point we see the problem: If we start at the start symbol, and follow the rules of our grammar, we will never get to a Term symbol. Which is exactly what that first warning message was saying. 💡

Now that we know what is wrong, we can reread our grammar, and see that our Expression -> Factor rule is wrong. It should be Expression -> Term. Change that and the problem is fixed. Periodt. 💅

🧚 Empty rules 🧚

When a symbol is nulled in your parse, show_progress does not show its expansion into rules. This reduces clutter, and seems to be the intuitive way to treat nulled rules. ✨

🌈 Access to the "raw" progress report information 🌈

Okay bestie, this whole section is about the progress() recognizer method — it's the one that hands you the raw, unfiltered progress report info. 💅 Lowkey important note: you do not need this for normal debugging and tracing. It's for the apps that wanna get creative and harness Marpa's "situational awareness" in ways nobody's thought of yet. main character energy fr fr.

💅 progress() 💅

my $report0 = $recce->progress(0);

my $latest_report = $recce->progress();

So here's the tea: you give it the parse location (that's the Early set ID) as its argument, and the progress() recognizer method hands back a reference to an array of "report items". 🎀

And yes, the parse location can be negative. An argument of -X gets interpreted as location N+X+1, where N is the latest Earley set. Which means:

-1 = the latest Earley set ✨
-2 = the Earley set just before the latest one
and so on down the line, you get it 💫

Each report item is a triple — an array of three elements. In order, those three are: rule ID, dot position, and origin. The data from the two displays above, plus the data for all the other parse locations in our example, is all shown below. 👑

The rule ID is the exact same number Marpa uses to identify rules in tracing and debugging output. So if you've got a rule ID, your app can go find its LHS and RHS symbols using the grammar's rule() method. handy!

Dot position is -1 for completions, and 0 for predictions. If the report item isn't for a completion or a prediction, dot position is N, where N is the number of RHS symbols successfully recognized at the parse location of the progress report.

Origin is the parse location (Earley set ID) where the rule application reported by the report item began. And for a prediction? Origin is always the same as the location of the parse report. always. no exceptions. 😌

🦋 Progress reports and efficiency 🦋

Okay so when you use progress reports for production parsing instead of just for debugging and tracing, efficiency starts to matter. Let's do the good news first because we're positive here. 💖

The good news: progress() is implemented in C, so your app can usually expect calls to progress() to be extremely fast. slay.

Now the potential bad news: if you're planning to call progress() a lot, you need to know that where there's a right recursion at a parse location, progress() has to follow the entire chain of recursions to build the progress report. 😭

And you'll always be able to tell this is happening from the parse report itself — it'll contain one report item for every completion in that right-recursive chain of completions. So if your app is trying to produce progress reports for a large number of parse locations, and those parse locations have long right-recursive chains? It can get computationally expensive. not cute.

Now, that efficiency thing I just mentioned about following right recursions? Not an issue for left recursions. Left recursions only produce at most two report items per parse location and are extremely fast to process. Also not an issue for Marpa's sequence rules, because sequence rules are implemented internally as left recursions. 🧚✨

📜 Appendix: full code and output for the example 📜

Below are the code, the trace outputs and the progress report for the example used in this document. 🫶

✨ Code ✨

my $grammar = Marpa::R2::Grammar->new(
    {   start          => 'Expression',
        actions        => 'My_Actions',
        default_action => 'first_arg',
        rules          => [
            ## This is a deliberate error in the grammar
            ## The next line should be:
            ## { lhs => 'Expression', rhs => [qw/Term/] },
            ## I have changed the Term to 'Factor' which
            ## will cause problems.
            { lhs => 'Expression', rhs => [qw/Factor/] },
            { lhs => 'Term',       rhs => [qw/Factor/] },
            { lhs => 'Factor',     rhs => [qw/Number/] },
            {   lhs    => 'Term',
                rhs    => [qw/Term Add Term/],
                action => 'do_add'
            },
            {   lhs    => 'Factor',
                rhs    => [qw/Factor Multiply Factor/],
                action => 'do_multiply'
            },
        ],
    }
);

$grammar->precompute();

my @tokens = (
    [ 'Number',   42 ],
    [ 'Multiply', q{*} ],
    [ 'Number',   1 ],
    [ 'Add',      q{+} ],
    [ 'Number',   7 ],
);

sub My_Actions::do_add {
    my ( undef, $t1, undef, $t2 ) = @_;
    return $t1 + $t2;
}

sub My_Actions::do_multiply {
    my ( undef, $t1, undef, $t2 ) = @_;
    return $t1 * $t2;
}

sub My_Actions::first_arg { shift; return shift; }

my $recce = Marpa::R2::Recognizer->new(
    { grammar => $grammar, trace_terminals => 2 } );

my $token_ix = 0;

TOKEN: for my $token_and_value (@tokens) {
    last TOKEN if not defined $recce->read( @{$token_and_value} );
}

$progress_report = $recce->show_progress( 0, -1 );

🔥 Trace output 🔥

Inaccessible symbol: Add
Inaccessible symbol: Term
Setting trace_terminals option
Expecting "Number" at earleme 0
Accepted "Number" at 0-1
Expecting "Multiply" at 1
Accepted "Multiply" at 1-2
Expecting "Number" at 2
Accepted "Number" at 2-3
Expecting "Multiply" at 3
Rejected "Add" at 3-4

Okay, notice the use of the term "earleme". If you're on the default input model, you can just assume earleme means "location" — the earleme and the location will always be exactly the same. ✨ Advanced users using alternative input models may set it up so earleme and location are two different things, and in that case, bestie, the distinction matters. 🦋

🎀 show_progress() output 🎀

P0 @0-0 Expression -> . Factor
P2 @0-0 Factor -> . Number
P4 @0-0 Factor -> . Factor Multiply Factor
F0 @0-1 Expression -> Factor .
F2 @0-1 Factor -> Number .
R4:1 @0-1 Factor -> Factor . Multiply Factor
P2 @2-2 Factor -> . Number
P4 @2-2 Factor -> . Factor Multiply Factor
R4:2 @0-2 Factor -> Factor Multiply . Factor
F0 @0-3 Expression -> Factor .
F2 @2-3 Factor -> Number .
R4:1 x2 @0,2-3 Factor -> Factor . Multiply Factor
F4 @0-3 Factor -> Factor Multiply Factor .

💖 progress() outputs 💖

This section contains the output of the progress() method — the progress reports in their "raw" format. The output is shown in Data::Dumper format, with "Data::Dumper::Indent" set to 0 and "Data::Dumper::Terse" set to 1. 💅

The "Data::Dumper" output from progress() at location 0:

[[0,0,0],[2,0,0],[4,0,0]]

The "Data::Dumper" output from progress() at location 1:

[[0,-1,0],[2,-1,0],[4,1,0]]

The "Data::Dumper" output from progress() at location 2:

[[2,0,2],[4,0,2],[4,2,0]]

The default progress() output is for the latest Earley set, which is location 3 in our example. Here's the progress() output for location 3.

[[0,-1,0],[2,-1,2],[4,-1,0],[4,1,0],[4,1,2]]

⭐ Copyright and License ⭐

Copyright 2022 Jeffrey Kegler
This file is part of Marpa::R2.  Marpa::R2 is free software: you can
redistribute it and/or modify it under the terms of the GNU Lesser
General Public License as published by the Free Software Foundation,
either version 3 of the License, or (at your option) any later version.

Marpa::R2 is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser
General Public License along with Marpa::R2.  If not, see
http://www.gnu.org/licenses/.

and that's a wrap on the whole page bestie!! 🥳💖 this one's from perl v5.42.3 and it's dated 2026-08-20 — so go forth, read your raw progress reports, and remember: negative earleme IDs are your friend 💅✨ periodt 🦋🎀

📦 where this page comes from

ubuntu stonking: libmarpa-r2-perl 12.000000-1build1 · LGPL-3+~Marpa
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms