Marpa:...::Order(3pm)

📦
version 12.000000 · 6 versions
, in ubuntu stonking

💖 NAME 💖

Marpa::R2::Semantics::Order — how the SLIF ranks ambiguous parses. iykyk 💅

✨ Description ✨

okay so bestie, here's the tea ☕ — Marpa allows ambiguous parses. like, it's not a bug, it's a feature. an unambiguous parse can produce at most one parse tree and one parse result, but an ambiguous parse will produce a parse series. and a parse series is literally just a sequence of parse trees, where each one has its own parse result. slay.

this doc is all about controlling the order in which the SLIF recognizer's value() method evaluates the parse trees of an ambiguous parse. 🦋 and also — plot twist — it describes ways to exclude selected parse trees from the parse series. we're doing both. periodt.

🌸 Default parse order 🌸

by calling the recognizer's value() method repeatedly (yes, over and over, she's persistent 💅), Marpa can produce all the parse results in the current parse series.

the default? the parse results come back in an arbitrary parse order. that corresponds to the "none" value of the recognizer's ranking_method named argument. it's giving chaos, but in a controlled way.

Traversal of the parse trees in arbitrary parse order will always be well-behaved in the sense that no two parse trees will be semantic duplicates, and no unique (semantic non-duplicate) parse tree will be omitted in it.

and that's genuinely the only promise you get, bestie. no other property of arbitrary parse order is guaranteed at all. like, the order may change every single time the parse series is traversed. 😭 iconic behaviour honestly.

🧚 Choicepoints 🧚

when ranking, the logic traverses each node of the parse bocage. in this context, those nodes are also called choicepoints. 🎀

from the point of view of the individual parse trees, the traversal will be top-down and left-to-right.

each choicepoint has one or more choices. often a choicepoint has only a single choice, and in that case it's called trivial (we love a low-effort node 🫶). and for two rule instances to be choices of the same choicepoint, they must end at the same location, and their rules must have the same LHS. no exceptions, no cap.

btw the terms choicepoint and choice are defined super carefully in a separate document 📄 — and that document also gives several examples of ranking, explained in detail. go read it, it eats.

👑 Ranking methods 👑

SLIF recognizer objects have a ranking_method named argument, and its value can be either:

  • the name of a ranking method, or
  • "none", which means the default ranking method is used 💅

that's it, that's the menu.

🔥 The "rule" ranking method 🔥

the rule method ranks alternative parses according to their rule alternatives. every rule alternative has a numeric rank. ✨

  • a rule's rank can be specified using the rank adverb argument for that RHS alternative
  • rule ranks must be integers
  • they may be negative (yes, go off, negative ranks are valid)
  • if no numeric rank is specified, the numeric rank is 0

💀 The "high_rule_only" ranking method 💀

high_rule_only is basically the rule method's sibling, same energy, one key difference: at every choicepoint, it discards all of the choices which have a rank lower than that of the highest ranked choice. ruthless. we respect it.

careful though — high_rule_only can reduce the ambiguity of a parse, but it does not necessarily do so. this is because, at each choicepoint among the parse trees, it's possible that several of the choices — or literally all of them — will have the same rank as the highest ranked choice. so like, ties happen. 🙈

🎀 Rule ranking 🎀

at each choicepoint, the choices are ranked like this:

  • Different numeric ranks: if the two parse choices have different numeric ranks, they must also have different rule alternatives. the parse choice whose rule alternative has the higher numeric rank will rank high. 💅
  • Same rule alternative: if the two parse choices have the same rule alternative, they rank as described under "Null variant ranking". (scroll down bestie, it's right there ⬇️)
  • Same numeric rank, different rule alternatives: two different rule alternatives can have the same numeric rank. if the two parse choices are for rule alternatives that are different, but that have the same numeric rank, the relative order of the two parse choices is arbitrary. shrug, it's giving random. ✨

one more thing — rule alternatives may be part of a single rule in the DSL, like a prioritized rule. but lexical order within a DSL rule makes no difference when ranking rule alternatives. for example, it makes no difference whether the two rule alternatives come from the same prioritized rule, or from two different prioritized rules. 🌈

🦋 Null variant ranking 🦋

some rules have a RHS which contains proper nullables: symbols which may be nulled, but which are not nulling symbols. (for the record — nulling symbols are symbols which are always nulled. just so we're clear. 💖)

when a rule alternative contains proper nullables, each instance of that rule creates a nulling variant. and a nulling variant is a specific pattern of null and non-null symbols in a rule instance's RHS. ✨

in many cases, this creates an ambiguity — because different nulling variants can match the same substring in the input. and in ambiguous parsings of this kind, some applications may want to rank nulling variants that start with non-null symbols higher. other applications may want the exact opposite, ranking nulling variants that start with null symbols higher. both are valid, no judgement 🫶

the null-ranking adverb for RHS alternatives specifies which nulling variants are ranked high or low:

  • if the null-ranking is "low", then the closer a nulling variant places its visible (non-null) symbols to the start of the rule instance, the higher it ranks. a null ranking of "low" is the default.
  • if the null-ranking is "high", then the closer a nulling variant places its null symbols to the start of the rule instance, the higher it ranks.

and when ranking nulling variants with more than one proper nullable, major-to-minor is left-to-right. 💅

🌈 A general approach to sorting parses 🌈

the most general way to sort Marpa parses is for the application to take control. main character energy, honestly. 👑

here's the move:

the application can set up the Marpa semantic actions so that the parse result of every parse tree is a <rank, true_value> duple. then the duples can be sorted by rank. once the results are sorted, the rank element of the duple can be discarded. ✨

(for my nerds who know — those familiar with the Schwartzian transform may note a resemblance. in Perl, duples can be implemented as references to arrays of 2 elements. 🧚)

bestie, be careful with this one. in theory, ambiguity can cause an exponential explosion in the number of results. 💀 and in practice, ambiguity tends to get out of hand very easily. producing and sorting all the parses can take a very long time. like, go make a snack. 🍓

💜 Details 💜

this section has extra explanations — not essential to understanding the rest of this document. often they're formal or mathematical. some people find them helpful, others find them distracting, which is exactly why they're segregated down here. 🙈

⭐ Duplicate parses ⭐

when evaluating the parse trees in a parse series, Marpa never evaluates the same parse tree twice. and what that means probably matches the programmer's intuition of what it should mean. ✨

Marpa considers two parse trees to be the same if they are semantic equivalents.

two parse trees are semantic equivalents if and only if a recursive, top-down evaluation of each applies the same rules in the same order at the same G1 locations. and if the semantics are deterministic, and if two parse trees are semantic equivalents according to this definition, then the two parse trees will always produce the same parse result. 💅

they're called semantic equivalents because, from the point of view of a deterministic semantics, they are indistinguishable. 🦋 and when the Marpa documentation refers to duplicate parses, unless otherwise stated, it means that the two are semantic equivalents.

formally, semantic equivalence is defined like this:

  • call the set of parse trees, T
  • semantic equivalence is an equivalence relation on T
  • call this relation ~
  • call E, the quotient set of T by ~

and in this document, the term arbitrary parse order is used to mean an arbitrary choice among the relations which are strict total orders of E. 🌈 periodt.

Copyright 2022 Jeffrey Kegler
This file is part of Marpa::R2.  Marpa::R2 is free software: you can
redistribute it and/or modify it under the terms of the GNU Lesser
General Public License as published by the Free Software Foundation,
either version 3 of the License, or (at your option) any later version.

Marpa::R2 is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser
General Public License along with Marpa::R2.  If not, see
http://www.gnu.org/licenses/.

okay that's the whole page bestie, you ate that 🫶 this was the Marpa:...::Order(3pm) page, shipped as part of Perl v5.42.3 and dated 2026-08-20 in the User Contributed Perl Documentation. go forth and rank your ambiguous parses responsibly. 💖✨🎀

📦 where this page comes from