LARK(7)
okay bestie, welcome to lark(7) โ this is the Lark Documentation manual, and we're about to read the whole thing together, no cap. ๐
๐ฆ NAME ๐ฆ
lark โ Lark Documentation
๐ญ PHILOSOPHY ๐ญ
So like, parsers? Bestie, they're innately complicated and confusing. ๐ They're difficult to understand, difficult to write, and difficult to use. Even experts on the subject can become baffled by the nuances of these complicated state-machines, fr fr.
Lark's mission is to make the process of writing them as simple and abstract as possible, by following these design principles:
โจ Design Principles โจ
- Readability matters โ periodt.
- Keep the grammar clean and simple.
- Don't force the user to decide on things that the parser can figure out on its own โ the parser is a big girl, she can handle it. ๐
- Usability is more important than performance.
- Performance is still very important.
- Follow the Zen of Python, whenever possible and applicable.
In accordance with these principles, I arrived at the following design choices:
๐ Design Choices ๐
- Separation of code and grammar โ Grammars are the de-facto reference for your language, and for the structure of your parse-tree. For any non-trivial language, the conflation of code and grammar always turns out convoluted and difficult to read. Lark's grammars are EBNF-inspired, so they are especially easy to read & work with. It's giving clean. โจ
- Always build a parse-tree (unless told not to) โ Trees are always simpler to work with than state-machines. Lowkey iconic.
- Trees allow you to see the "state-machine" visually.
- Trees allow your computation to be aware of previous and future states.
- Trees allow you to process the parse in steps, instead of forcing you to do it all at once.
And anyway, every parse-tree can be replayed as a state-machine, so there is no loss of information. We love a reversible queen. ๐
See this answer in more detail here: <https://github.com/erezsh/lark/issues/4>.
To improve performance, you can skip building the tree for LALR(1), by providing Lark with a transformer (see the JSON example: <https://github.com/erezsh/lark/blob/master/examples/json_parser.py>).
- Earley is the default โ The Earley algorithm can accept any context-free grammar you throw at it (i.e. any grammar you can write in EBNF, it can parse). That makes it extremely friendly to beginners, who are not aware of the strange and arbitrary restrictions that LALR(1) places on its grammars. It's giving no gatekeeping. ๐ซถ
As the users grow to understand the structure of their grammar, the scope of their target language, and their performance requirements, they may choose to switch over to LALR(1) to gain a huge performance boost, possibly at the cost of some language features.
Both Earley and LALR(1) can use the same grammar, as long as all constraints are satisfied.
In short, "Premature optimization is the root of all evil."
๐ Other design features ๐
- Automatically resolve terminal collisions whenever possible.
- Automatically keep track of line & column numbers.
๐ฅ FEATURES ๐ฅ
๐ Main Features ๐
- Earley parser, capable of parsing any context-free grammar.
- Implements SPPF, for efficient parsing and storing of ambiguous grammars.
- LALR(1) parser, limited in power of expression, but very efficient in space and performance (
O(n)).- Implements a parse-aware lexer that provides a better power of expression than traditional LALR implementations (such as
ply).
- Implements a parse-aware lexer that provides a better power of expression than traditional LALR implementations (such as
- EBNF-inspired grammar, with extra features (See: Grammar Reference
<>). - Builds a parse-tree (AST) automagically based on the grammar.
- Stand-alone parser generator โ create a small independent parser to embed in your project. (read more:
<tools.html#stand-alone-parser>) - Flexible error handling by using an interactive parser interface (LALR only).
- Automatic line & column tracking (for both tokens and matched rules).
- Automatic terminal collision resolution.
- Warns on regex collisions using the optional
interegularlibrary. (read more:<how_to_use.html#regex-collisions>) - Grammar composition โ Import terminals and rules from other grammars (see example:
<https://github.com/lark-parser/lark/tree/master/examples/composition>). - Standard library of terminals (strings, numbers, names, etc.)
- Unicode fully supported.
- Extensive test suite.
- Type annotations (MyPy support).
- Pure-Python implementation.
Read more about the parsers: <>
๐ธ Extra features ๐ธ
- Support for external
regexmodule (see here:<classes.html#using-unicode-character-classes-with-regex>). - Import grammars from Nearley.js (read more:
<tools.html#importing-grammars-from-nearleyjs>). - CYK parser.
- Visualize your parse trees as dot or png files (see_example:
<https://github.com/lark-parser/lark/blob/master/examples/fruitflies.py>). - Automatic reconstruction of input from parse-tree (see example:
<https://github.com/lark-parser/lark/blob/master/examples/advanced/reconstruct_json.py>and another example:<https://github.com/lark-parser/lark/blob/master/examples/advanced/reconstruct_python.py>). - Use Lark grammars in Julia (
<https://github.com/jamesrhester/Lerche.jl>) and Javascript (<https://github.com/lark-parser/Lark.js>).
๐ง PARSERS ๐ง
Lark implements the following parsing algorithms: Earley, LALR(1), and CYK. Slay. ๐
๐ Earley ๐
An Earley Parser (<https://www.wikiwand.com/en/Earley_parser>) is a chart parser capable of parsing any context-free grammar at O(n^3), and O(n^2) when the grammar is unambiguous. It can parse most LR grammars at O(n). Most programming languages are LR, and can be parsed at a linear time. Main character energy for real.
Lark's Earley implementation runs on top of a skipping chart parser, which allows it to use regular expressions, instead of matching characters one-by-one. This is a huge improvement to Earley that is unique to Lark. This feature is used by default, but can also be requested explicitly using lexer='dynamic'.
It's possible to bypass the dynamic lexing, and use the regular Earley parser with a basic lexer, that tokenizes as an independent first step. Doing so will provide a speed benefit, but will tokenize without using Earley's ambiguity-resolution ability. So choose this only if you know why! Activate with lexer='basic'. Not me warning you to read first, bestie. ๐
๐ซถ SPPF & Ambiguity resolution ๐ซถ
Lark implements the Shared Packed Parse Forest data-structure for the Earley parser, in order to reduce the space and computation required to handle ambiguous grammars.
You can read more about SPPF here: <https://web.archive.org/web/20191229100607/www.bramvandersanden.com/post/2014/06/shared-packed-parse-forest>.
As a result, Lark can efficiently parse and store every ambiguity in the grammar, when using Earley.
Lark provides the following options to combat ambiguity:
- Lark will choose the best derivation for you (default). Users can choose between different disambiguation strategies, and can prioritize (or demote) individual rules over others, using the rule-priority syntax.
- Users may choose to receive the set of all possible parse-trees (using
ambiguity='explicit'), and choose the best derivation themselves. While simple and flexible, it comes at the cost of space and performance, and so it isn't recommended for highly ambiguous grammars, or very long inputs. - As an advanced feature, users may use specialized visitors to iterate the SPPF themselves. There is also a 3rd party utility for iterating over the SPPF:
<https://github.com/chanicpanic/lark-ambig-tools>.
๐ lexer="dynamic_complete" ๐
Earley's "dynamic" lexer uses regular expressions in order to tokenize the text. It tries every possible combination of terminals, but it matches each terminal exactly once, returning the longest possible match.
That means, for example, that when lexer="dynamic" (which is the default), the terminal /a+/, when given the text "aa", will return one result, aa, even though a would also be correct.
This behavior was chosen because it is much faster, and it is usually what you would expect. Honestly, same.
Setting lexer="dynamic_complete" instructs the lexer to consider every possible regexp match. This ensures that the parser will consider and resolve every ambiguity, even inside the terminals themselves. This lexer provides the same capabilities as scannerless Earley, but with different performance tradeoffs.
Warning: This lexer can be much slower, especially for open-ended terminals such as /.*/. Do not say I didn't warn you. ๐
๐ LALR(1) ๐
LALR(1) (<https://www.wikiwand.com/en/LALR_parser>) is a very efficient, true-and-tested parsing algorithm. It's incredibly fast and requires very little memory. It can parse most programming languages (For example: Python and Java). Ate and left no crumbs.
LALR(1) stands for:
- Left-to-right parsing order.
- Rightmost derivation, bottom-up.
- Lookahead of 1 token.
Lark comes with an efficient implementation that outperforms every other parsing library for Python (including PLY).
Lark extends the traditional YACC-based architecture with a contextual lexer, which processes feedback from the parser, making the LALR(1) algorithm stronger than ever.
The contextual lexer communicates with the parser, and uses the parser's lookahead prediction to narrow its choice of terminals. So at each point, the lexer only matches the subgroup of terminals that are legal at that parser state, instead of all of the terminals. It's surprisingly effective at resolving common terminal collisions, and allows one to parse languages that LALR(1) was previously incapable of parsing. The way it just works? Obsessed. โจ
(If you're familiar with YACC, you can think of it as automatic lexer-states.)
This is an improvement to LALR(1) that is unique to Lark.
๐ Grammar constraints in LALR(1) ๐
Due to having only a lookahead of one token, LALR is limited in its ability to choose between rules, when they both match the input. Vibe check: sometimes it just can't decide. ๐ญ
Tips for writing a conforming grammar:
- Try to avoid writing different rules that can match the same sequence of characters.
- For the best performance, prefer left-recursion over right-recursion.
- Consider setting terminal priority only as a last resort.
For a better understanding of these constraints, it's recommended to learn how a SLR parser works. SLR is very similar to LALR but much simpler. Study buddy time, bestie. ๐
โญ CYK Parser โญ
A CYK parser (<https://www.wikiwand.com/en/CYK_algorithm>) can parse any context-free grammar at O(n^3*|G|).
It's too slow to be practical for simple grammars, but it offers good performance for highly ambiguous grammars. So like, not for your basic needs, but a queen for chaos. ๐
๐ JSON PARSER - TUTORIAL ๐
Lark is a parser โ a program that accepts a grammar and text, and produces a structured tree that represents that text. In this tutorial we will write a JSON parser in Lark, and explore Lark's various features in the process.
It has 5 parts.
- Writing the grammar.
- Creating the parser.
- Shaping the tree.
- Evaluating the tree.
- Optimizing.
Knowledge assumed:
- Using Python ๐
- A basic understanding of how to use regular expressions
๐ Part 1 โ The Grammar ๐
Lark accepts its grammars in a format called EBNF (that's Extended BackusโNaur form, iykyk). It basically looks like this:
rule_name : list of rules and TERMINALS to match
| another possible list of items
| etc.
TERMINAL: "some text to match"(btw a terminal is a string or a regular expression ๐ฆ)
The parser will try to match each rule (that's the left-part) by matching its items (the right-part) one after another, trying each alternative. (In practice, the parser is predictive so we don't have to try every alternative.)
How to structure those rules is beyond the scope of this tutorial, but ngl often it's enough to just follow one's intuition. โจ
In the case of JSON, the structure is simple: a json document is either a list, or a dictionary, or a string/number/etc.
The dictionaries and lists are recursive, and contain other json documents (or "values").
Let's write this structure in EBNF form:
value: dict
| list
| STRING
| NUMBER
| "true" | "false" | "null"
list : "[" [value ("," value)*] "]"
dict : "{" [pair ("," pair)*] "}"
pair : STRING ":" valueA quick explanation of the syntax, bestie:
- Parenthesis let us group rules together. ๐ซถ
rule*means any amount. That means, zero or more instances of that rule.[rule]means optional. That means zero or one instance of that rule.
Lark also supports the rule+ operator, meaning one or more instances. It also supports the rule? operator which is just another way to say optional. ๐
Of course, we still haven't defined "STRING" and "NUMBER". Luckily, both these literals are already defined in Lark's common library:
%import common.ESCAPED_STRING -> STRING
%import common.SIGNED_NUMBER -> NUMBERThe arrow (->) renames the terminals. But that only adds obscurity in this case, so going forward we'll just use their original names.
We'll also take care of the white-space, which is part of the text, by simply matching and then throwing it away. ๐๏ธ
%import common.WS
%ignore WSWe tell our parser to ignore whitespace. Otherwise, we'd have to fill our grammar with WS terminals. no thank u
By the way, if you're curious what these terminals signify, they are roughly equivalent to this:
NUMBER : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
STRING : /".*?(?<!\\)"/
%ignore /[ \t\n\f\r]+/Lark will accept this way of writing too, if you really want to complicate your life :) ๐
You can find the original definitions in common.lark. They don't strictly adhere to json.org โ but our purpose here is to accept json, not validate it. periodt.
Notice that terminals are written in UPPER-CASE, while rules are written in lower-case. I'll touch more on the differences between rules and terminals later. ๐
๐ธ Part 2 โ Creating the Parser ๐ธ
Once we have our grammar, creating the parser is very simple.
We simply instantiate Lark, and tell it to accept a "value":
from lark import Lark
json_parser = Lark(r"""
value: dict
| list
| ESCAPED_STRING
| SIGNED_NUMBER
| "true" | "false" | "null"
list : "[" [value ("," value)*] "]"
dict : "{" [pair ("," pair)*] "}"
pair : ESCAPED_STRING ":" value
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
""", start='value')It's that simple! Let's test it out:
>>> text = '{"key": ["item0", "item1", 3.14]}'
>>> json_parser.parse(text)
Tree(value, [Tree(dict, [Tree(pair, [Token(STRING, "key"), Tree(value, [Tree(list, [Tree(value, [Token(STRING, "item0")]), Tree(value, [Token(STRING, "item1")]), Tree(value, [Token(NUMBER, 3.14)])])])])])])
>>> print( _.pretty() )
value
dict
pair
"key"
value
list
value "item0"
value "item1"
value 3.14As promised, Lark automagically creates a tree that represents the parsed text. โจ
But something is suspiciously missing from the tree. Where are the curly braces, the commas and all the other punctuation literals?? ๐คจ
Lark automatically filters out literals from the tree, based on the following criteria:
- Filter out string literals without a name, or with a name that starts with an underscore.
- Keep regexps, even unnamed ones, unless their name starts with an underscore.
Unfortunately, this means that it will also filter out literals like "true" and "false", and we will lose that information. The next section, "Shaping the tree" deals with this issue, and others. ๐ซก
๐ฆ Part 3 โ Shaping the Tree ๐ฆ
We now have a parser that can create a parse tree (or: AST), but the tree has some issues:
"true","false"and"null"are filtered out (test it out yourself!)- Is has useless branches, like
value, that clutter-up our view.
I'll present the solution, and then explain it:
?value: dict
| list
| string
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
...
string : ESCAPED_STRING- Those little arrows signify aliases. An alias is a name for a specific part of the rule. In this case, we will name the true/false/null matches, and this way we won't lose the information. We also alias
SIGNED_NUMBERto mark it for later processing. ๐ฏ - The question-mark prefixing
value(?value) tells the tree-builder to inline this branch if it has only one member. In this case,valuewill always have only one member, and will always be inlined. - We turned the
ESCAPED_STRINGterminal into a rule. This way it will appear in the tree as a branch. This is equivalent to aliasing (like we did for the number), but nowstringcan also be used elsewhere in the grammar (namely, in thepairrule).
Here is the new grammar:
from lark import Lark
json_parser = Lark(r"""
?value: dict
| list
| string
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
list : "[" [value ("," value)*] "]"
dict : "{" [pair ("," pair)*] "}"
pair : string ":" value
string : ESCAPED_STRING
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
""", start='value')And let's test it out:
>>> text = '{"key": ["item0", "item1", 3.14, true]}'
>>> print( json_parser.parse(text).pretty() )
dict
pair
string "key"
list
string "item0"
string "item1"
number 3.14
trueAh! That is much much nicer. ๐
๐ Part 4 โ Evaluating the tree ๐
It's nice to have a tree, but what we really want is a JSON object.
The way to do it is to evaluate the tree, using a Transformer.
A transformer is a class with methods corresponding to branch names. For each branch, the appropriate method will be called with the children of the branch as its argument, and its return value will replace the branch in the tree. so iconic honestly
So let's write a partial transformer, that handles lists and dictionaries:
from lark import Transformer
class MyTransformer(Transformer):
def list(self, items):
return list(items)
def pair(self, key_value):
k, v = key_value
return k, v
def dict(self, items):
return dict(items)And when we run it, we get this:
>>> tree = json_parser.parse(text)
>>> MyTransformer().transform(tree)
{Tree(string, [Token(ANONRE_1, "key")]): [Tree(string, [Token(ANONRE_1, "item0")]), Tree(string, [Token(ANONRE_1, "item1")]), Tree(number, [Token(ANONRE_0, 3.14)]), Tree(true, [])]}This is pretty close. Let's write a full transformer that can handle the terminals too.
Also, our definitions of list and dict are a bit verbose. We can do better:
from lark import Transformer
class TreeToJson(Transformer):
def string(self, s):
(s,) = s
return s[1:-1]
def number(self, n):
(n,) = n
return float(n)
list = list
pair = tuple
dict = dict
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: FalseAnd when we run it:
>>> tree = json_parser.parse(text)
>>> TreeToJson().transform(tree)
{u'key': [u'item0', u'item1', 3.14, True]}Magic! ๐งโจ
๐ Part 5 โ Optimizing ๐
โญ Step 1 โ Benchmark โญ
By now, we have a fully working JSON parser, that can accept a string of JSON, and return its Pythonic representation.
But how fast is it? ๐โโ๏ธ
Now, of course there are JSON libraries for Python written in C, and we can never compete with them. But since this is applicable to any parser you would write in Lark, let's see how far we can take this.
The first step for optimizing is to have a benchmark. For this benchmark I'm going to take data from json-generator.com/. I took their default suggestion and changed it to 5000 objects. The result is a 6.6MB sparse JSON file. ๐พ
Our first program is going to be just a concatenation of everything we've done so far:
import sys
from lark import Lark, Transformer
json_grammar = r"""
?value: dict
| list
| string
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
list : "[" [value ("," value)*] "]"
dict : "{" [pair ("," pair)*] "}"
pair : string ":" value
string : ESCAPED_STRING
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""class TreeToJson(Transformer):
def string(self, s):
(s,) = s
return s[1:-1]
def number(self, n):
(n,) = n
return float(n)
list = list
pair = tuple
dict = dict
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False
json_parser = Lark(json_grammar, start='value', lexer='basic')
if __name__ == '__main__':
with open(sys.argv[1]) as f:
tree = json_parser.parse(f.read())
print(TreeToJson().transform(tree))Bestie. We run it and we get this:
$ time python tutorial_json.py json_data > /dev/null
real 0m36.257s
user 0m34.735s
sys 0m1.361sSix megabytes. Thirty-six seconds. That is not the vibe. ๐ญ Not for a fancy DSL, not for a config file โ but we're out here trying to chew through a huge pile of data, and this is simply not it.
But plot twist โจ โ turns out there's SO much we can do about it. Let's go.
โจ Step 2 - LALR(1) โจ
Soooo up until now we've been running on the Earley algorithm, which is literally the default in Lark. Earley is powerful, for real, but it is slow. And here's the tea: our grammar just so happens to be LR-compatible. Specifically, LALR(1) compatible. ๐
So bestie, let's switch to LALR(1) and see what happens:
json_parser = Lark(json_grammar, start='value', parser='lalr')
$ time python tutorial_json.py json_data > /dev/null
real 0m7.554s
user 0m7.352s
sys 0m0.148sOkay SEVEN SECONDS?? That's so much better. ๐ And the resulting JSON is of course exactly the same โ like, identical identical. You can run it yourself and see, no cap.
Big important note though: not all grammars are LR-compatible, so you literally cannot always just switch to LALR(1). But there's no harm in trying! If Lark lets you build the grammar, that means you're good to go. Slay.
๐ธ Step 3 - Tree-less LALR(1) ๐ธ
Okay so this whole time we've been building a full parse tree for our JSON, and then transforming it. Convenient? Sure bestie. The most efficient in terms of speed and memory? Absolutely not. ๐
Luckily, Lark lets us avoid building the tree entirely when we're parsing with LALR(1). Here's how:
json_parser = Lark(json_grammar, start='value', parser='lalr', transformer=TreeToJson())
if __name__ == '__main__':
with open(sys.argv[1]) as f:
print( json_parser.parse(f.read()) )See what we did? We took the transformer we already wrote and plugged it straight into the parser. ๐ซถ Now it can skip building the parse tree and just yeet the data directly into our transformer. And parse() now returns the transformed JSON instead of a tree.
Benchmark time:
real 0m4.866s
user 0m4.722s
sys 0m0.121sThat's a measurable improvement, periodt. ๐ Plus this way is more memory efficient โ peep the benchmark table at the end to see just how much.
General practice, bestie: it's recommended to work with parse trees, and only skip the tree-builder once your transformer is already working. ๐ง
๐ฆ Step 4 - PyPy ๐ฆ
PyPy is a JIT engine for running Python, and it's designed to be a drop-in replacement. And here's why that matters for us: Lark is written purely in Python, which makes it very suitable for PyPy. ๐
So let's grab some free performance:
$ time pypy tutorial_json.py json_data > /dev/null
real 0m1.397s
user 0m1.296s
sys 0m0.083sPyPy is awesome! ๐ฅ
โญ Conclusion โญ
We brought the run-time down from 36 seconds to 1.1 seconds in a series of small and simple steps. Not me crying over a 1.4-second parse. ๐ญ
Now let's compare the benchmarks in a nicely organized table.
I measured memory consumption using a little script called memusg (https://gist.github.com/netj/526585).
I added a few other parsers for comparison. PyParsing and funcparselib do pretty well on memory usage (they don't build a tree), but they can't compete with the run-time speed of LALR(1).
These benchmarks are for Lark's alpha version โ I already have several optimizations planned that will significantly improve run-time speed. Stay tuned. ๐
And once again, shout-out to PyPy for being so effective. ๐ซถ
๐ Afterword ๐
This is the end of the tutorial. I hoped you liked it and learned a little about Lark.
To see what else you can do with Lark, check out the examples (</examples>).
Read the documentation here: https://lark-parser.readthedocs.io/en/latest/
๐ HOW TO USE LARK - GUIDE ๐
๐ Work process ๐
This is the recommended process for working with Lark, bestie โ like a lil routine but make it parsing:
- Collect or create input samples that demonstrate key features or behaviors in the language you're trying to parse.
- Write a grammar. Try to aim for a structure that's intuitive, and in a way that imitates how you would explain your language to a fellow human.
- Try your grammar in Lark against each input sample. Make sure the resulting parse-trees make sense.
- Use Lark's grammar features to shape the tree (
<>): get rid of superfluous rules by inlining them, and use aliases when specific cases need clarification.
You can perform steps 1-4 repeatedly, gradually growing your grammar to include more sentences.
- Create a transformer to evaluate the parse-tree into a structure you'll be comfortable working with. This may include evaluating literals, merging branches, or even converting the entire tree into your own set of AST classes.
Of course, some specific use-cases may deviate from this process. Feel free to suggest these cases, and I'll add them to this page. ๐
โจ Getting started โจ
- Browse the Examples (
https://github.com/lark-parser/lark/tree/master/examples) to find a template that suits your purposes. - Read the tutorials to get a better understanding of how everything works. (links in the main page
<>) - Use the Cheatsheet (PDF) (
https://lark-parser.readthedocs.io/en/latest/_static/lark_cheatsheet.pdf) for quick reference. - Use the reference pages for more in-depth explanations. (links in the main page
<>)
๐ Debug ๐
Grammars may contain non-obvious bugs, usually caused by rules or terminals interfering with each other in subtle ways. Rude. ๐
When trying to debug a misbehaving grammar, the following methodology is recommended:
- Create a copy of the grammar, so you can change the parser/grammar without any worries.
- Find the minimal input that creates the error.
- Slowly remove rules from the grammar, while making sure the error still occurs.
Usually, by the time you get to a minimal grammar, the problem becomes clear. Iconic.
But if it doesn't, feel free to ask us on gitter, or even open an issue. Post a reproducing code, with the minimal grammar and input, and we'll do our best to help. ๐ซถ
๐ Regex collisions ๐
A likely source of bugs occurs when two regexes in a grammar can match the same input. If both terminals have the same priority, most lexers would arbitrarily choose the first one that matches โ which isn't always the desired one. (a notable exception is the dynamic_complete lexer, which always tries all variations. But its users pay for that with performance. ๐)
These collisions can be hard to notice, and their effects can be difficult to debug, as they are subtle and sometimes hard to reproduce.
To help with these situations, Lark can utilize a new external library called interegular. If it's installed, Lark uses it to check for collisions, and warn about any conflicts that it can find:
import logging
from lark import Lark, logger
logger.setLevel(logging.WARN)
collision_grammar = '''
start: A | B
A: /a+/
B: /[ab]+/
'''
p = Lark(collision_grammar, parser='lalr')
# Output:
# Collision between Terminals B and A. The lexer will choose between them arbitrarily
# Example Collision: aYou can install interegular for Lark using pip install 'lark[interegular]'.
Note 1: Interegular currently only runs when the lexer isbasicorcontextual.
Note 2: Some advanced regex features, such as lookahead and lookbehind, may prevent interegular from detecting existing collisions.
๐ Shift/Reduce collisions ๐
By default, Lark automatically resolves Shift/Reduce conflicts as Shift. It produces notifications as debug messages.
When users pass debug=True, those notifications are written as warnings.
Either way, to get the messages printed you have to configure the logger beforehand. For example:
import logging
from lark import Lark, logger
logger.setLevel(logging.DEBUG)
collision_grammar = '''
start: as as
as: a*
a: "a"
'''
p = Lark(collision_grammar, parser='lalr', debug=True)
# Shift/Reduce conflict for terminal A: (resolving as shift)
# * <as : >
# Shift/Reduce conflict for terminal A: (resolving as shift)
# * <as : __as_star_0>๐ Strict-Mode ๐
Lark, by default, accepts grammars with unresolved Shift/Reduce collisions (which it always resolves to shift), and regex collisions.
Strict-mode allows users to validate that their grammars don't contain these collisions.
When Lark is initialized with strict=True, it raises an exception on any Shift/Reduce or regex collision.
If interegular isn't installed, an exception is thrown. ๐ซ
When using strict-mode, users will be expected to resolve their collisions manually:
- To resolve Shift/Reduce collisions, adjust the priority weights of the rules involved, until there are no more collisions.
- To resolve regex collisions, change the involved regexes so that they can no longer both match the same input (Lark provides an example).
Strict-mode only applies to LALR for now.
from lark import Lark
collision_grammar = '''
start: as as
as: a*
a: "a"
'''
p = Lark(collision_grammar, parser='lalr', strict=True)
# Traceback (most recent call last):
# ...
# lark.exceptions.GrammarError: Shift/Reduce conflict for terminal A. [strict-mode]๐ง Tools ๐ง
Stand-alone parser
Lark can generate a stand-alone LALR(1) parser from a grammar.
The resulting module provides the same interface as Lark, but with a fixed grammar, and reduced functionality.
Run using:
python -m lark.tools.standaloneFor a play-by-play, read the tutorial (http://blog.erezsh.com/create-a-stand-alone-lalr1-parser-in-python/).
Import Nearley.js grammars
It is possible to import Nearley grammars into Lark. The Javascript code is translated using Js2Py.
See the tools page (<>) for more information.
๐ HOW TO DEVELOP LARK - GUIDE ๐
okay bestie, you wanna help out?? we are SO obsessed with that. there are literally so many ways you can pitch in:
- Help solve issues
- Improve the documentation
- Write new grammars for Lark's library
- Write a blog post introducing Lark to your audience
- Port Lark to another language
- Help with code development
If you're interested in taking one of these on, contact us on Gitter (https://gitter.im/lark-parser/Lobby) or Github Discussion (https://github.com/lark-parser/lark/discussions), and we will provide more details and assist you in the process. no pressure, we're nice ๐
Code Style
Lark does not follow a predefined code style. We accept any code style that makes sense, as long as it's Pythonic and easy to read. that's it, that's the whole policy. slay โจ
Unit Tests
Lark comes with an extensive set of tests. Many of the tests will run several times โ once for each parser configuration. (iconic behaviour honestly)
To run the tests, just go to the lark project root, and run the command:
python -m testsor
pypy -m testsFor a list of supported interpreters, you can consult the tox.ini file.
You can also run a single unittest using its class and method name, for example:
## test_package test_class_name.test_function_name
python -m tests TestLalrBasic.test_keep_all_tokenstox
To run all Unit Tests with tox, install tox and Python 2.7 up to the latest python interpreter supported (consult the file tox.ini). Then, run the command tox on the root of this project (where the main setup.py file is on).
And, for example, if you would like to only run the Unit Tests for Python version 2.7, you can run the command tox -e py27
pytest
You can also run the tests using pytest:
pytest testsUsing setup.py
Another way to run the tests is using setup.py:
python setup.py testBuilding the Documentation
To build the documentation:
cd docs/
pip install -r requirements.txt
make htmlTo review the result, open the built HTML files under _build/html/ in your browser. go look at your own handiwork, you earned it ๐
๐ RECIPES ๐
A collection of recipes to use Lark and its various features. little grab-bag of tricks, iykyk ๐ง
Use a transformer to parse integer tokens
Transformers are the common interface for processing matched rules and tokens.
They can be used during parsing for better performance. (yes, during. that's the whole flex.)
from lark import Lark, Transformer
class T(Transformer):
def INT(self, tok):
"Convert the value of `tok` from string to int, while maintaining line number & column."
return tok.update(value=int(tok))
parser = Lark("""
start: INT*
%import common.INT
%ignore " "
""", parser="lalr", transformer=T())
print(parser.parse('3 14 159'))Prints out:
Tree(start, [Token(INT, 3), Token(INT, 14), Token(INT, 159)])Collect all comments with lexer_callbacks
lexer_callbacks can be used to interface with the lexer as it generates tokens.
It accepts a dictionary of the form
{TOKEN_TYPE: callback}Where callback is of type f(Token) -> Token
It only works with the basic and contextual lexers. โ ๏ธ
This has the same effect of using a transformer, but can also process ignored tokens. (yeah. even the ones you told Lark to ignore.)
from lark import Lark
comments = []
parser = Lark("""
start: INT*
COMMENT: /#.*/
%import common (INT, WS)
%ignore COMMENT
%ignore WS
""", parser="lalr", lexer_callbacks={'COMMENT': comments.append})
parser.parse("""
1 2 3 # hello
# world
4 5 6
""")
print(comments)Prints out:
[Token(COMMENT, '# hello'), Token(COMMENT, '# world')]Note: We don't have to return a token, because comments are ignored ๐ซถ
CollapseAmbiguities
Parsing ambiguous texts with earley and ambiguity='explicit' produces a single tree with _ambig nodes to mark where the ambiguity occurred.
However, it's sometimes more convenient instead to work with a list of all possible unambiguous trees.
Lark provides a utility transformer for that purpose:
from lark import Lark, Tree, Transformer
from lark.visitors import CollapseAmbiguities
grammar = """
!start: x y
!x: "a" "b"
| "ab"
| "abc"
!y: "c" "d"
| "cd"
| "d"
"""
parser = Lark(grammar, ambiguity='explicit')
t = parser.parse('abcd')
for x in CollapseAmbiguities().transform(t):
print(x.pretty())This prints out:
start
x
a
b
y
c
d
start
x ab
y cd
start
x abc
y dWhile convenient, this should be used carefully, as highly ambiguous trees will soon create an exponential explosion of such unambiguous derivations. ๐ญ like genuinely, do not do this on a wild grammar and then act surprised.
Keeping track of parents when visiting
The following visitor assigns a parent attribute for every node in the tree.
If your tree nodes aren't unique (if there is a shared Tree instance), the assert will fail.
class Parent(Visitor):
def __default__(self, tree):
for subtree in tree.children:
if isinstance(subtree, Tree):
assert not hasattr(subtree, 'parent')
subtree.parent = proxy(tree)Unwinding VisitError after a transformer/visitor exception
Errors that happen inside visitors and transformers get wrapped inside a VisitError exception.
This can often be inconvenient, if you wish the actual error to propagate upwards, or if you want to catch it.
But, it's easy to unwrap it at the point of calling the transformer, by catching it and raising the VisitError.orig_exc attribute.
For example:
from lark import Lark, Transformer
from lark.visitors import VisitError
tree = Lark('start: "a"').parse('a')
class T(Transformer):
def start(self, x):
raise KeyError("Original Exception")
t = T()
try:
print( t.transform(tree))
except VisitError as e:
raise e.orig_excAdding a Progress Bar to Parsing with tqdm
Parsing large files can take a long time, even with the parser='lalr' option. To make this process more user-friendly, it's useful to add a progress bar. One way to achieve this is to use the InteractiveParser to display each token as it is processed. In this example, we use tqdm (https://github.com/tqdm/tqdm), but it should be easy to adapt to other kinds of progress bars. ๐
from tqdm import tqdm
def parse_with_progress(parser: Lark, text: str, start=None):
last = 0
progress = tqdm(total=len(text))
pi = parser.parse_interactive(text, start=start)
for token in pi.iter_parse():
if token.end_pos is not None:
progress.update(token.end_pos - last)
last = token.end_pos
return pi.resume_parse() # Finish up and get the resultKeep in mind that this implementation relies on the InteractiveParser and, therefore, only works with the LALR(1) parser, and not Earley. ๐ซ
Parsing a Language with Significant Indentation
If your grammar needs to support significant indentation (e.g. Python, YAML), you will need to use the Indenter class. Take a look at the indented tree example (examples/indented_tree.html) as well as the Python grammar (https://github.com/lark-parser/lark/blob/master/lark/grammars/python.lark) for inspiration.
๐ EXAMPLES FOR LARK ๐
How to run the examples:
After cloning the repo, open the terminal into the root directory of the project, and run the following:
[lark]$ python -m examples.<name_of_example>For example, the following will parse all the Python files in the standard library of your local installation:
[lark]$ python -m examples.advanced.python_parserBeginner Examples
Parsing Indentation
A demonstration of parsing indentation ("whitespace significant" language) and the usage of the Indenter class.
Since indentation is context-sensitive, a postlex stage is introduced to manufacture INDENT/DEDENT tokens.
It is crucial for the indenter that the NL_type matches the spaces (and tabs) after the newline.
If your whitespace-significant grammar supports comments, then NL_type must match those comments too. Otherwise, comments that appear in the middle of a line will confuse Lark (https://github.com/lark-parser/lark/issues/863). ๐
from lark import Lark
from lark.indenter import Indenter
tree_grammar = r"""
%import common.CNAME -> NAME
%import common.WS_INLINE
%import common.SH_COMMENT
%ignore WS_INLINE
%ignore SH_COMMENT
%declare _INDENT _DEDENT
?start: _NL* tree
tree: NAME _NL [_INDENT tree+ _DEDENT]
_NL: (/\r?\n[\t ]*/ | SH_COMMENT)+
"""
class TreeIndenter(Indenter):
NL_type = '_NL'
OPEN_PAREN_types = []
CLOSE_PAREN_types = []
INDENT_type = '_INDENT'
DEDENT_type = '_DEDENT'
tab_len = 8
parser = Lark(tree_grammar, parser='lalr', postlex=TreeIndenter())
test_tree = """
a
# check this comment out
b
c
d
e
f
g
"""
def test():
print(parser.parse(test_tree).pretty())
if __name__ == '__main__':
test()Lark Grammar
A reference implementation of the Lark grammar (using LALR(1))
import lark
from pathlib import Path
examples_path = Path(__file__).parent
lark_path = Path(lark.__file__).parent
parser = lark.Lark.open(lark_path / 'grammars/lark.lark', rel_to=__file__, parser="lalr")
grammar_files = [
examples_path / 'advanced/python2.lark',
examples_path / 'relative-imports/multiples.lark',
examples_path / 'relative-imports/multiple2.lark',
examples_path / 'relative-imports/multiple3.lark',
examples_path / 'tests/no_newline_at_end.lark',
examples_path / 'tests/negative_priority.lark',
examples_path / 'standalone/json.lark',
lark_path / 'grammars/common.lark',
lark_path / 'grammars/lark.lark',
lark_path / 'grammars/unicode.lark',
lark_path / 'grammars/python.lark',
]
def test():
for grammar_file in grammar_files:
tree = parser.parse(open(grammar_file).read())
print("All grammars parsed successfully")
if __name__ == '__main__':
test()Handling Ambiguity
A demonstration of ambiguity
This example shows how to use get explicit ambiguity from Lark's Earley parser.
import sys
from lark import Lark, treegrammar = """
sentence: noun verb noun -> simple
| noun verb "like" noun -> comparative
noun: adj? NOUN
verb: VERB
adj: ADJ
NOUN: "flies" | "bananas" | "fruit"
VERB: "like" | "flies"
ADJ: "fruit"
%import common.WS
%ignore WS
"""
parser = Lark(grammar, start='sentence', ambiguity='explicit')
sentence = 'fruit flies like bananas'
def make_png(filename):
tree.pydot__tree_to_png( parser.parse(sentence), filename)
def make_dot(filename):
tree.pydot__tree_to_dot( parser.parse(sentence), filename)
if __name__ == '__main__':
print(parser.parse(sentence).pretty())
# make_png(sys.argv[1])
# make_dot(sys.argv[1])
# Output:
#
# _ambig
# comparative
# noun fruit
# verb flies
# noun bananas
# simple
# noun
# fruit
# flies
# verb like
# noun bananas
#
# (or view a nicer version at "./fruitflies.png")Basic calculator ๐งฎโจ
A simple example of a REPL calculator
bestie this one is SO cute, it's a whole calculator with variables in it, like it just remembers stuff for you ๐ this example shows how to write a basic calculator with variables.
from lark import Lark, Transformer, v_args
try:
input = raw_input # For Python2 compatibility
except NameError:
pass
calc_grammar = """
?start: sum
| NAME "=" sum -> assign_var
?sum: product
| sum "+" product -> add
| sum "-" product -> sub
?product: atom
| product "*" atom -> mul
| product "/" atom -> div
?atom: NUMBER -> number
| "-" atom -> neg
| NAME -> var
| "(" sum ")"
%import common.CNAME -> NAME
%import common.NUMBER
%import common.WS_INLINE
%ignore WS_INLINE
"""
@v_args(inline=True) # Affects the signatures of the methods
class CalculateTree(Transformer):
from operator import add, sub, mul, truediv as div, neg
number = float
def __init__(self):
self.vars = {}
def assign_var(self, name, value):
self.vars[name] = value
return value
def var(self, name):
try:
return self.vars[name]
except KeyError:
raise Exception("Variable not found: %s" % name)
calc_parser = Lark(calc_grammar, parser='lalr', transformer=CalculateTree())
calc = calc_parser.parse
def main():
while True:
try:
s = input('> ')
except EOFError:
break
print(calc(s))
def test():
print(calc("a = 1+2"))
print(calc("1+a*-3"))
if __name__ == '__main__':
# test()
main()Turtle DSL ๐ข๐
ngl this one is so iconic โ it's a whole lil LOGO-style toy language that drives Python's turtle, and it comes with its own interpreter ๐
try:
input = raw_input # For Python2 compatibility
except NameError:
pass
import turtle
from lark import Lark
turtle_grammar = """
start: instruction+
instruction: MOVEMENT NUMBER -> movement
| "c" COLOR [COLOR] -> change_color
| "fill" code_block -> fill
| "repeat" NUMBER code_block -> repeat
code_block: "{" instruction+ "}"
MOVEMENT: "f"|"b"|"l"|"r"
COLOR: LETTER+
%import common.LETTER
%import common.INT -> NUMBER
%import common.WS
%ignore WS
"""
parser = Lark(turtle_grammar)
def run_instruction(t):
if t.data == 'change_color':
turtle.color(*t.children) # We just pass the color names as-is
elif t.data == 'movement':
name, number = t.children
{ 'f': turtle.fd,
'b': turtle.bk,
'l': turtle.lt,
'r': turtle.rt, }[name](int(number))
elif t.data == 'repeat':
count, block = t.children
for i in range(int(count)):
run_instruction(block)
elif t.data == 'fill':
turtle.begin_fill()
run_instruction(t.children[0])
turtle.end_fill()
elif t.data == 'code_block':
for cmd in t.children:
run_instruction(cmd)
else:
raise SyntaxError('Unknown instruction: %s' % t.data)
def run_turtle(program):
parse_tree = parser.parse(program)
for inst in parse_tree.children:
run_instruction(inst)
def main():
while True:
code = input('> ')
try:
run_turtle(code)
except Exception as e:
print(e)
def test():
text = """
c red yellow
fill { repeat 36 {
f200 l170
}}
"""
run_turtle(text)
if __name__ == '__main__':
# test()
main()Simple JSON Parser ๐
okay so this one? it's short, it's clear, and it outperforms every other parser (that's written in Python). huge slay ๐ for an explanation, check out the JSON parser tutorial at /docs/json_tutorial.md
import sys
from lark import Lark, Transformer, v_args
json_grammar = r"""
?start: value
?value: object
| array
| string
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
array : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair : string ":" value
string : ESCAPED_STRING
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""
class TreeToJson(Transformer):
@v_args(inline=True)
def string(self, s):
return s[1:-1].replace('\\"', '"')
array = list
pair = tuple
object = dict
number = v_args(inline=True)(float)
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False
### Create the JSON parser with Lark, using the Earley algorithm
# json_parser = Lark(json_grammar, parser='earley', lexer='basic')
# def parse(x):
# return TreeToJson().transform(json_parser.parse(x))
### Create the JSON parser with Lark, using the LALR algorithm
json_parser = Lark(json_grammar, parser='lalr',
# Using the basic lexer isn't required, and isn't usually recommended.
# But, it's good enough for JSON, and it's slightly faster.
lexer='basic',
# Disabling propagate_positions and placeholders slightly improves speed
propagate_positions=False,
maybe_placeholders=False,
# Using an internal transformer is faster and more memory efficient
transformer=TreeToJson())
parse = json_parser.parse
def test():
test_json = '''
{
"empty_object" : {},
"empty_array" : [],
"booleans" : { "YES" : true, "NO" : false },
"numbers" : [ 0, 1, -2, 3.3, 4.4e5, 6.6e-7 ],
"strings" : [ "This", [ "And" , "That", "And a \\"b" ] ],
"nothing" : null
}
'''
j = parse(test_json)
print(j)
import json
assert j == json.loads(test_json)
if __name__ == '__main__':
# test()
with open(sys.argv[1]) as f:
print(parse(f.read()))Advanced Examples ๐ฅ
GRAMMAR COMPOSITION ๐งฉ
this one shows how to do grammar composition in Lark, by creating a whole new file format that lets both CSV and JSON co-exist together โ besties living in harmony, we love to see it ๐ธ
and the way that it works?? by using namespaces, Lark grammars and their transformers can be fully reused โ they don't need to care if their grammar is used directly, or being imported, or who is doing the importing. literally so independent of them ๐
see main.py <https://github.com/lark-parser/lark/blob/master/examples/composition/main.py> for more details.
EXAMPLE GRAMMARS ๐
this directory is a collection of lark grammars, taken from real world projects. no cap, real ones ๐ซถ
Verilog<https://github.com/lark-parser/lark/blob/master/examples/grammars/verilog.lark> - Taken from <https://github.com/circuitgraph/circuitgraph/blob/main/circuitgraph/parsing/verilog.lark>
STANDALONE EXAMPLE ๐
to initialize, cd to this folder, and run:
./create_standalone.shor:
python -m lark.tools.standalone json.lark > json_parser.pythen run using:
python json_parser_main.py <path-to.json>Advanced Examples ๐
LALR's contextual lexer ๐
this example demonstrates the power of LALR's contextual lexer, by parsing a toy configuration language. yes ma'am, we love a lexer with context โจ
so here's the vibe: the terminals NAME and VALUE overlap. they can match the same input. a basic lexer would arbitrarily choose one over the other, based on priority, which would lead to a (confusing) parse error ๐ญ however, due to the unambiguous structure of the grammar, Lark's LALR(1) algorithm knows which one of them to expect at each point during the parse. the lexer then only matches the tokens that the parser expects. the result is a correct parse, something that is impossible with a regular lexer. ate, no crumbs ๐
another approach is to use the Earley algorithm. it will handle more cases than the contextual lexer, but at the cost of performance. see examples/conf_earley.py for an example of that approach.
from lark import Lark
parser = Lark(r"""
start: _NL? section+
section: "[" NAME "]" _NL item+
item: NAME "=" VALUE? _NL
NAME: /\w/+
VALUE: /./+
%import common.NEWLINE -> _NL
%import common.WS_INLINE
%ignore WS_INLINE
""", parser="lalr")
sample_conf = """
[bla]
a=Hello
this="that",4
empty=
"""
print(parser.parse(sample_conf).pretty())Templates ๐
this example shows how to use Lark's templates to achieve cleaner grammars. we stan a tidy grammar, fr fr ๐
from lark import Larkgrammar = r"""
start: list | dict
list: "[" _seperated{atom, ","} "]"
dict: "{" _seperated{key_value, ","} "}"
key_value: atom ":" atom
_seperated{x, sep}: x (sep x)* // Define a sequence of 'x sep x sep x ...'
atom: NUMBER | ESCAPED_STRING
%import common (NUMBER, ESCAPED_STRING, WS)
%ignore WS
"""
parser = Lark(grammar)
print(parser.parse('[1, "a", 2]'))
print(parser.parse('{"a": 2, "b": 6}'))ok that's the little grammar moment ๐ anyway. next up, the actual example files, babes โ buckle in ๐ซถ
Earley's dynamic lexer ๐ฆ
ok so this one is demonstrating the power of Earley's dynamic lexer on a lil toy configuration language. adorable.
and here's why it's a whole thing: using a lexer for configuration files is tricky because the values don't have to be surrounded by delimiters. so if you try to use a basic lexer for this? it just <highkey>won't work</highkey>. not a vibe. ๐
in this example we use a dynamic lexer and let the Earley parser resolve the ambiguity. main character behaviour fr.
ngl there's also another approach: use the contextual lexer with LALR. it's less powerful than Earley, but it can handle some ambiguity when lexing and it's much faster. for an example of that approach, check out examples/conf_lalr.py โจ
from lark import Lark
parser = Lark(r"""
start: _NL? section+
section: "[" NAME "]" _NL item+
item: NAME "=" VALUE? _NL
NAME: /\w/+
VALUE: /./+
%import common.NEWLINE -> _NL
%import common.WS_INLINE
%ignore WS_INLINE
""", parser="earley")
def test():
sample_conf = """
[bla]
a=Hello
this="that",4
empty=
"""
r = parser.parse(sample_conf)
print (r.pretty())
if __name__ == '__main__':
test()Error handling using an interactive parser ๐
bestie this one is demonstrating error handling using an interactive parser in LALR โ yes, handling, we don't just cry about it (well, we do, but productively).
so when the parser runs into an UnexpectedToken exception, it makes an interactive parser with the current parse-state, and lets you control how to proceed step-by-step. and once you've achieved the correct parse-state, you can resume the run by returning True. she's interactive, she's cooperative, she's serving โจ
from lark import Token
from _json_parser import json_parser
def ignore_errors(e):
if e.token.type == 'COMMA':
# Skip comma
return True
elif e.token.type == 'SIGNED_NUMBER':
# Try to feed a comma and retry the number
e.interactive_parser.feed_token(Token('COMMA', ','))
e.interactive_parser.feed_token(e.token)
return True
# Unhandled error. Will stop parse and raise exception
return False
def main():
s = "[0 1, 2,, 3,,, 4, 5 6 ]"
res = json_parser.parse(s, on_error=ignore_errors)
print(res) # prints [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
main()Reconstruct a JSON ๐ง
this one is demonstrating the experimental text-reconstruction feature. experimental, so treat her gently bestie ๐
so Reconstructor takes a parse tree (already filtered from punctuation, obviously, of course) and reconstructs it back into correct text that can be parsed correctly. super useful for making "hooks" to alter data before handing it off to other parsers. and you can also use it to generate samples from scratch ๐
import json
from lark import Lark
from lark.reconstruct import Reconstructor
from _json_parser import json_grammar
test_json = '''
{
"empty_object" : {},
"empty_array" : [],
"booleans" : { "YES" : true, "NO" : false },
"numbers" : [ 0, 1, -2, 3.3, 4.4e5, 6.6e-7 ],
"strings" : [ "This", [ "And" , "That", "And a \\"b" ] ],
"nothing" : null
}
'''
def test_earley():
json_parser = Lark(json_grammar, maybe_placeholders=False)
tree = json_parser.parse(test_json)
new_json = Reconstructor(json_parser).reconstruct(tree)
print (new_json)
print (json.loads(new_json) == json.loads(test_json))
def test_lalr():
json_parser = Lark(json_grammar, parser='lalr', maybe_placeholders=False)
tree = json_parser.parse(test_json)
new_json = Reconstructor(json_parser).reconstruct(tree)
print (new_json)
print (json.loads(new_json) == json.loads(test_json))
test_earley()
test_lalr()Custom lexer ๐
this one is demonstrating using a custom lexer to parse a non-textual stream of data. yes, non-textual. we're going off-script, we're breaking the rules, we're iconic.
so you can use a custom lexer to tokenize text when the lexers Lark offers are too slow or not flexible enough. and you can also use it โ like in this example โ to tokenize streams of objects. not me making a lexer that eats python objects for breakfast ๐ญ๐ฅ
from lark import Lark, Transformer, v_args
from lark.lexer import Lexer, Token
class TypeLexer(Lexer):
def __init__(self, lexer_conf):
pass
def lex(self, data):
for obj in data:
if isinstance(obj, int):
yield Token('INT', obj)
elif isinstance(obj, (type(''), type(u''))):
yield Token('STR', obj)
else:
raise TypeError(obj)
parser = Lark("""
start: data_item+
data_item: STR INT*
%declare STR INT
""", parser='lalr', lexer=TypeLexer)
class ParseToDict(Transformer):
@v_args(inline=True)
def data_item(self, name, *numbers):
return name.value, [n.value for n in numbers]
start = dict
def test():
data = ['alice', 1, 27, 3, 'bob', 4, 'carrie', 'dan', 8, 6]
print(data)
tree = parser.parse(data)
res = ParseToDict().transform(tree)
print('-->')
print(res) # prints {'alice': [1, 27, 3], 'bob': [4], 'carrie': [], 'dan': [8, 6]}
if __name__ == '__main__':
test()Transform a Forest ๐
and this one! demonstrating how to subclass TreeForestTransformer to directly transform a SPPF. a whole forest, transformed. the way that just works โจ
from lark import Lark
from lark.parsers.earley_forest import TreeForestTransformer, handles_ambiguity, Discard
class CustomTransformer(TreeForestTransformer):
@handles_ambiguity
def sentence(self, trees):
return next(tree for tree in trees if tree.data == 'simple')
def simple(self, children):
children.append('.')
return self.tree_class('simple', children)
def adj(self, children):
return Discard
def __default_token__(self, token):
return token.capitalize()
grammar = """
sentence: noun verb noun -> simple
| noun verb "like" noun -> comparative
noun: adj? NOUN
verb: VERB
adj: ADJ
NOUN: "flies" | "bananas" | "fruit"
VERB: "like" | "flies"
ADJ: "fruit"
%import common.WS
%ignore WS
"""
parser = Lark(grammar, start='sentence', ambiguity='forest')
sentence = 'fruit flies like bananas'
forest = parser.parse(sentence)
tree = CustomTransformer(resolve_ambiguity=False).transform(forest)
print(tree.pretty())
# Output:
#
# simple
# noun Flies
# verb Like
# noun Bananas
# .
#Simple JSON Parser ๐
bestie. this one. the code is short and clear, and it outperforms every other parser (that's written in Python, anyway). for an explanation, check out the JSON parser tutorial at /docs/json_tutorial.md ๐
(and this is here for use by the other examples, so it's doing community service basically)
from lark import Lark, Transformer, v_args
json_grammar = r"""
?start: value
?value: object
| array
| string
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
array : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair : string ":" value
string : ESCAPED_STRING
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""
class TreeToJson(Transformer):
@v_args(inline=True)
def string(self, s):
return s[1:-1].replace('\\"', '"')
array = list
pair = tuple
object = dict
number = v_args(inline=True)(float)
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False
### Create the JSON parser with Lark, using the LALR algorithm
json_parser = Lark(json_grammar, parser='lalr',
# Using the basic lexer isn't required, and isn't usually recommended.
# But, it's good enough for JSON, and it's slightly faster.
lexer='basic',
# Disabling propagate_positions and placeholders slightly improves speed
propagate_positions=False,
maybe_placeholders=False,
# Using an internal transformer is faster and more memory efficient
transformer=TreeToJson())Custom SPPF Prioritizer ๐ฆ
last one for this chunk, and she's a whole mood: demonstrating how to subclass ForestVisitor to make a custom SPPF node prioritizer, to be used in conjunction with TreeForestTransformer ๐
so, the plot: our prioritizer is gonna count the number of descendants of a node that are tokens. and by negating that count, our prioritizer prefers nodes with fewer token descendants. which means we choose the more specific parse. that's it, that's the tea ๐
from lark import Lark
from lark.parsers.earley_forest import ForestVisitor, TreeForestTransformer
class TokenPrioritizer(ForestVisitor):
def visit_symbol_node_in(self, node):
# visit the entire forest by returning node.children
return node.children
def visit_packed_node_in(self, node):
return node.children
def visit_symbol_node_out(self, node):
priority = 0
for child in node.children:
# Tokens do not have a priority attribute
# count them as -1
priority += getattr(child, 'priority', -1)
node.priority = priority
def visit_packed_node_out(self, node):
priority = 0
for child in node.children:
priority += getattr(child, 'priority', -1)
node.priority = priority
def on_cycle(self, node, path):
raise Exception("Oops, we encountered a cycle.")grammar = """
start: hello " " world | hello_world
hello: "Hello"
world: "World"
hello_world: "Hello World"
"""
parser = Lark(grammar, parser='earley', ambiguity='forest')
forest = parser.parse("Hello World")
print("Default prioritizer:")
tree = TreeForestTransformer(resolve_ambiguity=True).transform(forest)
print(tree.pretty())
forest = parser.parse("Hello World")
print("Custom prioritizer:")
tree = TreeForestTransformer(resolve_ambiguity=True, prioritizer=TokenPrioritizer()).transform(forest)
print(tree.pretty())
# Output:
#
# Default prioritizer:
# start
# hello Hello
#
# world World
#
# Custom prioritizer:
# start
# hello_world Hello WorldNo cap, two parses, two completely different vibes, and the prioritizer is the only thing that changed. That's the whole plot twist. ๐
Python 3 to Python 2 converter (tree templates)
Bestie, this one is a glow-up demo. It shows you how to translate between two trees using tree templates. It parses Python 3, translates it into a Python 2 AST, and then spits it back out as Python 2 code. Time travel but make it syntax. ๐ฐ๏ธโจ
Uses reconstruct_python.py for generating the final Python 2 code.
from lark import Lark
from lark.tree_templates import TemplateConf, TemplateTranslator
from lark.indenter import PythonIndenter
from reconstruct_python import PythonReconstructor
#
# 1. Define a Python parser that also accepts template vars in the code (in the form of $var)
#
TEMPLATED_PYTHON = r"""
%import python (single_input, file_input, eval_input, atom, var, stmt, expr, testlist_star_expr, _NEWLINE, _INDENT, _DEDENT, COMMENT, NAME)
%extend atom: TEMPLATE_NAME -> var
TEMPLATE_NAME: "$" NAME
?template_start: (stmt | testlist_star_expr _NEWLINE)
%ignore /[\t \f]+/ // WS
%ignore /\\[\t \f]*\r?\n/ // LINE_CONT
%ignore COMMENT
"""
parser = Lark(TEMPLATED_PYTHON, parser='lalr', start=['single_input', 'file_input', 'eval_input', 'template_start'], postlex=PythonIndenter(), maybe_placeholders=False)
def parse_template(s):
return parser.parse(s + '\n', start='template_start')
def parse_code(s):
return parser.parse(s + '\n', start='file_input')
#
# 2. Define translations using templates (each template code is parsed to a template tree)
#
pytemplate = TemplateConf(parse=parse_template)
translations_3to2 = {
'yield from $a':
'for _tmp in $a: yield _tmp',
'raise $e from $x':
'raise $e',
'$a / $b':
'float($a) / $b',
}
translations_3to2 = {pytemplate(k): pytemplate(v) for k, v in translations_3to2.items()}
#
# 3. Translate and reconstruct Python 3 code into valid Python 2 code
#
python_reconstruct = PythonReconstructor(parser)
def translate_py3to2(code):
tree = parse_code(code)
tree = TemplateTranslator(translations_3to2).translate(tree)
return python_reconstruct.reconstruct(tree)
#
# Test Code
#
_TEST_CODE = '''
if a / 2 > 1:
yield from [1,2,3]
else:
raise ValueError(a) from e
'''
def test():
print(_TEST_CODE)
print(' -----> ')
print(translate_py3to2(_TEST_CODE))
if __name__ == '__main__':
test()Three lil steps, that's it: define the parser, define the translations as templates, translate + reconstruct. Breaking it down: $a / $b becomes float($a) / $b, yield from $a becomes a for loop, and that whole raise ... from ... situation gets flattened because Python 2 doesn't know her. ๐
Grammar-complete Python Parser
A fully-working Python 2 & 3 parser (but not production ready yet!) ๐จ
This example demonstrates usage of the included Python grammars.
import sys
import os, os.path
from io import open
import glob, time
from lark import Lark
from lark.indenter import PythonIndenter
kwargs = dict(postlex=PythonIndenter(), start='file_input')
# Official Python grammar by Lark
python_parser3 = Lark.open_from_package('lark', 'python.lark', ['grammars'], parser='lalr', **kwargs)
# Local Python2 grammar
python_parser2 = Lark.open('python2.lark', rel_to=__file__, parser='lalr', **kwargs)
python_parser2_earley = Lark.open('python2.lark', rel_to=__file__, parser='earley', lexer='basic', **kwargs)
try:
xrange
except NameError:
chosen_parser = python_parser3
else:
chosen_parser = python_parser2
def _read(fn, *args):
kwargs = {'encoding': 'iso-8859-1'}
with open(fn, *args, **kwargs) as f:
return f.read()
def _get_lib_path():
if os.name == 'nt':
if 'PyPy' in sys.version:
return os.path.join(sys.base_prefix, 'lib-python', sys.winver)
else:
return os.path.join(sys.base_prefix, 'Lib')
else:
return [x for x in sys.path if x.endswith('%s.%s' % sys.version_info[:2])][0]
def test_python_lib():
path = _get_lib_path()
start = time.time()
files = glob.glob(path+'/*.py')
total_kb = 0
for f in files:
r = _read(os.path.join(path, f))
kb = len(r) / 1024
print( '%s -\t%.1f kb' % (f, kb))
chosen_parser.parse(r + '\n')
total_kb += kb
end = time.time()
print( "test_python_lib (%d files, %.1f kb), time: %.2f secs"%(len(files), total_kb, end-start) )
def test_earley_equals_lalr():
path = _get_lib_path()
files = glob.glob(path+'/*.py')
for f in files:
print( f )
tree1 = python_parser2.parse(_read(os.path.join(path, f)) + '\n')
tree2 = python_parser2_earley.parse(_read(os.path.join(path, f)) + '\n')
assert tree1 == tree2
if __name__ == '__main__':
test_python_lib()
# test_earley_equals_lalr()
# python_parser3.parse(_read(sys.argv[1]) + '\n')Obsessed with the little try: xrange sniff test โ it's literally checking if she's on Python 2 by looking for a function that got yeeted in Python 3. Iconic detective work. ๐ต๏ธโโ๏ธ๐
Creating an AST from the parse tree
This example demonstrates how to transform a parse-tree into an AST using lark.ast_utils.
create_transformer() collects every subclass of Ast subclass from the module, and creates a Lark transformer that builds the AST with no extra code.
Heads up bestie: this example only works with Python 3. ๐โจ
import sys
from typing import List
from dataclasses import dataclass
from lark import Lark, ast_utils, Transformer, v_args
from lark.tree import Meta
this_module = sys.modules[__name__]
#
# Define AST
#
class _Ast(ast_utils.Ast):
# This will be skipped by create_transformer(), because it starts with an underscore
pass
class _Statement(_Ast):
# This will be skipped by create_transformer(), because it starts with an underscore
pass
@dataclass
class Value(_Ast, ast_utils.WithMeta):
"Uses WithMeta to include line-number metadata in the meta attribute"
meta: Meta
value: object
@dataclass
class Name(_Ast):
name: str
@dataclass
class CodeBlock(_Ast, ast_utils.AsList):
# Corresponds to code_block in the grammar
statements: List[_Statement]
@dataclass
class If(_Statement):
cond: Value
then: CodeBlock
@dataclass
class SetVar(_Statement):
# Corresponds to set_var in the grammar
name: str
value: Value
@dataclass
class Print(_Statement):
value: Value
class ToAst(Transformer):
# Define extra transformation functions, for rules that don't correspond to an AST class.
def STRING(self, s):
# Remove quotation marks
return s[1:-1]
def DEC_NUMBER(self, n):
return int(n)
@v_args(inline=True)
def start(self, x):
return x
#
# Define Parser
#
parser = Lark("""
start: code_block
code_block: statement+
?statement: if | set_var | print
if: "if" value "{" code_block "}"
set_var: NAME "=" value ";"
print: "print" value ";"
value: name | STRING | DEC_NUMBER
name: NAME
%import python (NAME, STRING, DEC_NUMBER)
%import common.WS
%ignore WS
""",
parser="lalr",
)
transformer = ast_utils.create_transformer(this_module, ToAst())
def parse(text):
tree = parser.parse(text)
return transformer.transform(tree)
#
# Test
#
if __name__ == '__main__':
print(parse("""
a = 1;
if a {
print "a is 1";
a = 2;
}
"""))The underscore rule is the cutest part โ _Ast and _Statement start with an underscore, so create_transformer() skips them on purpose. They're the base classes just vibing in the background, not the main characters. ๐
Example-Driven Error Reporting
A demonstration of example-driven error reporting with the Earley parser.
See also: error_reporting_lalr.py ๐ธ
from lark import Lark, UnexpectedInput
from _json_parser import json_grammar # Using the grammar from the json_parser example
json_parser = Lark(json_grammar)
class JsonSyntaxError(SyntaxError):
def __str__(self):
context, line, column = self.args
return '%s at line %s, column %s.\n\n%s' % (self.label, line, column, context)
class JsonMissingValue(JsonSyntaxError):
label = 'Missing Value'
class JsonMissingOpening(JsonSyntaxError):
label = 'Missing Opening'
class JsonMissingClosing(JsonSyntaxError):
label = 'Missing Closing'
class JsonMissingComma(JsonSyntaxError):
label = 'Missing Comma'
class JsonTrailingComma(JsonSyntaxError):
label = 'Trailing Comma'โฆand annnnd we're right back inside the function, bestie โ last part literally cut us off mid-code so we're just picking up the thread ๐
def parse(json_text):
try:
j = json_parser.parse(json_text)
except UnexpectedInput as u:
exc_class = u.match_examples(json_parser.parse, {
JsonMissingOpening: ['{"foo": ]}',
'{"foor": }}',
'{"foo": }'],
JsonMissingClosing: ['{"foo": [}',
'{',
'{"a": 1',
'[1'],
JsonMissingComma: ['[1 2]',
'[false 1]',
'["b" 1]',
'{"a":true 1:4}',
'{"a":1 1:4}',
'{"a":"b" 1:4}'],
JsonTrailingComma: ['[,]',
'[1,]',
'[1,2,]',
'{"foo":1,}',
'{"foo":false,"bar":true,}']
}, use_accepts=True)
if not exc_class:
raise
raise exc_class(u.get_context(json_text), u.line, u.column)
def test():
try:
parse('{"example1": "value"')
except JsonMissingClosing as e:
print(e)
try:
parse('{"example2": ] ')
except JsonMissingOpening as e:
print(e)
if __name__ == '__main__':
test()โจ Example-Driven Error Reporting
A lil demo of example-driven error reporting, but this time with the LALR parser. (See also: error_reporting_earley.py โ the girlies are sisters, not twins ๐ซถ)
from lark import Lark, UnexpectedInput
from _json_parser import json_grammar # Using the grammar from the json_parser example
json_parser = Lark(json_grammar, parser='lalr')
class JsonSyntaxError(SyntaxError):
def __str__(self):
context, line, column = self.args
return '%s at line %s, column %s.\n\n%s' % (self.label, line, column, context)
class JsonMissingValue(JsonSyntaxError):
label = 'Missing Value'
class JsonMissingOpening(JsonSyntaxError):
label = 'Missing Opening'
class JsonMissingClosing(JsonSyntaxError):
label = 'Missing Closing'
class JsonMissingComma(JsonSyntaxError):
label = 'Missing Comma'
class JsonTrailingComma(JsonSyntaxError):
label = 'Trailing Comma'
def parse(json_text):
try:
j = json_parser.parse(json_text)
except UnexpectedInput as u:
exc_class = u.match_examples(json_parser.parse, {
JsonMissingOpening: ['{"foo": ]}',
'{"foor": }}',
'{"foo": }'],
JsonMissingClosing: ['{"foo": [}',
'{',
'{"a": 1',
'[1'],
JsonMissingComma: ['[1 2]',
'[false 1]',
'["b" 1]',
'{"a":true 1:4}',
'{"a":1 1:4}',
'{"a":"b" 1:4}'],
JsonTrailingComma: ['[,]',
'[1,]',
'[1,2,]',
'{"foo":1,}',
'{"foo":false,"bar":true,}']
}, use_accepts=True)
if not exc_class:
raise
raise exc_class(u.get_context(json_text), u.line, u.column)
def test():
try:
parse('{"example1": "value"')
except JsonMissingClosing as e:
print(e)
try:
parse('{"example2": ] ')
except JsonMissingOpening as e:
print(e)
if __name__ == '__main__':
test()๐ช Reconstruct Python
okay this one is actually iconic, no exaggeration. It demonstrates how Lark's experimental text-reconstruction feature can recreate functional Python code from its parse-tree โ using just the correct grammar and a small formatter. Like, it parses Python and then writes Python back out ๐๐
from lark import Token, Lark
from lark.reconstruct import Reconstructor
from lark.indenter import PythonIndenter
# Official Python grammar by Lark
python_parser3 = Lark.open_from_package('lark', 'python.lark', ['grammars'],
parser='lalr', postlex=PythonIndenter(), start='file_input',
maybe_placeholders=False # Necessary for reconstructor
)
SPACE_AFTER = set(',+-*/~@<>="|:')
SPACE_BEFORE = (SPACE_AFTER - set(',:')) | set('\'')
def special(sym):
return Token('SPECIAL', sym.name)
def postproc(items):
stack = ['\n']
actions = []
last_was_whitespace = True
for item in items:
if isinstance(item, Token) and item.type == 'SPECIAL':
actions.append(item.value)
else:
if actions:
assert actions[0] == '_NEWLINE' and '_NEWLINE' not in actions[1:], actions
for a in actions[1:]:
if a == '_INDENT':
stack.append(stack[-1] + ' ' * 4)
else:
assert a == '_DEDENT'
stack.pop()
actions.clear()
yield stack[-1]
last_was_whitespace = True
if not last_was_whitespace:
if item[0] in SPACE_BEFORE:
yield ' '
yield item
last_was_whitespace = item[-1].isspace()
if not last_was_whitespace:
if item[-1] in SPACE_AFTER:
yield ' '
last_was_whitespace = True
yield "\n"
class PythonReconstructor:
def __init__(self, parser):
self._recons = Reconstructor(parser, {'_NEWLINE': special, '_DEDENT': special, '_INDENT': special})
def reconstruct(self, tree):
return self._recons.reconstruct(tree, postproc)
def test():
python_reconstructor = PythonReconstructor(python_parser3)
self_contents = open(__file__).read()
tree = python_parser3.parse(self_contents+'\n')
output = python_reconstructor.reconstruct(tree)
tree_new = python_parser3.parse(output)
print(tree.pretty())
print(tree_new.pretty())
# assert tree.pretty() == tree_new.pretty()
assert tree == tree_new
print(output)
if __name__ == '__main__':
test()๐ฆ Using lexer dynamic_complete
bestie sometimes your data is highly ambiguous, or just straight-up 'broken' in some sense ๐ญ. When you're running parser='earley' and lexer='dynamic_complete', Lark will be able parse just about anything โ as long as there is a valid way to generate it from the Grammar, including looking 'into' the Regexes. It's giving no rules, only vibes โจ
This examples shows how to parse a json input where the quotes have been replaced by underscores: {_foo_:{}, _bar_: [], _baz_: __} Notice that underscores might still appear inside strings, so a potentially valid reading of the above is: {"foo_:{}, _bar": [], "baz": ""}
from pprint import pprint
from lark import Lark, Tree, Transformer, v_args
from lark.visitors import Transformer_InPlace
GRAMMAR = r"""
%import common.SIGNED_NUMBER
%import common.WS_INLINE
%import common.NEWLINE
%ignore WS_INLINE
?start: value
?value: object
| array
| string
| SIGNED_NUMBER -> number
| "true" -> true
| "false" -> false
| "null" -> null
array : "[" (value ("," value)*)? "]"
object : "{" (pair ("," pair)*)? "}"
pair : string ":" value
string: STRING
STRING : ESCAPED_STRING
ESCAPED_STRING: QUOTE_CHAR _STRING_ESC_INNER QUOTE_CHAR
QUOTE_CHAR: "_"
_STRING_INNER: /.*/
_STRING_ESC_INNER: _STRING_INNER /(?<!\\)(\\\\)*?/
"""
def score(tree: Tree):
"""
Scores an option by how many children (and grand-children, and
grand-grand-children, ...) it has.
This means that the option with fewer large terminals gets selected
Between
object
pair
string _foo_
object
pair
string _bar_: [], _baz_
string __
and
object
pair
string _foo_
object
pair
string _bar_
array
pair
string _baz_
string __
this will give the second a higher score. (9 vs 13)
"""
return sum(len(t.children) for t in tree.iter_subtrees())
class RemoveAmbiguities(Transformer_InPlace):
"""
Selects an option to resolve an ambiguity using the score function above.
Scores each option and selects the one with the higher score, e.g. the one
with more nodes.
If there is a performance problem with the Tree having to many _ambig and
being slow and to large, this can instead be written as a ForestVisitor.
Look at the 'Custom SPPF Prioritizer' example.
"""
def _ambig(self, options):
return max(options, key=score)
class TreeToJson(Transformer):
"""
This is the same Transformer as the json_parser example.
"""
@v_args(inline=True)
def string(self, s):
return s[1:-1].replace('\\"', '"')
array = list
pair = tuple
object = dict
number = v_args(inline=True)(float)
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False
parser = Lark(GRAMMAR, parser='earley', ambiguity="explicit", lexer='dynamic_complete')
EXAMPLES = [
r'{_array_:[1,2,3]}',
r'{_abc_: _array must be of the following format [_1_, _2_, _3_]_}',
r'{_foo_:{}, _bar_: [], _baz_: __}',โฆand the list keeps going, so don't you dare close the tab โ there's more coming ๐
r'{_error_:_invalid_client_, _error_description_:_AADSTS7000215: Invalid '
r'client secret is provided.\r\nTrace ID: '
r'a0a0aaaa-a0a0-0a00-000a-00a00aaa0a00\r\nCorrelation ID: '
r'aa0aaa00-0aaa-0000-00a0-00000aaaa0aa\r\nTimestamp: 1997-10-10 00:00:00Z_, '
r'_error_codes_:[7000215], _timestamp_:_1997-10-10 00:00:00Z_, '
r'_trace_id_:_a0a0aaaa-a0a0-0a00-000a-00a00aaa0a00_, '
r'_correlation_id_:_aa0aaa00-0aaa-0000-00a0-00000aaaa0aa_, '
r'_error_uri_:_https://example.com_}',
]
for example in EXAMPLES:
tree = parser.parse(example)
tree = RemoveAmbiguities().transform(tree)
result = TreeToJson().transform(tree)
pprint(result)โจ Syntax Highlighting โจ
okay so picture this: you want a whole syntax-highlighted editor, with Qt, powered by Lark. yes bestie, that's a real thing and it's giving main character energy ๐ this example shows how to write a syntax-highlighted editor with Qt and Lark.
Requirements: PyQt5==5.15.8 QScintilla==2.13.4
import sys
import textwrap
from PyQt5.QtWidgets import QApplication
from PyQt5.QtGui import QColor, QFont, QFontMetrics
from PyQt5.Qsci import QsciScintilla
from PyQt5.Qsci import QsciLexerCustom
from lark import Lark
class LexerJson(QsciLexerCustom):
def __init__(self, parent=None):
super().__init__(parent)
self.create_parser()
self.create_styles()
def create_styles(self):
deeppink = QColor(249, 38, 114)
khaki = QColor(230, 219, 116)
mediumpurple = QColor(174, 129, 255)
mediumturquoise = QColor(81, 217, 205)
yellowgreen = QColor(166, 226, 46)
lightcyan = QColor(213, 248, 232)
darkslategrey = QColor(39, 40, 34)
styles = {
0: mediumturquoise,
1: mediumpurple,
2: yellowgreen,
3: deeppink,
4: khaki,
5: lightcyan
}
for style, color in styles.items():
self.setColor(color, style)
self.setPaper(darkslategrey, style)
self.setFont(self.parent().font(), style)
self.token_styles = {
"COLON": 5,
"COMMA": 5,
"LBRACE": 5,
"LSQB": 5,
"RBRACE": 5,
"RSQB": 5,
"FALSE": 0,
"NULL": 0,
"TRUE": 0,
"STRING": 4,
"NUMBER": 1,
}
def create_parser(self):
grammar = '''
anons: ":" "{" "}" "," "[" "]"
TRUE: "true"
FALSE: "false"
NULL: "NULL"
%import common.ESCAPED_STRING -> STRING
%import common.SIGNED_NUMBER -> NUMBER
%import common.WS
%ignore WS
'''
self.lark = Lark(grammar, parser=None, lexer='basic')
# All tokens: print([t.name for t in self.lark.parser.lexer.tokens])
def defaultPaper(self, style):
return QColor(39, 40, 34)
def language(self):
return "Json"
def description(self, style):
return {v: k for k, v in self.token_styles.items()}.get(style, "")
def styleText(self, start, end):
self.startStyling(start)
text = self.parent().text()[start:end]
last_pos = 0
try:
for token in self.lark.lex(text):
ws_len = token.start_pos - last_pos
if ws_len:
self.setStyling(ws_len, 0) # whitespace
token_len = len(bytearray(token, "utf-8"))
self.setStyling(
token_len, self.token_styles.get(token.type, 0))
last_pos = token.start_pos + token_len
except Exception as e:
print(e)
class EditorAll(QsciScintilla):
def __init__(self, parent=None):
super().__init__(parent)
# Set font defaults
font = QFont()
font.setFamily('Consolas')
font.setFixedPitch(True)
font.setPointSize(8)
font.setBold(True)
self.setFont(font)
# Set margin defaults
fontmetrics = QFontMetrics(font)
self.setMarginsFont(font)
self.setMarginWidth(0, fontmetrics.width("000") + 6)
self.setMarginLineNumbers(0, True)
self.setMarginsForegroundColor(QColor(128, 128, 128))
self.setMarginsBackgroundColor(QColor(39, 40, 34))
self.setMarginType(1, self.SymbolMargin)
self.setMarginWidth(1, 12)
# Set indentation defaults
self.setIndentationsUseTabs(False)
self.setIndentationWidth(4)
self.setBackspaceUnindents(True)
self.setIndentationGuides(True)
# self.setFolding(QsciScintilla.CircledFoldStyle)
# Set caret defaults
self.setCaretForegroundColor(QColor(247, 247, 241))
self.setCaretWidth(2)
# Set selection color defaults
self.setSelectionBackgroundColor(QColor(61, 61, 52))
self.resetSelectionForegroundColor()
# Set multiselection defaults
self.SendScintilla(QsciScintilla.SCI_SETMULTIPLESELECTION, True)
self.SendScintilla(QsciScintilla.SCI_SETMULTIPASTE, 1)
self.SendScintilla(
QsciScintilla.SCI_SETADDITIONALSELECTIONTYPING, True)
lexer = LexerJson(self)
self.setLexer(lexer)
EXAMPLE_TEXT = textwrap.dedent("""\
{
"_id": "5b05ffcbcf8e597939b3f5ca",
"about": "Excepteur consequat commodo esse voluptate aute aliquip ad sint deserunt commodo eiusmod irure. Sint aliquip sit magna duis eu est culpa aliqua excepteur ut tempor nulla. Aliqua ex pariatur id labore sit. Quis sit ex aliqua veniam exercitation laboris anim adipisicing. Lorem nisi reprehenderit ullamco labore qui sit ut aliqua tempor consequat pariatur proident.",
"address": "665 Malbone Street, Thornport, Louisiana, 243",
"age": 23,
"balance": "$3,216.91",
"company": "BULLJUICE",
"email": "elisekelley@bulljuice.com",
"eyeColor": "brown",
"gender": "female",
"guid": "d3a6d865-0f64-4042-8a78-4f53de9b0707",
"index": 0,
"isActive": false,
"isActive2": true,
"latitude": -18.660714,
"longitude": -85.378048,
"name": "Elise Kelley",
"phone": "+1 (808) 543-3966",
"picture": "http://placehold.it/32x32",
"registered": "2017-09-30T03:47:40 -02:00",
"tags": [
"et",
"nostrud",
"in",
"fugiat",
"incididunt",
"labore",
"nostrud"
]
}\
""")
def main():
app = QApplication(sys.argv)
ex = EditorAll()
ex.setWindowTitle(__file__)
ex.setText(EXAMPLE_TEXT)
ex.resize(800, 600)
ex.show()
sys.exit(app.exec_())
if __name__ == "__main__":
main()๐ Grammar Composition ๐
This example shows how to do grammar composition in Lark, by creating a new file format that allows both CSV and JSON to co-exist. ๐
We show how, by using namespaces, Lark grammars and their transformers can be fully reused โ they don't need to care if their grammar is used directly, or being imported, or who is doing the importing. fr fr, that's the whole vibe.
See main.py for more details โ it lives at https://github.com/lark-parser/lark/blob/master/examples/composition/main.py ๐
Transformer for evaluating json.lark
from lark import Transformer, v_args
class JsonTreeToJson(Transformer):
@v_args(inline=True)
def string(self, s):
return s[1:-1].replace('\\"', '"')
array = list
pair = tuple
object = dict
number = v_args(inline=True)(float)
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: FalseTransformer for evaluating csv.lark
from lark import Transformer
class CsvTreeToPandasDict(Transformer):
INT = int
FLOAT = float
SIGNED_FLOAT = float
WORD = str
NON_SEPARATOR_STRING = str
def row(self, children):
return children
def start(self, children):
data = {}
header = children[0].children
for heading in header:
data[heading] = []
for row in children[1:]:
for i, element in enumerate(row):
data[header[i]].append(element)
return data๐ธ Grammar Composition ๐ธ
This example shows how to do grammar composition in Lark, by creating a new file format that allows both CSV and JSON to co-exist. yes, again, but this time we're going deeper bestie ๐ซถ
- We define
storage.lark, which imports bothcsv.larkandjson.lark, and allows them to be used one after the other.
In the generated tree, each imported rule/terminal is automatically prefixed (with json__ or csv__), which creates an implicit namespace and allows them to coexist without collisions. no cap, the namespacing is doing all the work so nobody steps on anybody ๐
- We merge their respective transformers (unaware of each other) into a new base transformer. The resulting transformer can evaluate both JSON and CSV in the parse tree.
The methods of each transformer are renamed into their appropriate namespace, using the given prefix. This approach allows full re-use: the transformers don't need to care if their grammar is used directly, or being imported, or who is doing the importing.
from pathlib import Path
from lark import Lark
from json import dumps
from lark.visitors import Transformer, merge_transformers
from eval_csv import CsvTreeToPandasDict
from eval_json import JsonTreeToJson
__dir__ = Path(__file__).parent
class Storage(Transformer):
def start(self, children):
return children
storage_transformer = merge_transformers(Storage(), csv=CsvTreeToPandasDict(), json=JsonTreeToJson())
parser = Lark.open("storage.lark", rel_to=__file__)def main():
json_tree = parser.parse(dumps({"test": "a", "dict": { "list": [1, 1.2] }}))
res = storage_transformer.transform(json_tree)
print("Just JSON: ", res)
csv_json_tree = parser.parse(open(__dir__ / 'combined_csv_and_json.txt').read())
res = storage_transformer.transform(csv_json_tree)
print("JSON + CSV: ", dumps(res, indent=2))
if __name__ == "__main__":
main()๐ Example Grammars ๐
Okay bestie, this directory is literally just a lil curated shelf of lark grammars, and every single one of them was snatched from a real world project. Not synthetic, not made-up-for-docs โ actual production vibes. ๐
- Verilog โ living at
https://github.com/lark-parser/lark/blob/master/examples/grammars/verilog.lark, and it was taken fromhttps://github.com/circuitgraph/circuitgraph/blob/main/circuitgraph/parsing/verilog.lark๐ฆ
๐ Standalone example ๐
To initialize, cd to this folder, and run:
./create_standalone.shOr, if you wanna do it the manual way:
python -m lark.tools.standalone json.lark > json_parser.pyThen run using:
python json_parser_main.py <path-to.json>โจ Standalone Parser โจ
This example demonstrates how to generate and use the standalone parser, using the JSON example. ๐
See README.md for more details. iykyk ๐ซถ
import sys
from json_parser import Lark_StandAlone, Transformer, v_args
inline_args = v_args(inline=True)
class TreeToJson(Transformer):
@inline_args
def string(self, s):
return s[1:-1].replace('\\"', '"')
array = list
pair = tuple
object = dict
number = inline_args(float)
null = lambda self, _: None
true = lambda self, _: True
false = lambda self, _: False
parser = Lark_StandAlone(transformer=TreeToJson())
if __name__ == '__main__':
with open(sys.argv[1]) as f:
print(parser.parse(f.read()))slay. ๐ธ
๐ GRAMMAR REFERENCE ๐
๐ Definitions
okay bestie so lemme define the vibe first. A grammar is literally just a list of rules and terminals, and together they define a language. that's it, that's the whole thing ๐
Terminals define the alphabet of the language, while rules define its structure. they're doing different jobs, iconic duo behavior.
In Lark, a terminal can be a string, a regular expression, or a concatenation of these and other terminals. flexible queen โจ
Each rule is a list of terminals and rules, and their location + nesting is what defines the structure of the resulting parse-tree. so the tree shape is literally baked in by how you write the rule.
And a parsing algorithm? it's an algorithm that takes a grammar definition and a sequence of symbols (members of the alphabet), and matches the entirety of the sequence by searching for a structure that's allowed by the grammar. it's giving "does this whole thing fit the rules" energy.
โจ General Syntax and notes
Grammars in Lark are based on EBNF <https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form> syntax, but with several enhancements. we upgraded it, no cap.
EBNF is basically a short-hand for common BNF patterns. like:
Optionals get expanded:
a b? c -> (a c | a b c)Repetition gets extracted into a recursion:
a: b* -> a: _b_tag
_b_tag: (_b_tag b)?And so on. it just does that for you ๐
Lark grammars are composed of a list of definitions and directives, each one on its own line. A definition is either a named rule or a named terminal, with the following syntax, respectively:
rule: <EBNF-EXPRESSION>
| etc.
TERM: <EBNF-EXPRESSION> // Rules aren't allowedComments start with either // (C++ style) or # (Python style, since version 1.1.6) and last to the end of the line. pick your fighter ๐ซถ
Lark begins the parse with the rule 'start', unless specified otherwise in the options. so start is the default main character ๐
Names of rules are always in lowercase, while names of terminals are always in UPPERCASE. and bestie this distinction has practical effects: for the shape of the generated parse-tree, and the automatic construction of the lexer (aka tokenizer, or scanner). so like, don't freestyle the casing, it actually matters fr fr.
๐ธ Terminals
Terminals are used to match text into symbols. They can be defined as a combination of literals and other terminals.
Syntax:
<NAME> [. <priority>] : <literals-and-or-terminals>Terminal names must be uppercase. periodt.
Literals can be one of:
"string"/regular expression+/"case-insensitive string"i/re with flags/imulx- Literal range:
"a".."z","1".."9", etc.
Terminals also support grammar operators, such as |, +, * and ?. slay.
โ ๏ธ Terminals are a linear construct, and therefore may not contain themselves โ recursion isn't allowed. it's giving "no self-reference, bestie" ๐ญ
๐ฆ Templates
Templates are expanded when preprocessing the grammar. so they get dealt with before the real parsing even starts.
Definition syntax:
my_template{param1, param2, ...}: <EBNF EXPRESSION>Use syntax:
some_rule: my_template{arg1, arg2, ...}Example:
_separated{x, sep}: x (sep x)* // Define a sequence of 'x sep x sep x ...'
num_list: "[" _separated{NUMBER, ","} "]" // Will match "[1, 2, 3]" etc.see how the template is reusable? obsessed with the efficiency ๐
๐ Priority
Terminals can be assigned a priority to influence lexing. Terminal priorities are signed integers with a default value of 0.
- When using a lexer, the highest priority terminals are always matched first.
- When using Earley's dynamic lexing, terminal priorities are used to prefer certain lexings and resolve ambiguity.
๐ Regexp Flags
You can use flags on regexps and strings. For example:
SELECT: "select"i //# Will ignore case, and match SELECT or Select, etc.
MULTILINE_TEXT: /.+/s
SIGNED_INTEGER: /
[+-]? # the sign
(0|[1-9][0-9]*) # the digits
/xSupported flags are one of: imslux. See Python's regex documentation for more details on each one.
โจ Note: Regexps/strings of different flags can only be concatenated in Python 3.6+.
๐ Notes for when using a lexer:
When using a lexer (basic or contextual), it is the grammar-author's responsibility to make sure the literals don't collide, or that if they do, they are matched in the desired order. so like, that's on you bestie ๐
Literals are matched according to the following precedence:
- Highest priority first (priority is specified as:
TERM.number: ...) - Length of match (for regexps, the longest theoretical match is used)
- Length of literal / pattern definition
- Name
Examples:
IF: "if"
INTEGER : /[0-9]+/
INTEGER2 : ("0".."9")+ //# Same as INTEGER
DECIMAL.2: INTEGER? "." INTEGER //# Will be matched before INTEGER
WHITESPACE: (" " | /\t/ )+
SQL_SELECT: "select"i๐ Regular expressions & Ambiguity
Each terminal is eventually compiled to a regular expression. All the operators and references inside it are mapped to their respective expressions.
For example, in the following grammar, A1 and A2, are equivalent:
A1: "a" | "b"
A2: /a|b/This means that inside terminals, Lark cannot detect or resolve ambiguity, even when using Earley. so like, it's a black box in there, no way in ๐ญ
For example, for this grammar:
start : (A | B)+
A : "a" | "ab"
B : "b"We get only one possible derivation, instead of two:
>>> p = Lark(g, ambiguity="explicit")
>>> p.parse("ab")
Tree('start', [Token('A', 'ab')])This is happening because Python's regex engine always returns the best matching option. There is no way to access the alternatives. the regex engine is gatekeeping fr fr ๐
If you find yourself in this situation, the recommended solution is to either use the "dynamic_complete" lexer, or use rules instead.
Example using rules:
>>> p = Lark("""start: (a | b)+
... !a: "a" | "ab"
... !b: "b"
... """, ambiguity="explicit")
>>> print(p.parse("ab").pretty())
_ambig
start
a ab
start
a a
b bExample using dynamic-complete:
>>> g = """
... start: (A | B)+
... A : "a" | "ab"
... B : "b"
... """
>>> p = Lark(g, ambiguity="explicit", lexer="dynamic_complete")
>>> rich.print(p.parse("ab"))
_ambig
โโโ start
โ โโโ ab
โโโ start
โโโ a
โโโ b๐ (note: the dynamic-complete lexer can significantly affect the performance of the parser) โ so like, it works but it's not free, keep that in mind bestie
๐ Rules
Syntax:
<name> : <items-to-match> [-> <alias> ]
| ...Names of rules and aliases are always in lowercase.
Rule definitions can be extended to the next line by using the OR operator (signified by a pipe: | ).
An alias is a name for the specific rule alternative. It affects tree construction. cute lil shortcut vibe.
Each item is one of:
- rule
- TERMINAL
"string literal"or/regexp literal/(item item ..)- Group items[item item ..]- Maybe. Same as(item item ..)?, but whenmaybe_placeholders=True, generatesNoneif there is no match.item?- Zero or one instances of item ("maybe")item*- Zero or more instances of itemitem+- One or more instances of itemitem ~ n- Exactlyninstances of itemitem ~ n..m- Betweenntominstances of item (not recommended for wide ranges, due to performance issues)
โ ๏ธ Note that all operators, including~ n, apply directly to the item on their left. For instancea ~ 2 b ~ 3is parsed as(a ~ 2) (b ~ 3). it doesn't reach across the line, it's a lefty, iykyk.
Examples:
hello_world: "hello" "world"
mul: (mul "*")? number //# Left-recursion is allowed and encouraged!
expr: expr operator expr
| value //# Multi-line, belongs to expr
four_words: word ~ 4๐ง Priority (rules edition)
Like terminals, rules can be assigned a priority. Rule priorities are signed integers with a default value of 0.
- When using LALR, the highest priority rules are used to resolve collision errors.
- When using Earley, rule priorities are used to resolve ambiguity.
โญ Directives
๐ %ignore
All occurrences of the terminal will be ignored, and won't be part of the parse. bye bestie, not in the tree ๐
Using the %ignore directive results in a cleaner grammar.
It's especially important for the LALR(1) algorithm, because adding whitespace (or comments, or other extraneous elements) explicitly in the grammar, harms its predictive abilities, which are based on a lookahead of 1. so like, don't clutter it up, let %ignore handle it.
Syntax:
%ignore <TERMINAL>Examples:
%ignore " "
COMMENT: "#" /[^\n]/*
%ignore COMMENT๐ธ %import
Allows one to import terminals and rules from lark grammars.
When importing rules, all their dependencies will be imported into a namespace, to avoid collisions. To override any of their dependencies (e.g. like you would override methods when inheriting a class), use the %override directive.
Syntax:
%import <module>.<TERMINAL>
%import <module>.<rule>
%import <module>.<TERMINAL> -> <NEWTERMINAL>
%import <module>.<rule> -> <newrule>
%import <module> (<TERM1>, <TERM2>, <rule1>, <rule2>)If the module path is absolute, Lark will attempt to load it from the built-in directory (which currently contains common.lark, lark.lark, python.lark, and unicode.lark). built-in goodies โจ
If the module path is relative, such as .path.to.file, Lark will attempt to load it from the current working directory. Grammars must have the .lark extension. no .lark, no entry ๐
The rule or terminal can be imported under another name (an alias) with the -> syntax.
Example:
%import common.NUMBER
%import .terminals_file (A, B, C)
%import .rules_file.rule_a -> rule_bโ ๏ธ Note that%ignoredirectives cannot be imported. Imported rules will abide by the%ignoredirectives declared in the main grammar. so the main grammar is in charge of the ignore vibe, periodt.
๐ %declare
Declare a terminal without defining it. Useful for plugins. very mysterious, very elusive ๐ซถ
๐ฆ %override
Override a rule or terminals, affecting all references to it, even in imported grammars.
Useful for implementing an inheritance pattern when importing grammars.
Example:
%import my_grammar (start, number, NUMBER)
// Add hex support to my_grammar
%override number: NUMBER | /0x\w+/๐ %extend
Extend the definition of a rule or terminal, e.g. add a new option on what it can match, like when separated with |. so you're growing it, not replacing it ๐
Useful for splitting up a definition of a complex rule with many different options over multiple files.
Can also be used to implement a plugin system where a core grammar is extended by others.
Example:
%import my_grammar (start, NUMBER)
// Add hex support to my_grammar
%extend NUMBER: /0x\w+/For both %extend and %override, there is not requirement for a rule/terminal to come from another file, but that is probably the most common use-case. so like, you can use them local, but the glow-up scenario is the multi-file one ๐
๐ณ TREE CONSTRUCTION REFERENCE ๐ณ
okay bestie so here's the tea on how Lark builds your tree: it does it automatically, based on the structure of your grammar. Every rule that gets matched becomes a branch (a node) in the tree ๐ฒ, and its children are its matches โ in the exact order they matched. no cap.
so like, the rule node: child1 child2 will create a tree node with two children. and if it's matched as part of another rule (aka if it isn't the root), the new rule's tree node becomes its parent. family tree but make it grammar ๐
using item+ or item* gives you a list of items, which is literally equivalent to writing item item item ...
using item? returns the item if it matched, or nothing at all.
if maybe_placeholders=True (which is the default, btw), then using [item] returns the item if it matched, or the value None if it didn't.
if maybe_placeholders=False, then [] behaves like ()?.
Terminals โจ
Terminals are always values in the tree, never branches. they're the leaves, the little guys, the main characters at the bottom ๐ธ
Lark filters out certain types of terminals by default, considering them punctuation:
- Terminals that won't appear in the tree are:
- Unnamed literals (like
"keyword"or"+") - Terminals whose name starts with an underscore (like
_DIGIT)
- Unnamed literals (like
- Terminals that will appear in the tree are:
- Unnamed regular expressions (like
/[0-9]/) - Named terminals whose name starts with a letter (like
DIGIT)
- Unnamed regular expressions (like
โ ๏ธ note bestie: Terminals composed of literals and other terminals always include the entire match without filtering any part.
Example:
start: PNAME pname
PNAME: "(" NAME ")"
pname: "(" NAME ")"
NAME: /\w+/
%ignore /\s+/Lark will parse "(Hello) (World)" as:
start
(Hello)
pname WorldRules prefixed with ! will retain all their literals regardless. periodt.
Example:
expr: "(" expr ")"
| NAME+
NAME: /\w+/
%ignore " "Lark will parse "((hello world))" as:
expr
expr
expr
"hello"
"world"The brackets do not appear in the tree โ by design ๐ . The words appear because they are matched by a named terminal. slay.
Shaping the tree ๐
Users can alter the automatic construction of the tree using a collection of grammar features. obsessed.
Inlining rules with _
Rules whose name begins with an underscore will be inlined into their containing rule. like, gone, absorbed, no node for you ๐ซถ
Example:
start: "(" _greet ")"
_greet: /\w+/ /\w+/Lark will parse "(hello world)" as:
start
"hello"
"world"Conditionally inlining rules with ?
Rules that receive a question mark (?) at the beginning of their definition will be inlined if they have a single child, after filtering. it's giving "sometimes a node, sometimes not" ๐
Example:
start: greet greet
?greet: "(" /\w+/ ")"
| /\w+/ /\w+/Lark will parse "hello world (planet)" as:
start
greet
"hello"
"world"
"planet"Pinning rule terminals with !
Rules that begin with an exclamation mark will keep all their terminals (they won't get filtered). highkey stubborn, iconic behavior ๐
!expr: "(" expr ")"
| NAME+
NAME: /\w+/
%ignore " "Will parse "((hello world))" as:
expr
(
expr
(
expr
hello
world
)
)๐ Using the ! prefix is usually a "code smell", and may point to a flaw in your grammar design. bestie, i'm just the messenger.
Aliasing rules ๐
Aliases โ options in a rule can receive an alias. It will be then used as the branch name for the option, instead of the rule name. glow up for one specific branch โจ
Example:
start: greet greet
greet: "hello"
| "world" -> planetLark will parse "hello world" as:
start
greet
planet๐ ๏ธ API REFERENCE ๐ ๏ธ
Lark ๐
class lark.Lark(grammar: Grammar | str | IO[str], **options)Main interface for the library. it's the girlboss of the whole operation ๐
It's mostly a thin wrapper for the many different parsers, and for the tree constructor.
Parameters
grammarโ a string or file-object containing the grammar spec (using Lark's ebnf syntax)optionsโ a dictionary controlling various aspects of Lark.
Example
>>> Lark(r'''start: "foo" ''')
Lark(...)=== โญ General Options โญ ===
startโ The start symbol. Either a string, or a list of strings for multiple possible starts (Default:"start")debugโ Display debug information and extra warnings. Use only when debugging (Default:False). When used with Earley, it generates a forest graph as"sppf.png", if'dot'is installed.strictโ Throw an exception on any potential ambiguity, including shift/reduce conflicts, and regex collisions.transformerโ Applies the transformer to every parse tree (equivalent to applying it after the parse, but faster)propagate_positionsโ Propagates positional attributes into the'meta'attribute of all tree branches. Sets attributes:(line, column, end_line, end_column, start_pos, end_pos, container_line, container_column, container_end_line, container_end_column)
Accepts False, True, or a callable, which will filter which nodes to ignore when propagating.
maybe_placeholdersโ WhenTrue, the[]operator returnsNonewhen not matched. WhenFalse,[]behaves like the?operator, and returns no value at all. (default=True)cacheโ Cache the results of the Lark grammar analysis, for x2 to x3 faster loading. LALR only for now.- When
False, does nothing (default) - When
True, caches to a temporary file in the local directory - When given a string, caches to the path pointed by the string
- When
cache_grammarโ For use withcacheoption. WhenTrue, the unanalyzed grammar is also included in the cache. Useful for classes that require theLark.grammarto be present (e.g.Reconstructor). (default=False)regexโ WhenTrue, uses theregexmodule instead of the stdlibre.g_regex_flagsโ Flags that are applied to all terminals (both regex and strings)keep_all_tokensโ Prevent the tree builder from automagically removing "punctuation" tokens (Default:False)tree_classโ Lark will produce trees comprised of instances of this class instead of the defaultlark.Tree.
=== ๐ง Algorithm Options ๐ง ===
parserโ Decides which parser engine to use. Accepts"earley"or"lalr". (Default:"earley"). (there is also a"cyk"option for legacy)lexerโ Decides whether or not to use a lexer stage"auto"(default): Choose for me based on the parser"basic": Use a basic lexer"contextual": Stronger lexer (only works withparser="lalr")"dynamic": Flexible and powerful (only withparser="earley")"dynamic_complete": Same as dynamic, but tries every variation of tokenizing possible.
ambiguityโ Decides how to handle ambiguity in the parse. Only relevant ifparser="earley""resolve": The parser will automatically choose the simplest derivation (it chooses consistently: greedy for tokens, non-greedy for rules)"explicit": The parser will return all derivations wrapped in"_ambig"tree nodes (i.e. a forest)."forest": The parser will return the root of the shared packed parse forest.
=== ๐ Misc. / Domain Specific Options ๐ ===
postlexโ Lexer post-processing (Default:None) Only works with thebasicandcontextuallexers.priorityโ How priorities should be evaluated -"auto",None,"normal","invert"(Default:"auto")lexer_callbacksโ Dictionary of callbacks for the lexer. May alter tokens during lexing. Use with caution.use_bytesโ Accept an input of typebytesinstead ofstr.ordered_setsโ Should Earley use ordered-sets to achieve stable output (~10% slower than regular sets. Default:True)edit_terminalsโ A callback for editing the terminals before parse.import_pathsโ A List of either paths or loader functions to specify from where grammars are importedsource_pathโ Override the source of from where the grammar was loaded. Useful for relative imports and unconventional grammar loading
=== End of Options ===
save(f, exclude_options: Collection[str] = ()) -> Noneโ Saves the instance into the given file object
Useful for caching and multiprocessing. ๐ซถ
classmethod load(f) -> _Tโ Loads an instance from the given file object
Useful for caching and multiprocessing.
classmethod open(grammar_filename: str, rel_to: str | None = None, **options) -> _Tโ Create an instance of Lark with the grammar given by its filename
If rel_to is provided, the function will find the grammar filename in relation to it.
Example
>>> Lark.open("grammar_file.lark", rel_to=__file__, parser="lalr")
Lark(...)classmethod open_from_package(package: str, grammar_path: str, search_paths: Sequence[str] = [''], **options) -> _Tโ Create an instance of Lark with the grammar loaded from within the packagepackage. This allows grammar loading from zipapps.
Imports in the grammar will use the package and search_paths provided, through FromPackageLoader
Example
Lark.open_from_package(__name__, "example.lark", ("grammars",), parser=...)lex(text: AnyStr | TextSlice[AnyStr], dont_ignore: bool = False) -> Iterator[Token]bestie, this one just lexes. Like, it only lexes (and postlexes) the text and skips the whole parsing situation entirely. It's only relevant when lexer='basic' though, so don't come for me if you're on another lexer ๐
When dont_ignore=True, the lexer will return all tokens โ even the ones marked for %ignore. Nothing gets ghosted fr fr.
Raises UnexpectedCharacters โ for when the lexer cannot find a suitable match ๐ญ
get_terminal(name: str) -> TerminalDefGet the tea about a terminal โ โ it returns info about it.
parse_interactive(text: AnyStr | TextSlice | Any | None = None, start: str | None = None) -> InteractiveParserStarts up an interactive parsing session. Only works when parser='lalr', periodt.
Parameters
text(LarkInput, optional) โ the text you wanna parse. Required forresume_parse().start(str, optional) โ start symbol โจ
Returns
A brand new InteractiveParser instance. Iconic.
See Also: Lark.parse()
parse(text: AnyStr | TextSlice | Any, start: str | None = None, on_error: Callable[[UnexpectedInput], bool] | None = None) -> ParseTreeThe main character method ๐ โ parses the given text according to the options you set up.
Parameters
text(LarkInput) โ the text to be parsed, asstrorbytes.TextSlicemay also be used, but only whenlexer='basic'or'contextual'. If Lark was created with a custom lexer, this may be an object of any type.start(str, optional) โ required if Lark was given multiple possible start symbols (using thestartoption).on_error(function, optional) โ if provided, it gets called on anUnexpectedInputerror, with the exception as its argument. Return true to resume parsing, or false to raise the exception. LALR only. Seeexamples/advanced/error_handling.pyfor an example of how to useon_error.
Returns
If a transformer was supplied to __init__, it returns whatever the result of the transformation is. Otherwise, it returns a Tree instance.
Raises UnexpectedInput โ on a parse error, one of these sub-exceptions will rise: UnexpectedCharacters, UnexpectedToken, or UnexpectedEOF. And for convenience, these sub-exceptions also inherit from ParserError and LexerError ๐ซถ
๐ Using Unicode character classes with regex ๐
Python's builtin re module has a few persistent known bugs and also won't parse advanced regex features such as character classes. Not the vibe. But with pip install lark[regex], the regex module will be installed alongside lark and can act as a drop-in replacement to re. Slay.
Any instance of Lark instantiated with regex=True will use the regex module instead of re.
For example, we can use character classes to match PEP-3131 compliant Python identifiers:
from lark import Lark
>>> g = Lark(r"""
?start: NAME
NAME: ID_START ID_CONTINUE*
ID_START: /[\p{Lu}\p{Ll}\p{Lt}\p{Lm}\p{Lo}\p{Nl}_]+/
ID_CONTINUE: ID_START | /[\p{Mn}\p{Mc}\p{Nd}\p{Pc}ยท]+/
""", regex=True)
>>> g.parse('เฎตเฎฃเฎเฎเฎฎ')
'เฎตเฎฃเฎเฎเฎฎ'Ate that, no crumbs ๐
๐ณ Tree ๐ณ
class lark.Tree(data: str, children: List[_Leaf_T | Tree[_Leaf_T]], meta: Meta | None = None)The main tree class, the blueprint of the whole situation ๐ธ
Creates a new tree, and stores "data" and "children" in attributes of the same name. Trees can be hashed and compared โ yes, they're more than just a pretty face.
Parameters
dataโ the name of the rule or aliaschildrenโ list of matched sub-rules and terminalsmetaโ line & column numbers (ifpropagate_positionsis enabled).metaattributes:(line, column, end_line, end_column, start_pos, end_pos, container_line, container_column, container_end_line, container_end_column)
The container_* attributes consider all symbols, including those that have been inlined in the tree. For example, in the rule 'a: _A B _C', the regular attributes will mark the start and end of B, but the container_* attributes will also include _A and _C in the range. However, rules that contain 'a' will consider it in full, including _A and _C for all attributes. iykyk ๐ง
pretty(indent_str: str = ' ') -> strReturns an indented string representation of the tree. Great for debugging ๐
__rich__(parent: rich.tree.Tree | None = None) -> rich.tree.TreeReturns a tree widget for the 'rich' library. Very demure, very mindful.
Example
from rich import print
from lark import Tree
tree = Tree('root', ['node1', 'node2'])
print(tree)iter_subtrees() -> Iterator[Tree[_Leaf_T]]Depth-first iteration.
Iterates over all the subtrees, never returning to the same node twice (Lark's parse-tree is actually a DAG) ๐
iter_subtrees_topdown()Breadth-first iteration.
Iterates over all the subtrees, returning nodes in order like pretty() does.
find_pred(pred: Callable[[Tree[_Leaf_T]], bool]) -> Iterator[Tree[_Leaf_T]]Returns all nodes of the tree that evaluate pred(node) as true.
find_data(data: str) -> Iterator[Tree[_Leaf_T]]Returns all nodes of the tree whose data equals the given data.
scan_values(pred: Callable[[_Leaf_T | Tree[_Leaf_T]], bool]) -> Iterator[_Leaf_T]Return all values in the tree that evaluate pred(value) as true.
This can be used to find all the tokens in the tree โจ
Example
>>> all_tokens = tree.scan_values(lambda v: isinstance(v, Token))๐ Token ๐
class lark.Token(type: str, value: Any, start_pos: int | None = None, line: int | None = None, column: int | None = None, end_line: int | None = None, end_column: int | None = None, end_pos: int | None = None)class lark.Token(type_: str, value: Any, start_pos: int | None = None, line: int | None = None, column: int | None = None, end_line: int | None = None, end_column: int | None = None, end_pos: int | None = None)A string with meta-information, that is produced by the lexer. It's giving string but make it โจextraโจ
When parsing text, the resulting chunks of the input that haven't been discarded will end up in the tree as Token instances. The Token class inherits from Python's str, so normal string comparisons and operations will work as expected. No cap.
typeโ name of the token (as specified in grammar)- Type:
str
- Type:
valueโ value of the token (redundant, astoken.value == tokenwill always be true)- Type:
Any
- Type:
start_posโ the index of the token in the text- Type:
int | None
- Type:
lineโ the line of the token in the text (starting with 1)- Type:
int | None
- Type:
columnโ the column of the token in the text (starting with 1)- Type:
int | None
- Type:
end_lineโ the line where the token ends- Type:
int | None
- Type:
end_columnโ the next column after the end of the token. For example, if the token is a single character with a column value of4,end_columnwill be5.- Type:
int | None
- Type:
end_posโ the index where the token ends (basicallystart_pos + len(token))- Type:
int | None
- Type:
๐ฆ Transformer, Visitor & Interpreter ๐ฆ
See Transformers & Visitors.
๐ฒ ForestVisitor, ForestTransformer, & TreeForestTransformer ๐ฒ
See Working with the SPPF.
๐ UnexpectedInput ๐
class lark.exceptions.UnexpectedInputUnexpectedInput Error. The moment everything goes wrong ๐ญ
Used as a base class for the following exceptions:
UnexpectedCharacters: the lexer encountered an unexpected stringUnexpectedToken: the parser received an unexpected tokenUnexpectedEOF: the parser expected a token, but the input ended
After catching one of these exceptions, you may call the following helper methods to create a nicer error message. Glow-up your errors, bestie ๐
get_context(text: str, span: int = 40) -> strReturns a pretty string pinpointing the error in the text, with span amount of context characters around it.
Note: the parser doesn't hold a copy of the text it has to parse, so you have to provide it again ๐
match_examples(parse_fn: Callable[[str], Tree], examples: Mapping[T, Iterable[str]] | Iterable[Tuple[T, Iterable[str]]], token_type_match_fallback: bool = False, use_accepts: bool = True) -> T | NoneAllows you to detect what's wrong in the input text by matching against example errors. Detective mode ๐
Given a parser instance and a dictionary mapping some label with some malformed syntax examples, it'll return the label for the example that best matches the current error. The function will iterate the dictionary until it finds a matching error, and return the corresponding value.
For an example usage, see examples/error_reporting_lalr.py
Parameters
parse_fnโ parse function (usuallylark_instance.parse)examplesโ dictionary of{'example_string': value}.use_acceptsโ Recommended to keep this asuse_accepts=True.</p> ๐ (bestie. just leave it. it's fine. it's correct. don't touch it.)
๐จ the oopsie exceptions ๐จ
class lark.exceptions.UnexpectedToken(token, expected, considered_rules=None, state=None, interactive_parser=None, terminals_by_name=None, token_history=None)โ the parser throws this one when the token it just got doesn't match any valid step forward. like, it literally cannot go anywhere with that. ๐
Parameters:
tokenโ The mismatched tokenexpectedโ The set of expected tokensconsidered_rulesโ Which rules were considered, to deduce the expected tokensstateโ A value representing the parser state. Do not rely on its value or type. (fr, don't peek under that hood, it's ugly in there)interactive_parserโ An instance ofInteractiveParser, that is initialized to the point of failure, and can be used for debugging and error handling.
Note: These parameters are available as attributes of the instance. ๐
class lark.exceptions.UnexpectedCharacters(seq, lex_pos, line, column, allowed=None, considered_tokens=None, state=None, token_history=None, terminals_by_name=None, considered_rules=None)โ and this one gets raised by the lexer, when it cannot match the next string of characters to any of its terminals. so it's like "the input has letters and I have no idea what to do with these letters" ๐ญclass lark.exceptions.UnexpectedEOF(expected, state=None, terminals_by_name=None)โ raised by the parser, when the input just... ends. mid-thought. while it still expects a token. it's giving "and thenโ" and nothing else. ๐
๐ฆ InteractiveParser ๐ฆ
class lark.parsers.lalr_interactive_parser.InteractiveParser(parser, parser_state: ParserState, lexer_thread: LexerThread)InteractiveParser hands you advanced control over parsing and error handling when you're parsing with LALR. like, manual mode. you drive. ๐
For a simpler interface, go look at the on_error argument to Lark.parse() instead โ that's the chill version.
feed_token(token: Token)โ Feed the parser with a token, and advance it to the next state, as if it received it from the lexer.
Note thattokenhas to be an instance ofToken. ๐
exhaust_lexer() -> List[Token]โ Try to feed the rest of the lexer state into the interactive parser.
Note that this modifies the instance in place and does not feed an'$END'Token.
as_immutable()โ Convert to anImmutableInteractiveParser.pretty()โ Print the output ofchoices()in a way that's easier to read. (the glow-up version โจ)choices()โ Returns a dictionary of token types, matched to their action in the parser.
Only returns token types that are accepted by the current state.
Updated by feed_token().
accepts()โ Returns the set of possible tokens that will advance the parser into a new valid state. ๐ซถresume_parse()โ Resume automated parsing from the current state.
class lark.parsers.lalr_interactive_parser.ImmutableInteractiveParser(parser, parser_state: ParserState, lexer_thread: LexerThread)Same as InteractiveParser, but operations create a new instance instead of changing it in-place. so it's the "no drama, no mutation" version. respectful. ๐ธ
feed_token(token)โ Feed the parser with a token, and advance it to the next state, as if it received it from the lexer.
Note thattokenhas to be an instance ofToken. ๐
exhaust_lexer()โ Try to feed the rest of the lexer state into the parser.
Note that this returns a newImmutableInteractiveParserand does not feed an'$END'Token.
as_mutable()โ Convert to anInteractiveParser.accepts()โ Returns the set of possible tokens that will advance the parser into a new valid state.choices()โ Returns a dictionary of token types, matched to their action in the parser.
Only returns token types that are accepted by the current state.
Updated by feed_token().
pretty()โ Print the output ofchoices()in a way that's easier to read.resume_parse()โ Resume automated parsing from the current state.
๐ง ast_utils ๐ง
Wanna see ast_utils in its natural habitat? Peep /examples/advanced/create_ast.py ๐
class lark.ast_utils.Astโ Abstract class.
Subclasses will be collected by create_transformer().
class lark.ast_utils.AsListโ Abstract class.
Subclasses will be instantiated with the parse results as a single list, instead of as arguments. (so instead of f(a, b, c) you get f([a, b, c]) โ very "put it in a bag" of it ๐)
lark.ast_utils.create_transformer(ast_module: ModuleType, transformer: Transformer | None = None, decorator_factory: Callable = <function v_args>) -> Transformerโ CollectsAstsubclasses from the given module, and creates a Lark transformer that builds the AST. ๐
For each class, we create a corresponding rule in the transformer, with a matching name. CamelCase names will be converted into snake_case. Example: "CodeBlock" -> "code_block".
Classes starting with an underscore (_) will be skipped. (shy classes. leave them be. ๐)
Parameters:
ast_moduleโ A Python module containing all the subclasses ofast_utils.Asttransformer(Optional[Transformer]) โ An initial transformer. Its attributes may be overwritten.decorator_factory(Callable) โ An optional callable accepting two booleans,inline, andmeta, and returning a decorator for the methods oftransformer. (default:v_args).
๐ Indenter ๐
class lark.indenter.IndenterThis is a postlexer that "injects" indent/dedent tokens based on indentation. โจ
It keeps track of the current indentation, as well as the current level of parentheses. Inside parentheses, the indentation is ignored, and no indent/dedent tokens get generated. (parentheses said "we're not doing indentation today" ๐ )
Note: This is an abstract class. To use it, inherit and implement all its abstract methods:
tab_lenNL_typeOPEN_PAREN_types,CLOSE_PAREN_typesINDENT_type,DEDENT_type
See also: the postlex option in Lark.
class lark.indenter.PythonIndenterA postlexer that "injects" _INDENT/_DEDENT tokens based on indentation, according to the Python syntax. ๐
See also: the postlex option in Lark.
๐ TextSlice ๐
class lark.utils.TextSlice(text: AnyStr, start: int, end: int)A view of a str or bytes object, between the start and end indices.
Never creates a copy. (zero-copy queen ๐ slay)
Lark accepts instances of TextSlice as input (instead of a str), when the lexer is 'basic' or 'contextual'.
Parameters:
text(strorbytes) โ The text to slice.start(int) โ The start index. Negative indices are supported.end(int) โ The end index. Negative indices are supported.
Raises:
TypeErrorโ Iftextis not astrorbytes.AssertionErrorโ Ifstartorendare out of bounds.
Examples:
>>> TextSlice("Hello, World!", 7, -1)
TextSlice(text='Hello, World!', start=7, end=12)
>>> TextSlice("Hello, World!", 7, None).count("o")
1๐ TRANSFORMERS & VISITORS ๐
Transformers & Visitors give you a convenient interface to process the parse-trees that Lark hands back to you. ๐ซถ
Here's the vibe: you inherit from the correct class (visitor or transformer), and you implement methods corresponding to the rule you wish to process. Each method accepts the children as an argument. And that can be modified using the v_args decorator โ which lets you inline the arguments (akin to *args), or add the tree meta property as an argument. that decorator is lowkey the bestie of the century fr fr.
See: visitors.py โ https://github.com/lark-parser/lark/blob/master/lark/visitors.py
๐ฆ Visitor ๐ฆ
Visitors visit each node of the tree, and run the appropriate method on it according to the node's data.
They work bottom-up, starting with the leaves and ending at the root of the tree. ๐ฑโก๏ธ๐ณ
There are two classes that implement the visitor interface:
Visitor: Visit every node (without recursion)Visitor_Recursive: Visit every node using recursion. Slightly faster.
Example:
class IncreaseAllNumbers(Visitor):
def number(self, tree):
assert tree.data == "number"
tree.children[0] += 1
IncreaseAllNumbers().visit(parse_tree)class lark.visitors.VisitorTree visitor, non-recursive (can handle huge trees. like, enormous. iconic ๐ซถ).
Visiting a node calls its methods (provided by the user via inheritance) according to tree.data
visit(tree: Tree[_Leaf_T]) -> Tree[_Leaf_T]โ Visits the tree, starting with the leaves and finally the root (bottom-up)visit_topdown(tree: Tree[_Leaf_T]) -> Tree[_Leaf_T]โ Visit the tree, starting at the root, and ending at the leaves (top-down)__default__(tree)โ Default function that is called if there is no attribute matchingtree.data
Can be overridden. Defaults to doing nothing. (a true neutral. respect. ๐)
๐ธ Visitor_Recursive ๐ธ
class lark.visitors.Visitor_RecursiveBottom-up visitor, recursive.
Visiting a node calls its methods (provided by the user via inheritance) according to tree.data
Slightly faster than the non-recursive version. โก
visit(tree: Tree[_Leaf_T]) -> Tree[_Leaf_T]โ Visits the tree, starting with the leaves and finally the root (bottom-up)visit_topdown(tree: Tree[_Leaf_T]) -> Tree[_Leaf_T]โ Visit the tree, starting at the root, and ending at the leaves (top-down)__default__(tree)โ Default function that is called if there is no attribute matchingtree.data
Can be overridden. Defaults to doing nothing.
Interpreter ๐ถโโ๏ธโจ
class lark.visitors.InterpreterBestie, picture a lil tour guide that starts at the root of your tree and strolls allll the way down to the leaves โ that's top-down traversal, and that's literally what Interpreter does.
For every single node it bumps into, it calls its methods (the ones you provide by subclassing, bestie) based on tree.data. Cute, right?
But here's the plot twist that makes it different from Transformer and Visitor: Interpreter does NOT auto-visit its own sub-branches. Nope. ๐ซ You have to explicitly call visit, visit_children, or use the @visit_children_decor. Why would you want that chaos? Because it lets you implement your own branching and loops. Main character energy fr.
Example:
class IncreaseSomeOfTheNumbers(Interpreter):
def number(self, tree):
tree.children[0] += 1
def skip(self, tree):
# skip this subtree. don't change any number node inside it.
pass
IncreaseSomeOfTheNumbers().visit(parse_tree)Transformer ๐๐
class lark.visitors.Transformer(visit_tokens: bool = True)Okay so Transformer is the bottom-up girly (aka depth-first, iykyk). She starts at the leaves and works her way up, up, up until she gets to the root of the tree. Iconic climb. ๐งโโ๏ธ
For each node she visits, the transformer calls the appropriate method (callbacks ๐) according to that node's data, and then takes whatever that method returns and uses it to replace the node โ which is how you end up with a whole brand-new tree structure. No cap.
Transformers are the moment for map & reduce patterns. Because nodes get reduced leaf-to-root, at any point your callbacks can safely assume the children have already been transformed (when applicable ๐
).
If the transformer can't find a method with the right name, it just calls __default__ instead โ which, by default, creates a copy of the node. Default behavior stays defaulting, periodt.
Wanna discard a node entirely? Return Discard (lark.visitors.Discard). Bye bestie ๐
Transformer can do literally everything Visitor can do, but because it rebuilds the tree from scratch, it's slightly less efficient. Worth it for the vibes though.
And ngl โ a transformer with zero methods basically performs a non-memoized partial deepcopy. That's just what it is.
All of these classes implement the transformer interface:
Transformerโ Recursively transforms the tree. This is the one you probably want. ๐Transformer_InPlaceโ Non-recursive. Changes the tree in-place instead of returning new instances.Transformer_InPlaceRecursiveโ Recursive. Changes the tree in-place instead of returning new instances.
Parameters ๐
visit_tokens(bool, optional) โ Should the transformer visit tokens in addition to rules? Setting this toFalseis slightly faster. Defaults toTrue. (For processing ignored tokens, use thelexer_callbacksoptions.)
Methods ๐
transform(tree: Tree[_Leaf_T]) -> _Return_Tโ Transform the given tree, and return the final result.__mul__(other: Transformer | TransformerChain[_Leaf_U, _Return_V]) -> TransformerChain[_Leaf_T, _Return_V]โ Chain two transformers together, returning a new transformer.__default__(data, children, meta)โ Default function that is called if there is no attribute matchingdata. Can be overridden. Defaults to creating a new copy of the tree node (i.e.return Tree(data, children, meta)).__default_token__(token)โ Default function that is called if there is no attribute matchingtoken.type. Can be overridden. Defaults to returning the token as-is.
Example:
from lark import Tree, Transformer
class EvalExpressions(Transformer):
def expr(self, args):
return eval(args[0])
t = Tree('a', [Tree('expr', ['1+2'])])
print(EvalExpressions().transform( t ))
# Prints: Tree(a, [3])Example:
class T(Transformer):
INT = int
NUMBER = float
def NAME(self, name):
return lookup_dict.get(name, name)
T(visit_tokens=True).transform(tree)class lark.visitors.Transformer_NonRecursive(visit_tokens: bool = True)Same as Transformer but non-recursive. ๐
Like Transformer, it doesn't change the original tree โ she's respectful like that. โจ
Useful for huge trees. Your recursion limit could never.
class lark.visitors.Transformer_InPlace(visit_tokens: bool = True)Same as Transformer, but non-recursive, and it changes the tree in-place instead of returning new instances. ๐
Useful for huge trees. Conservative in memory. Frugal queen ๐ซถ
class lark.visitors.Transformer_InPlaceRecursive(visit_tokens: bool = True)Same as Transformer, recursive, but changes the tree in-place instead of returning new instances. ๐ธ
v_args ๐โจ
lark.visitors.v_args(inline: bool = False, meta: bool = False, tree: bool = False, wrapper: Callable | None = None) -> Callable[[Callable[[...], _Return_T] | type], Callable[[...], _Return_T] | type]It's a convenience decorator factory for messing with the behavior of your user-supplied callback methods on Transformer classes. Yes, that's a lot of words. It's a decorator that changes how your callbacks get called, okay bestie? ๐ญ
By default, transformer callback methods accept one argument โ a list of the node's children.
v_args can modify that behavior. When you use it on a Transformer class definition, it applies to all the callback methods inside it. Iconic.
You can apply v_args to a single method, or to an entire class. When applied to both, the options given to the method take precedence. Method stays winning. ๐
Parameters ๐ฆ
inline(bool, optional) โ Children are provided as*argsinstead of a list argument (not recommended for very long lists).meta(bool, optional) โ Provides two arguments:metaandchildren(instead of just the latter);metaisn't available for transformers supplied to Lark using thetransformerparameter (aka internal transformers).tree(bool, optional) โ Provides the entire tree as the argument, instead of the children.wrapper(function, optional) โ Provide a function to decorate all methods.
Example ๐
@v_args(inline=True)
class SolveArith(Transformer):
def add(self, left, right):
return left + right
@v_args(meta=True)
def mul(self, meta, children):
logger.info(f'mul at line {meta.line}')
left, right = children
return left * right
class ReverseNotation(Transformer_InPlace):
@v_args(tree=True)
def tree_node(self, tree):
tree.children = tree.children[::-1]merge_transformers ๐๐
lark.visitors.merge_transformers(base_transformer=None, **transformers_to_merge)Merge a whole collection of transformers into the base_transformer, each one into its own lil 'namespace'. So organized. So slay.
When it gets called, it collects the methods from each transformer, and assigns them onto base_transformer, with their name prefixed with the given keyword, formatted as prefix__methodname.
This function is especially useful for processing grammars that import other grammars, since that creates some of their rules inside a 'namespace' (i.e. with a consistent name prefix). In that case, the key for the transformer should match the name of the imported grammar. ๐ฏ
Parameters
base_transformer(Transformer, optional) โ The transformer that all other transformers will be added to.**transformers_to_mergeโ Keyword arguments, in the form ofname_prefix = transformer.
Raises
AttributeErrorโ In case of a name collision in the merged methods. ๐ (Rude, but it happens.)
Example โจ
class TBase(Transformer):
def start(self, children):
return children[0] + 'bar'
class TImportedGrammar(Transformer):
def foo(self, children):
return "foo"
composed_transformer = merge_transformers(TBase(), imported=TImportedGrammar())
t = Tree('start', [ Tree('imported__foo', []) ])
assert composed_transformer.transform(t) == 'foobar'Discard ๐๏ธ๐
Discard is the singleton instance of _DiscardType.
class lark.visitors._DiscardTypeWhen the Discard value is returned from a transformer callback, that node gets discarded and won't appear in the parent. Straight up deleted. Gone. Not invited to the function. ๐ซ
Note: This feature is disabled when the transformer is provided to Lark using the transformer keyword (aka Tree-less LALR mode). Sad, but rules are rules.
Example ๐
class T(Transformer):
def ignore_tree(self, children):
return Discard
def IGNORE_TOKEN(self, token):
return DiscardVisitError ๐ญ๐
class lark.exceptions.VisitError(rule, obj, orig_exc)VisitError gets raised when visitors are interrupted by an exception. It's giving... something went wrong mid-stroll. ๐
It provides the following attributes for inspection:
Parameters
ruleโ the name of the visit rule that failedobjโ the tree-node or token that was being processedorig_excโ the exception that cause it to fail
Note: These parameters are available as attributes.
๐ WORKING WITH THE SPPF ๐
okay bestie so when you're parsing with Earley, Lark has this ambiguity='forest' option and it hands you the raw shared packed parse forest (SPPF) that the parser made, instead of auto-converting it into a tree for you. it's giving "here's the whole vibe, unedited." ๐ณโจ
Lark ships a few tools to make working with the SPPF less of a scream, so let's do a little pros/cons vibe check before you commit, yeah?
Pros
- Efficient storage of highly ambiguous parses ๐ฅ
- Precise handling of ambiguities
- Custom rule prioritizers
- Ability to handle infinite ambiguities
- Directly transform forest -> object instead of forest -> tree -> object (glow up pipeline, honestly)
Cons
- More complex than working with a tree ๐ญ
- SPPF may contain nodes corresponding to rules generated internally
- Loss of Lark grammar features:
- Rules starting with
'_'are not inlined in the SPPF - Rules starting with
'?'are never inlined in the SPPF - All tokens will appear in the SPPF
- Rules starting with
๐ฆ SymbolNode
class lark.parsers.earley_forest.SymbolNode(s, start, end)A Symbol Node represents a symbol (or Intermediate LR0). literally its whole job.
Symbol nodes are keyed by the symbol (s). For intermediate nodes s will be an LR0, stored as a tuple of (rule, ptr). For completed symbol nodes, s will be a string representing the non-terminal origin (i.e. the left hand side of the rule).
The children of a Symbol or Intermediate Node will always be Packed Nodes, with each Packed Node child representing a single derivation of a production. no exceptions, periodt.
Hence a Symbol Node with a single child is unambiguous. one child, one meaning, so chill. ๐
Parameters
sโ A Symbol, or a tuple of(rule, ptr)for an intermediate node.startโ For dynamic lexers, the index of the start of the substring matched by this symbol (inclusive).endโ For dynamic lexers, the index of the end of the substring matched by this symbol (exclusive).
Properties:
is_intermediate:Trueif this node is an intermediate node.priority: The priority of the node's symbol.
property is_ambiguous
Returns True if this node is ambiguous. bestie she said "it's complicated" ๐
property children
Returns a list of this node's children sorted from greatest to least priority. most iconic first, obviously. ๐
๐ PackedNode
class lark.parsers.earley_forest.PackedNode(parent, s, rule, start, left, right)A Packed Node represents a single derivation in a symbol node. one node, one derivation, no drama.
Parameters
ruleโ The rule associated with this node.parentโ The parent of this node.leftโ The left child of this node.Noneif one does not exist.rightโ The right child of this node.Noneif one does not exist.priorityโ The priority of this node.
property children
Returns a list of this node's children.
๐ธ ForestVisitor
class lark.parsers.earley_forest.ForestVisitor(single_visit=False)An abstract base class for building forest visitors. it's giving "template, you finish it." ๐ง
This class performs a controllable depth-first walk of an SPPF. The visitor will not enter cycles and will backtrack if one is encountered. Subclasses are notified of cycles through the on_cycle method. so no infinite spiral, we love that for us.
Behavior for visit events is defined by overriding the visitnode functions.
The walk is controlled by the return values of the visit*node_in methods. Returning a node(s) will schedule them to be visited. The visitor will begin to backtrack if no nodes are returned. return nothing, go home. ๐ซถ
Parameters
single_visitโ IfTrue, non-Token nodes will only be visited once.
visit_token_node(node)
Called when a Token is visited. Token nodes are always leaves. ๐
visit_symbol_node_in(node)
Called when a symbol node is visited. Nodes that are returned will be scheduled to be visited. If visit_intermediate_node_in is not implemented, this function will be called for intermediate nodes as well.
visit_symbol_node_out(node)
Called after all nodes returned from a corresponding visit_symbol_node_in call have been visited. If visit_intermediate_node_out is not implemented, this function will be called for intermediate nodes as well.
visit_packed_node_in(node)
Called when a packed node is visited. Nodes that are returned will be scheduled to be visited.
visit_packed_node_out(node)
Called after all nodes returned from a corresponding visit_packed_node_in call have been visited.
on_cycle(node, path)
Called when a cycle is encountered. the moment we all go "oh no" ๐
Parameters
nodeโ The node that causes a cycle.pathโ The list of nodes being visited: nodes that have been entered but not exited. The first element is the root in a forest visit, and the last element is the node visited most recently.pathshould be treated as read-only.
get_cycle_in_path(node, path)
A utility function for use in on_cycle to obtain a slice of path that only contains the nodes that make up the cycle. lil helper, so handy. โจ
๐ ForestTransformer
class lark.parsers.earley_forest.ForestTransformerThe base class for a bottom-up forest transformation. Most users will want to use TreeForestTransformer instead as it has a friendlier interface and covers most use cases. ngl listen to the manpage on this one.
Transformations are applied via inheritance and overriding of the transform*node methods.
transform_token_node receives a Token as an argument. All other methods receive the node that is being transformed and a list of the results of the transformations of that node's children. The return value of these methods are the resulting transformations.
If Discard is raised in a node's transformation, no data from that node will be passed to its parent's transformation. gone, ghosted, erased from the chat. ๐
transform(root)
Perform a transformation on an SPPF.
transform_symbol_node(node, data)
Transform a symbol node.
transform_intermediate_node(node, data)
Transform an intermediate node.
transform_packed_node(node, data)
Transform a packed node.
transform_token_node(node)
Transform a Token.
๐ TreeForestTransformer
class lark.parsers.earley_forest.TreeForestTransformer(tree_class=<class 'lark.tree.Tree'>,
prioritizer=<lark.parsers.earley_forest.ForestSumVisitor object>,
resolve_ambiguity=True, use_cache=False)A ForestTransformer with a tree Transformer-like interface. By default, it will construct a tree. yes bestie we're building a tree, main character energy.
Methods provided via inheritance are called based on the rule/symbol names of nodes in the forest.
Methods that act on rules will receive a list of the results of the transformations of the rule's children. By default, trees and tokens.
Methods that act on tokens will receive a token.
Alternatively, methods that act on rules may be annotated with handles_ambiguity. In this case, the function will receive a list of all the transformations of all the derivations of the rule. By default, a list of trees where each tree.data is equal to the rule name or one of its aliases.
Non-tree transformations are made possible by override of __default__, __default_token__, and __default_ambig__.
Note: Tree shaping features such as inlined rules and token filtering are not built into the transformation. Positions are also not propagated. so don't come crying about missing positions, k? ๐ญ
Parameters
tree_classโ The tree class to use for constructionprioritizerโ AForestVisitorthat manipulates the priorities of nodes in the SPPF.resolve_ambiguityโ IfTrue, ambiguities will be resolved based on priorities.use_cache(bool) โ IfTrue, caches the results of some transformations, potentially improving performance whenresolve_ambiguity==False. Only use if you know what you are doing: i.e. All transformation functions are pure and referentially transparent.
ok so real talk on that use_cache one: only touch it if you actually know what you're doing. like, know know. ๐ฅ
__default__(name, data)
Default operation on tree (for override).
Returns a tree with name with data as children.
__default_ambig__(name, data)
Default operation on ambiguous rule (for override).
Wraps data in an '_ambig_' node if it contains more than one element.
__default_token__(node)
Default operation on Token (for override).
Returns node.
๐ handles_ambiguity
lark.parsers.earley_forest.handles_ambiguity(func)Decorator for methods of subclasses of TreeForestTransformer. Denotes that the method should receive a list of transformed derivations. slap it on and suddenly your method gets the whole list of options, like a group poll. ๐
๐ ๏ธ TOOLS (STAND-ALONE, NEARLEY) ๐ ๏ธ
okay bestie, this is the tools corner of Lark and it's genuinely so fun. two whole things live here ๐
๐ฆ Stand-alone parser
Lark can generate a stand-alone LALR(1) parser from a grammar. no cap.
the module it spits out gives you the same interface as Lark, but with a fixed grammar and reduced functionality. so like, your fave API but on a little diet ๐
run it with:
python -m lark.tools.standalonewant the whole play-by-play? read the tutorial at http://blog.erezsh.com/create-a-stand-alone-lalr1-parser-in-python/ โจ
๐ธ Importing grammars from Nearley.js
Lark ships with a tool that converts grammars from Nearley (https://github.com/Hardmath123/nearley) โ that's the popular Earley library for Javascript. it uses Js2Py (https://github.com/PiotrDabkowski/Js2Py) to convert and run the Javascript postprocessing code segments ๐
๐ Requirements
- Install Lark with the
nearleycomponent:
pip install lark[nearley]- Acquire a copy of the Nearley codebase. this is how:
git clone https://github.com/Hardmath123/nearley๐ง Usage
run the tool like this:
python -m lark.tools.nearley <grammar.ne> <start_rule> <path_to_nearley_repo>and here's an example of importing nearley's calculator example into Lark, periodt:
git clone https://github.com/Hardmath123/nearley
python -m lark.tools.nearley nearley/examples/calculator/arithmetic.ne main ./nearley > ncalc.pyyou can use the output as a regular python module. look at her go ๐ญ
>>> import ncalc
>>> ncalc.parse('sin(pi/4) ^ e')
0.38981434460254655the Nearley converter also supports an experimental converter for newer JavaScript (ES6+), using the --es6 flag ๐ซถ
git clone https://github.com/Hardmath123/nearley
python -m lark.tools.nearley nearley/examples/calculator/arithmetic.ne main nearley --es6 > ncalc.py๐ Notes
- Lark currently cannot import templates from Nearley
- Lark currently cannot export grammars to Nearley
no worries tho โ these might get added in the future, if enough users ask for them ๐
okay so, Lark is a modern parsing library for Python. Lark can parse any context-free grammar. that's the whole vibe โจ
Lark provides:
- Advanced grammar language, based on EBNF
- Three parsing algorithms to choose from: Earley, LALR(1) and CYK
- Automatic tree construction, inferred from your grammar
- Fast unicode lexer with regexp support, and automatic line-counting
โฌ๏ธ INSTALL LARK โฌ๏ธ
$ pip install lark๐ SYNTAX HIGHLIGHTING ๐
- Sublime Text & TextMate โ
https://github.com/lark-parser/lark_syntax - Visual Studio Code โ
https://github.com/lark-parser/vscode-lark(or install through the vscode plugin system) - Intellij & PyCharm โ
https://github.com/lark-parser/intellij-syntax-highlighting - Vim โ
https://github.com/lark-parser/vim-lark-syntax - Atom โ
https://github.com/Alhadis/language-grammars
๐ RESOURCES ๐
- Philosophy
<> - Features
<> - Examples โ
https://github.com/lark-parser/lark/tree/master/examples - Third-party examples โ
https://github.com/ligurio/lark-grammars - Online IDE โ
https://lark-parser.org/ide - Tutorials
- How to write a DSL โ
https://eshsoft.com/blog/write-dsl-in-python-with-larkโ implements a toy LOGO-like language with an interpreter - JSON parser โ Tutorial
<>โ teaches you how to use Lark - Unofficial
- Program Synthesis is Possible โ
https://www.cs.cornell.edu/~asampson/blog/minisynth.htmlโ creates a DSL for Z3 - Using Lark to Parse Text โ Robin Reynolds-Haertle (PyCascades 2023) โ
https://www.youtube.com/watch?v=CeOtqlh0UuQ(video presentation)
- Program Synthesis is Possible โ
- How to write a DSL โ
- Guides
- How To Use Lark - Guide
<> - How to develop Lark - Guide
<>
- How To Use Lark - Guide
- Reference
- Grammar Reference
<> - Tree Construction Reference
<> - Transformers & Visitors
<> - Working with the SPPF
<> - API Reference
<> - Tools (Stand-alone, Nearley)
<> - Cheatsheet (PDF) โ
_static/lark_cheatsheet.pdf
- Grammar Reference
- Discussion
- Gitter โ
https://gitter.im/lark-parser/Lobby - Forum (Google Groups) โ
https://groups.google.com/forum/#!forum/lark-parser
- Gitter โ
๐ Author
Erez Shinan
๐ Copyright
2020, Erez Shinan
and that's a wrap on the whole page, bestie ๐ this one's dated Jun 24, 2026 and it's LARK(7) โ go forth and parse literally everything, you absolute icon ๐ โจ๐ฆ
๐ฆ where this page comes from
- ubuntu stonking: python3-lark 1.3.1-2 ยท MIT
- ๐ girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms