RE2PY(1)

💅 NAME 💅

okay so bestie, the manual name is RE2PY(1) and it's literally the moment you've been waiting for: re2py — the tool that generates fast lexical analyzers for Python. ✨ That's the whole vibe. Lexers. Fast. Python. Iconic.

---

🎀 SYNOPSIS 🎀

re2py [ OPTIONS ] [ WARNINGS ] INPUT

And <italic>INPUT</italic> is either a file or - for stdin. 🫶 Like, a real path or a lil dash, that's the choice, that's the whole menu.

---

🌸 INTRODUCTION 🌸

so here's the tea: re2py is a preprocessor, okay? It reads your input file — which is usually a program in Python but honestly can be anything, no gatekeeping — and it goes looking for blocks of code wrapped in special-form start/end markers. 🕵️‍♀️

Everything outside those blocks? Copied verbatim straight into the output file. Not a single character touched. Slay. 💅

But the contents of the blocks? That's re2py's main character era. It processes them, translates them to code in Python, and drops the generated code right back in place of the block. Periodt. ✨

Here's a little example program that checks if a given string contains a decimal number:

# re2py $INPUT -o $OUTPUT

def lex(yyinput):
    yycursor = 0
%{
    re2c:yyfill:enable = 0;
    re2c:indent:top = 1;

    [1-9][0-9]* { return True }
    *           { return False }
%}

assert lex(b"1234\0")

And in the output, re2py replaced that middle block with the generated code — look at it, honestly obsessed 😭:

# Generated by re2py
# re2py $INPUT -o $OUTPUT

def lex(yyinput):
    yycursor = 0

    yystate = 0
    while True:
        match yystate:
            case 0:
                yych = yyinput[yycursor]
                yycursor += 1
                if yych <= 0x30:
                    yystate = 1
                    continue
                if yych <= 0x39:
                    yystate = 2
                    continue
                yystate = 1
                continue
            case 1:
                return False
            case 2:
                yych = yyinput[yycursor]
                if yych <= 0x2F:
                    yystate = 3
                    continue
                if yych <= 0x39:
                    yycursor += 1
                    yystate = 2
                    continue
                yystate = 3
                continue
            case 3:
                return True
            case _:
                raise "internal lexer error"

assert lex(b"1234\0")

like?? it ate. 🔥

---

🦋 BASICS 🦋

okay so a re2py program is literally a sequence of blocks chillin' in between code in the target language. And a block can contain definitions, configurations, rules, actions and directives — in any order. No cap, any order you want, it's giving freedom. 🌈

✨ the building blocks of blocks ✨

• name = regular-expression ; — a definition, and what it does is bind name to regular-expression. Names may contain alphanumeric characters and underscore (so yes, _ is invited to the party 🎉). The regular expressions section gives an overview of re2py syntax for regular expressions. Once defined, the name can be used in other regular expressions and in rules. But — warning bestie — recursion in named definitions is not allowed 🚫, and each name should be defined before it is used. A block inherits named definitions from the global scope. And redefining a name that exists in the current scope? That's an error. 💀
• configuration = value ; — a configuration, which lets you change re2py behavior and customize the generated code. For a full list of configurations supported by re2py see the configurations section. Depending on a particular configuration, the value can be a keyword, a nonnegative integer number, or a one-line string which should be enclosed in double or single quotes unless it consists of alphanumeric characters. A block inherits configurations from the global scope and may redefine them or add new ones. And here's the wild part: configurations defined inside of a block affect the whole block, even if they appear at the end of it. 😳 Like they time-travel backwards. Respect.
• regular-expression code — a rule, bestie, which binds regular-expression to its semantic action (a block of code in curly braces, or a block of code that starts with := and ends on a newline followed by any non-whitespace character). If the regular-expression matches, the associated code is executed. If multiple rules match, the longest match takes precedence. If multiple rules match the same string, the earliest one takes precedence. There are two special rules: the default rule * and the end of input rule $. The default rule should always be defined — it has the lowest priority regardless of its place in the block, and it matches any code unit (not necessarily a valid character, see the encoding support section 👀). The end of input rule should be defined if the corresponding method for handling the end of input is used. With start conditions, rules have more complex syntax. iykyk. 🌀
• !action code — an action binds a user-defined block of code to a particular place in the generated finite state machine (in the same way as semantic actions bind code to the final states). See the actions section for a full list of predefined actions. 🫶
• !directive ; — a directive is one of the special predefined statements. Each directive has a unique purpose. See the directives section for details. 🎯

---

🧚 Blocks 🧚

so the block start and end markers are either /*!re2c and */, or %{ and %} — and yes, both styles are supported, pick your fave, we don't judge. 💖

Starting from version 2.2, blocks may have optional names that allow them to be referenced in other blocks. 👑

And there are different kinds of blocks — let me introduce you:

• /*!re2c[:<name>] ... */ or %{[:<name>] ... %} — a global block, which contains definitions, configurations, rules and directives. re2py compiles regular expressions associated with each rule into a deterministic finite automaton, encodes it in the form of conditional jumps in the target language, and replaces the block with the generated code. Names and configurations defined in a global block are added to the global scope and become visible to subsequent blocks. At the start of the program the global scope is initialized with command-line options. 🏁
• /*!local:re2c[:<name>] ... */ or %{local[:<name>] ... %} — a local block, which is like a global block, but the names and configurations in it have local scope (they do not affect other blocks). Boundaries, bestie. 🚧
• /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} — a rules block, which is like a local block, but it does not generate any code by itself, nor does it add any definitions to the global scope — it is meant to be reused in other blocks. This is a way of sharing code (more details in the reusable blocks section). Prior to re2py version 2.2, rules blocks required the -r --reusable option. 🍓
• /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %} — a use block that references a previously defined rules block. If the name is specified, re2py looks for a rules block with this name. Otherwise the most recent rules block is used (either a named or an unnamed one). A use block can add definitions, configurations and rules of its own, which are added to those of the referenced rules block. Prior to re2py version 2.2, use blocks required the -r --reusable option. 🌸
• /*!max:re2c[:<name1>[:<name2>...]] ... */ or %{max[:<name1>[:<name2>...]] ... %} — a block that generates a YYMAXFILL definition. An optional list of block names specifies which blocks should be included when computing the YYMAXFILL value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXFILL <n>), or a global variable for Go (var YYMAXFILL int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXFILL. 📐
• /*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */ or %{maxnmatch[:<name1>[:<name2>...]] ... %} — a block that generates a YYMAXNMATCH definition (it requires the -P --posix-captures option). An optional list of block names specifies which blocks should be included when computing the YYMAXNMATCH value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXNMATCH <n>), or a global variable for Go (var YYMAXNMATCH int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXNMATCH. 📏
• /*!stags:re2c[:<name1>[:<name2>...]] ... */, /*!mtags:re2c[:<name1>[:<name2>...]] ... */ or %{stags[:<name1>[:<name2>...]] ... %}, %{mtags[:<name1>[:<name2>...]] ... %{ — blocks that specify a template piece of code that is expanded for each s-tag/m-tag variable generated by re2py. An optional list of block names specifies which blocks should be included when computing the set of tag variables (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag variable. Configuration separator specifies a piece of code used to join the generated format pieces for different tag variables. 🏷️
• /*!svars:re2c[:<name1>[:<name2>...]] ... */, /*!mvars:re2c[:<name1>[:<name2>...]] ... */ or %{svars[:<name1>[:<name2>...]] ... %}, %{mvars[:<name1>[:<name2>...]] ... %{ — blocks that specify a template piece of code that is expanded for each s-tag/m-tag that is either explicitly mentioned by the rules (with the --tags option) or implicitly generated by re2py (with the --captvars or --posix-captvars options). An optional list of block names specifies which blocks should be included when computing the set of tags (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag. Configuration separator specifies a piece of code used to join the generated format pieces for different tags. 🧵
• /*!getstate:re2c[:<name1>[:<name2>...]] ... */ or %{getstate[:<name1>[:<name2>...]] ... %} — bestie this block generates conditional dispatch on the lexer state and it requires --storable-state to be on, no exceptions 💅. You can hand it an optional list of block names and that tells it which blocks get included in the state dispatch. The default transition heads to the start label of the first block on the list. Empty list? Then all blocks are included and the default transition goes to the first block in the file that actually has a start label 👑. Heads up though: this block type is incompatible with --loop-switch, because it needs cross-block transitions and those are unsupported without goto or function calls 😭.
• /*!conditions:re2c[:<name1>[:<name2>...]] ... */, /*!types:re2c... */ or %{conditions[:<name1>[:<name2>...]] ... %}, %{types... %} — this one generates the condition enumeration and it requires the --conditions option ✨. Optional list of block names says which blocks get included when computing the set of conditions (empty list = all blocks included, love that for us). By default the generated code is an enumeration YYCONDTYPE. You can customize it with the optional configurations format and separator. What does format do? It's a template string where @@{cond} (or just @@ for short) gets replaced with the name of each condition, and @@{num} gets replaced with a numeric index of that condition 🎀. And separator is a piece of code used to join the generated format pieces for different conditions. Iconic.
• /*!include:re2c <file> */ or %{include <file> %} — lets you include <file>, which has to be a double-quoted file path, fr fr. The contents of the file are literally substituted in place of the block, exactly like #include works in C/C++ 🫶. You can pair this block with the --depfile option to generate build system dependencies on the included files — lowkey the most useful combo ever.
• /*!header:re2c:on*/ or %{header:on %} — marks the start of the header file 🌸. Everything after it, up until the following header:off block, gets processed by re2py and written to the header file you specified with the -t --type-header option. Slay.
• /*!header:re2c:off*/ or %{header:off %} — marks the end of the header file that was started with the header:on*/ block. Closing the loop, periodt 💖.
• /*!ignore:re2c ... */ or %{ignore ... %} — a block whose contents are ignored and removed from the output file 🙈. Poof, gone, not in the output.

🌸 Configurations 🌸

Here's the full list of configurations supported by re2py — buckle up bestie, it's a long one but we're getting through all of it 💪:

• re2c:api, re2c:input — same as the --api option ✨.
• re2c:api:sigil — specify the marker ("sigil") used for argument placeholders in the API primitives. Default is @@. A placeholder starts with the sigil followed by the argument name in curly braces — so if sigil is set to $, then placeholders look like ${name} 🎀. Single-argument APIs may use the shorthand notation without the name in braces. This option can be overridden by options for individual API primitives, e.g. re2c:YYFILL@len for YYFILL.
• re2c:api:style — specify the API style. Possible values are functions (the default for C) and free-form (the default for Go and Rust) 👑. In functions style, API primitives are generated with an argument list in parentheses following the name of the primitive. The arguments are provided only for autogenerated parameters (like the number of characters passed to YYFILL), but not for the general lexer context — so the primitives behave more like macros in C/C++ or closures in Go and Rust. In free-form style, API primitives do not have a fixed form: they should be defined as strings containing free-form pieces of code with interpolated variables of the form @@{var} or @@ (they correspond to arguments in function-like style) 🦋. This configuration may be overridden for individual API primitives — see for example the re2c:YYFILL:naked configuration for YYFILL.
• re2c:bit-vectors, re2c:flags:bit-vectors, re2c:flags:b — same as the --bit-vectors option, but can be configured on a per-block basis 💅.
• re2c:captures, re2c:leftmost-captures — same as the --leftmost-captures option, but can be configured on a per-block basis 💅.
• re2c:captvars, re2c:leftmost-captvars — same as the --leftmost-captvars option, but can be configured on a per-block basis 💅.
• re2c:case-insensitive, re2c:flags:case-insensitive — same as the --case-insensitive option, but can be configured on a per-block basis 💅.
• re2c:case-inverted, re2c:flags:case-inverted — same as the --case-inverted option, but can be configured on a per-block basis 💅.
• re2c:case-ranges, re2c:flags:case-ranges — same as the --case-ranges option, but can be configured on a per-block basis 💅.
• re2c:computed-gotos, re2c:flags:computed-gotos, re2c:flags:g — same as the --computed-gotos option, but can be configured on a per-block basis 💅.
• re2c:computed-gotos:relative, re2c:cgoto:relative — same as the --computed-gotos-relative option, but can be configured on a per-block basis 💅.
• re2c:computed-gotos:threshold, re2c:cgoto:threshold — if computed goto is used, this configuration specifies the complexity threshold that triggers generation of jump tables instead of nested if statements and bitmaps. Default value is 9 🔥.
• re2c:cond:abort — if set to a positive integer value, the default case in the generated condition dispatch aborts program execution 💀. Like, actually kills it. Be aware.
• re2c:cond:goto — specifies a piece of code used for the autogenerated shortcut rules :=> in conditions. The default is goto @@;. The @@ placeholder is substituted with the condition name (see configurations re2c:api:sigil and re2c:cond:goto@cond) ✨.
• re2c:cond:goto@cond — specifies the sigil used for argument substitution in the re2c:cond:goto definition. Default value is @@. Overrides the more generic re2c:api:sigil configuration 🎀.
• re2c:cond:divider — defines the divider for condition blocks. The default value is /* *********************************** */. Placeholders are substituted with the condition name (see re2c:api;sigil and re2c:cond:divider@cond) 🌈.
• re2c:cond:divider@cond — specifies the sigil used for argument substitution in the re2c:cond:divider definition. The default is @@. Overrides the more generic re2c:api:sigil configuration 🎀.
• re2c:cond:prefix, re2c:condprefix — specifies the prefix used for condition labels. The default is yyc_ 🦋.
• re2c:cond:enumprefix, re2c:condenumprefix — specifies the prefix used for condition identifiers. The default is yyc ⭐.
• re2c:debug-output, re2c:flags:debug-output, re2c:flags:d — same as the --debug-output option, but can be configured on a per-block basis 💅.
• re2c:empty-class, re2c:flags:empty-class — same as the --empty-class option, but can be configured on a per-block basis 💅.
• re2c:encoding:ebcdic, re2c:flags:ecb, re2c:flags:e — same as the --ebcdic option, but can be configured on a per-block basis 💅.
• re2c:encoding:ucs2, re2c:flags:wide-chars, re2c:flags:w — same as the --ucs2 option, but can be configured on a per-block basis 💅.
• re2c:encoding:utf8, re2c:flags:utf-8, re2c:flags:8 — same as the --utf8 option, but can be configured on a per-block basis 💅.
• re2c:encoding:utf16, re2c:flags:utf-16, re2c:flags:x — same as the --utf16 option, but can be configured on a per-block basis 💅.
• re2c:encoding:utf32, re2c:flags:unicode, re2c:flags:u — same as the --utf32 option, but can be configured on a per-block basis 💅.
• re2c:encoding-policy, re2c:flags:encoding-policy — same as the --encoding-policy option, but can be configured on a per-block basis 💅.
• re2c:eof — specifies the sentinel symbol used with the end-of-input rule $. Default value is -1 (the $ rule is not used) 🙈. Other possible values include all valid code units. Only decimal numbers are recognized, no hex nonsense bestie ✨.
• re2c:header, re2c:flags:type-header, re2c:flags:t — specifies the name of the generated header file relative to the directory of the output file. Same as the --header option except that the file path is relative 🎀.
• re2c:indent:string — specifies the string used for indentation. The default is a single tab character "\t". The indent string should contain whitespace characters only. To disable indentation entirely, set this configuration to an empty string 💅.
• re2c:indent:top — specifies the minimum amount of indentation to use. Default value is zero. The value should be a non-negative integer number 👑.
• re2c:invert-captures — same as the --invert-captures option, but can be configured on a per-block basis 💅.
• re2c:label:prefix, re2c:labelprefix — specifies the prefix used for DFA state labels. The default is yy 🦋.
• re2c:label:start, re2c:startlabel — controls the generation of a block start label. The default value is zero, which means the start label is generated only if it is used. An integer value greater than zero forces generation of the start label even if it's unused by the lexer. A string value also forces start label generation and sets the label name to that specified string ✨. Note that this configuration applies only to the current block (it gets reset to default for the next block) — so don't expect it to stick around 💀.
• re2c:label:yyFillLabel — specifies the prefix of YYFILL labels used with re2c:eof and in storable state mode 🎀.
• re2c:label:yyloop — specifies the name of the label marking the start of the lexer loop with the --loop-switch option. The default is yyloop 🌸.
• re2c:label:yyNext — specifies the name of the optional label that follows the YYGETSTATE switch in storable state mode (enabled with re2c:state:nextlabel). The default is yyNext ⭐.
• re2c:lookahead, re2c:flags:lookahead — deprecated (see the deprecated --no-lookahead option). Yeah bestie, this one's old news 💀.
• re2c:monadic — if set to non-zero, the generated lexer will use monadic notation (this configuration is specific to Haskell) 🧚.
• re2c:nested-ifs, re2c:flags:nested-ifs, re2c:flags:s — same as the --nested-ifs option, but can be configured on a per-block basis 💅.
• re2c:posix-captures, re2c:flags:posix-captures, re2c:flags:P — bestie this is literally the --posix-captures option but you can flip it per-block instead of globally. one block can have it, another doesn't, they don't have to agree 💅
• re2c:posix-captvars — same deal as the --posix-captvars option, just configurable per-block. per-block era fr.
• re2c:tags, re2c:flags:tags, re2c:flags:T — yup, this is the --tags option again, but per-block so you can be picky about it.
• re2c:tags:expression — sets the expression used for tag variables. default vibe is expressions of the form yyt<N>, which ngl can be annoying — like if your tag variables are defined as fields in a struct, that's not gonna fly. every occurrence of @@{tag} or @@ gets swapped out for the actual tag name. for example, re2c:tags:expression = "s.@@"; gives you expressions of the form s.yyt<N> in the generated code. also peep re2c:api:sigil configuration 👀
• re2c:tags:negative — the constant expression used for a negative tag value. typically that's -1 if tags are integer offsets into the input string, or a null pointer if they're pointers. pick your fighter.
• re2c:tags:prefix — the prefix for tag variable names. default is yyt. iconic, keep it or don't.
• re2c:sentinel — the sentinel symbol used for end-of-input checks (this is for when bounds checks are disabled with re2c:yyfill:enable = 0; and re2c:eof is not set). heads up: this configuration does not affect code generation — its whole job is to verify that the sentinel isn't allowed in the middle of a rule, and to make sure the lexer won't read past the end of the buffer. default value is -1 (and in that case re2py assumes the sentinel is zero, which is the most common case). only decimal numbers are recognized, so don't get creative 🎀
• re2c:state:abort — if you set this to a positive integer value, the default case in the generated state dispatch aborts program execution, and an explicit -1 case contains a transition to the start of the block. dramatic but fair.
• re2c:state:nextlabel — controls whether the YYGETSTATE switch is followed by a yyNext label (default value is zero, which means no label, sorry). alternatively you can use re2c:label:start to generate a specific start label, or an explicit getstate block to generate the YYGETSTATE switch separately from the lexer block. options on options, we love to see it ✨
• re2c:unsafe, re2c:flags:unsafe — same as the --no-unsafe option, but configurable per-block. if set to zero, it suppresses the generation of unsafe wrappers around YYPEEK. default is non-zero (wrappers are generated). oh and this configuration is specific to Rust, so don't come for me if it doesn't do anything in your C project 🦀
• re2c:YYBACKUP, re2c:define:YYBACKUP — defines the generic API primitive YYBACKUP.
• re2c:YYBACKUPCTX, re2c:define:YYBACKUPCTX — defines the generic API primitive YYBACKUPCTX.
• re2c:YYCONDTYPE, re2c:define:YYCONDTYPE — defines the API primitive YYCONDTYPE.
• re2c:YYCTYPE, re2c:define:YYCTYPE — defines the API primitive YYCTYPE.
• re2c:YYCTXMARKER, re2c:define:YYCTXMARKER — defines the API primitive YYCTXMARKER.
• re2c:YYCURSOR, re2c:define:YYCURSOR — defines the API primitive YYCURSOR.
• re2c:YYDEBUG, re2c:define:YYDEBUG — defines the API primitive YYDEBUG.
• re2c:YYFILL, re2c:define:YYFILL — defines the API primitive YYFILL.
• re2c:YYFILL@len, re2c:define:YYFILL@len — the sigil used for argument substitution in the YYFILL definition. defaults to @@. this one overrides the more generic re2c:api:sigil configuration, so it wins the argument 💅
• re2c:YYFILL:naked, re2c:define:YYFILL:naked — overrides the more generic re2c:api:style configuration for YYFILL. zero value corresponds to free-form API style.
• re2c:YYFN — defines the API primitive YYFN.
• re2c:YYINPUT — defines the API primitive YYINPUT.
• re2c:YYGETCOND, re2c:define:YYGETCONDITION — defines the API primitive YYGETCOND.
• re2c:YYGETCOND:naked, re2c:define:YYGETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYGETCOND. zero value = free-form API style.
• re2c:YYGETSTATE, re2c:define:YYGETSTATE — defines the API primitive YYGETSTATE.
• re2c:YYGETSTATE:naked, re2c:define:YYGETSTATE:naked — overrides the more generic re2c:api:style configuration for YYGETSTATE. zero value = free-form API style.
• re2c:YYGETACCEPT, re2c:define:YYGETACCEPT — defines the API primitive YYGETACCEPT.
• re2c:YYLESSTHAN, re2c:define:YYLESSTHAN — defines the generic API primitive YYLESSTHAN.
• re2c:YYLIMIT, re2c:define:YYLIMIT — defines the API primitive YYLIMIT.
• re2c:YYMARKER, re2c:define:YYMARKER — defines the API primitive YYMARKER.
• re2c:YYMTAGN, re2c:define:YYMTAGN — defines the generic API primitive YYMTAGN.
• re2c:YYMTAGP, re2c:define:YYMTAGP — defines the generic API primitive YYMTAGP.
• re2c:YYPEEK, re2c:define:YYPEEK — defines the generic API primitive YYPEEK.
• re2c:YYRESTORE, re2c:define:YYRESTORE — defines the generic API primitive YYRESTORE.
• re2c:YYRESTORECTX, re2c:define:YYRESTORECTX — defines the generic API primitive YYRESTORECTX.
• re2c:YYRESTORETAG, re2c:define:YYRESTORETAG — defines the generic API primitive YYRESTORETAG.
• re2c:YYSETCOND, re2c:define:YYSETCONDITION — defines the API primitive YYSETCOND.
• re2c:YYSETCOND@cond, re2c:define:YYSETCONDITION@cond — the sigil used for argument substitution in the YYSETCOND definition. default value is @@. overrides the more generic re2c:api:sigil configuration.
• re2c:YYSETCOND:naked, re2c:define:YYSETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYSETCOND. zero value = free-form API style. bestie the pattern is the pattern.
• re2c:YYSETSTATE, re2c:define:YYSETSTATE — defines the API primitive YYSETSTATE.
• re2c:YYSETSTATE@state, re2c:define:YYSETSTATE@state — the sigil used for argument substitution in the YYSETSTATE definition. default value is @@. overrides the more generic re2c:api:sigil configuration.
• re2c:YYSETSTATE:naked, re2c:define:YYSETSTATE:naked — overrides the more generic re2c:api:style configuration for YYSETSTATE. zero value = free-form API style.
• re2c:YYSETACCEPT, re2c:define:YYSETACCEPT — defines the API primitive YYSETACCEPT.
• re2c:YYSKIP, re2c:define:YYSKIP — defines the generic API primitive YYSKIP.
• re2c:YYSHIFT, re2c:define:YYSHIFT — defines the generic API primitive YYSHIFT.
• re2c:YYCOPYMTAG, re2c:define:YYCOPYMTAG — defines the generic API primitive YYCOPYMTAG.
• re2c:YYCOPYSTAG, re2c:define:YYCOPYSTAG — defines the generic API primitive YYCOPYSTAG.
• re2c:YYSHIFTMTAG, re2c:define:YYSHIFTMTAG — defines the generic API primitive YYSHIFTMTAG.
• re2c:YYSHIFTSTAG, re2c:define:YYSHIFTSTAG — defines the generic API primitive YYSHIFTSTAG.
• re2c:YYSTAGN, re2c:define:YYSTAGN — defines the generic API primitive YYSTAGN.
• re2c:YYSTAGP, re2c:define:YYSTAGP — defines the generic API primitive YYSTAGP.
• re2c:yyaccept, re2c:variable:yyaccept — defines the API primitive yyaccept.
• re2c:yybm, re2c:variable:yybm — defines the API primitive yybm.
• re2c:yybm:hex, re2c:variable:yybm:hex — if set to nonzero, bitmaps for the --bit-vectors option are generated in hexadecimal format. default is zero (bitmaps come out in decimal format, like a normal person).
• re2c:yych, re2c:variable:yych — defines the API primitive yych.
• re2c:yych:emit, re2c:variable:yych:emit — if set to zero, the yych definition is not generated. default is non-zero.
• re2c:yych:conversion, re2c:variable:yych:conversion — if set to non-zero, re2py automatically generates a conversion to YYCTYPE every single time yych is read. default is zero (no conversion).
• re2c:yych:literals, re2c:variable:yych:literals — the form of literals that yych is matched against. possible values are: char (character literals in single quotes, non-printable ones use escape sequences that start with backslash), hex (hexadecimal integers) and char_or_hex (a mixture of both — character literals for printable characters and hexadecimal integers for the others). pick the one that matches your aesthetic 🌈
• re2c:yyctable, re2c:variable:yyctable — defines the API primitive yyctable.
• re2c:yynmatch, re2c:variable:yynmatch — defines the API primitive yynmatch.
• re2c:yypmatch, re2c:variable:yypmatch — defines the API primitive yypmatch.
• re2c:yytarget, re2c:variable:yytarget — defines the API primitive yytarget.
• re2c:yystable, re2c:variable:yystable — Deprecated. that's it. that's the whole description. it's giving retired icon 💀
• re2c:yystate, re2c:variable:yystate — defines the API primitive yystate.
• re2c:yyfill, re2c:variable:yyfill — defines the API primitive yyfill.
• re2c:yyfill:check — if set to zero, suppresses the generation of the pre-YYFILL check for the number of input characters (that's the YYLESSTHAN definition in the generic API and the YYLIMIT-based comparison in the C pointer API). default is non-zero, meaning the check is generated. stay safe out here 🫶
• re2c:yyfill:enable — if set to zero, suppresses the generation of YYFILL (together with the check). you'd use this when the whole input fits into one piece of memory (so there's no need for buffering) and your end-of-input checks don't rely on the YYFILL checks (for example if you're using a sentinel character). use warnings (-W option) and the re2c:sentinel configuration to verify that the generated lexer can't read past the end of input.

bestie please actually do the verifying. this is the "turn off the safety rails" knob and it does not come with a warning label drawn on it 😭

default is non-zero (YYFILL is enabled).

• re2c:yyfill:parameter — if set to zero, suppresses the generation of the parameter passed to YYFILL. that parameter is the minimum number of characters that must be supplied. defaults to non-zero (the parameter is generated). this configuration can be overridden with re2c:YYFILL:naked or re2c:api:style. no cap, overrides stack.
• re2c:yyfn:sep — the separator used in YYFN elements (defaults to semicolon).
• re2c:yyfn:throw — the exceptions thrown by the YYFN function (defaults to empty, which means no exceptions — a very chill function).

---

🌸 Regular expressions 🌸

okay so re2py uses the following syntax for regular expressions, and honestly it's all pretty intuitive — here's the whole menu 💖

• "foo" — a case-sensitive string literal. exact match only, no games.
• 'foo' — a case-insensitive string literal. chaotic neutral, it'll match any casing.
• [a-xyz], [^a-xyz] — a character class (possibly negated). the second one is the "everything except these" version.
• . — any character except newline. yes, except newline. don't @ me.
• R \ S — the difference of character classes R and S.
• R* — zero or more occurrences of R. the classic.
• R+ — one or more occurrences of R. the classic but needy.
• R? — optional R. might be there, might not, it's a vibe.
• R{n} — repetition of R exactly n times. no more, no less, periodt.
• R{n,} — repetition of R at least n times. Slay, keep going.
• R{n,m} — repetition of R from n to m times.
• (R) — just R; the parentheses are there to override precedence. If submatch extraction is enabled, (R) is a capturing or a non-capturing group depending on the --invert-captures option.
• (!R) — if submatch extraction is enabled, (!R) is a non-capturing or a capturing group depending on the --invert-captures option.
• R S — concatenation: R followed by S.
• R | S — alternative: R or S.
• R / S — lookahead: R followed by S, but S is not consumed.
• name — regular expression defined as name (or literal string "name" in Flex compatibility mode).
• {name} — regular expression defined as name in Flex compatibility mode.
• @stag — an s-tag: saves the last input position at which @stag matches in a variable named stag.
• #mtag — an m-tag: saves all input positions at which #mtag matches in a variable named mtag.
• $ — end of input.

Character classes and string literals may contain these escape sequences, bestie: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and \Uhhhhhhhh.

🎬 Actions

Here is the list of predefined actions supported by re2py — these are the little behind-the-scenes hooks, iykyk:

• !entry code — the entry action binds a user-defined block of code to the start state of the current finite state machine. If start conditions are used, the entry action can be set individually for each condition. This action may be used to perform initialization, e.g. to save the start location of a lexeme.
• !pre_rule code — the pre-rule action prepends a user-defined block of code to semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to factor out the common part of all semantic actions (e.g. saving the end location of a lexeme).
• !post_rule code — the post-rule action appends a user-defined block of code to semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to emit trap statements that guard against unintended control flow.

📜 Directives

Here is the full list of directives supported by re2py:

• !use:name ; — an in-block use directive that merges a previously defined rules block with the specified name into the current block. Named definitions, configurations and rules of the referenced block are added to the current ones. Conflicts between overlapping rules and configurations are resolved in the usual way: the first rule takes priority, and the latest configuration overrides the preceding ones. One exception is the special rules *, $ and <!> for which a block-local definition always takes priority. A use directive can be placed anywhere inside of a block, and multiple use directives are allowed.
• !include file ; — this directive is the same as include block: it inserts file contents verbatim in place of the directive.

🔌 Program interface

The generated code interfaces with the outer program with the help of primitives, collectively referred to as the API. Which primitives should be defined for a particular program depends on multiple factors, including the complexity of regular expressions, input representation, buffering and the use of various features. All the necessary primitives should be defined by the user in the form of macros, functions, variables or any other suitable form that makes the generated code syntactically and semantically correct.

⚠️ Real talk: re2py does not (and cannot) check the definitions, so if anything is missing or defined incorrectly, the generated program may have compile-time or run-time errors. This manual provides examples of API definitions in the most common cases.

re2py has three API flavors that define the core set of primitives used by a program:

• Simple API — this is the default API for the Python backend. It consists of the following primitives: YYINPUT (which should be defined as a sequence of code units, e.g. a string) and YYCURSOR, YYMARKER, YYCTXMARKER, YYLIMIT (which should be defined as indices in YYINPUT).
• Record API — Record API is useful in cases when lexer state must be stored in a class. It is enabled with --api record option or re2c:api = record configuration. This API consists of a variable yyrecord (the name can be overridden with re2c:yyrecord) that should be defined as a class with attributes yyinput, yycursor, yymarker, yyctxmarker, yylimit (only the fields used by the generated code need to be defined, and their names can be configured).
• Generic API — this is the most flexible API. It is enabled with --api generic option or re2c:api = generic configuration. It contains primitives for generic operations: YYPEEK, YYSKIP, YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE, YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG, YYLESSTHAN, YYEND.

Here is a full list of API primitives that may be used by the generated code in order to interface with the outer program.

• YYCTYPE — the type of the input characters (code units). For ASCII, EBCDIC and UTF-8 encodings it should be 1-byte unsigned integer. For UTF-16 or UCS-2 it should be 2-byte unsigned integer. For UTF-32 it should be 4-byte unsigned integer.
• YYCURSOR — an l-value that stores the current input position (a pointer or an integer offset in YYINPUT). Initially YYCURSOR should point to the first input character, and later it is advanced by the generated code. When a rule matches, YYCURSOR position is the one after the last matched character.
• YYLIMIT — an r-value that stores the end of input position (a pointer or an integer offset in YYINPUT). Initially YYLIMIT should point to the position after the last available input character. It is not changed by the generated code. The lexer compares YYCURSOR to YYLIMIT in order to determine if there are enough input characters left.
• YYMARKER — an l-value that stores the position of the latest matched rule (a pointer or an integer offset in YYINPUT). It is used to restore the YYCURSOR position if the longer match fails and the lexer needs to rollback. Initialization is not needed.
• YYCTXMARKER — an l-value that stores the position of the trailing context (a pointer or an integer offset in YYINPUT). No initialization is needed. YYCTXMARKER is needed only if the lookahead operator / is used.
• YYFILL — a generic API primitive with one variable len. YYFILL should provide at least len more input characters or fail. If re2c:eof is used, then len is always 1 and YYFILL should always return to the calling function; zero return value indicates success. If re2c:eof is not used, then YYFILL return value is ignored and it should not return on failure. The maximum value of len is YYMAXFILL.
• YYFN — a primitive that defines function prototype in --recursive-functions code model. Its value should be an array of one or more strings, where each string contains two or three components separated by the string specified in re2c:fn:sep configuration (typically a semicolon). The first array element defines function name and return type (empty for a void function). Subsequent elements define function arguments: first, the expression for the argument used in function body (usually just a name); second, argument type; third, an optional formal parameter (it defaults to the first component - usually both the argument and the parameter are the same identifier).
• YYINPUT — an r-value that stores the current input character sequence (string, buffer, etc.).
• YYMAXFILL — an integral constant equal to the maximum value of the argument to YYFILL. It can be generated with a max block.
• YYLESSTHAN — a generic API primitive with one variable len. It should be defined as an r-value of boolean type that equals true if and only if there are less than len input characters left.
• YYEND — a generic API primitive with no variables. It should be defined as an r-value of boolean type that equals true if and only if the logical end of input has been reached (excluding any padding or sentinel symbols). YYEND is used to implement $ symbol in regular expressions. It differs from YYLESSTHAN, which is used to ensure that the lexer won't read past the end of buffer.
• YYPEEK — a generic API primitive with no variables. It should be defined as an r-value of type YYCTYPE that is equal to the character at the current input position.
• YYSKIP — a generic API primitive that should advance the current input position by one code unit.
• YYBACKUP — a generic API primitive that should save the current input position (to be restored with YYRESTORE later).
• YYRESTORE — a generic API primitive that should restore the current input position to the value saved by YYBACKUP.
• YYBACKUPCTX — a generic API primitive that should save the current input position as the position of the trailing context (to be restored with YYRESTORECTX later).
• YYRESTORECTX — a generic API primitive that should restore the trailing context position saved with YYBACKUPCTX.
• YYRESTORETAG — a generic API primitive with one variable tag that should restore the trailing context position to the value of tag.
• YYSTAGP — a generic API primitive with one variable tag, where tag can be a pointer or an offset in YYINPUT (see submatch extraction section for details). YYSTAGP should set tag to the current input position.
• YYSTAGN — a generic API primitive with one variable tag, where tag can be a pointer or an offset in YYINPUT (see submatch extraction section for details). YYSTAGN should set tag to a value that represents non-existent input position.
• YYMTAGP — a generic API primitive with one variable tag. YYMTAGP should append the current position to the submatch history of tag (see the submatch extraction section for details.)
• YYMTAGN — a generic API primitive with one variable, tag. It should append a value that means "there's no input position here" to the submatch history of tag (peep the submatch extraction section for the deets). It's basically telling the tag "bestie, nothing happened here" 💅
• YYSHIFT — a generic API primitive with one variable, shift, that should shift the current input position by shift characters. And yeah, shift can be negative, so it slides both ways 🦋
• YYCOPYSTAG — a generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side s-tag variable rhs into the left-hand-side s-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs. Copy-paste but make it fashion ✨
• YYCOPYMTAG — a generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side m-tag variable rhs into the left-hand-side m-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs. Same energy, different tag family 🎀
• YYSHIFTSTAG — a generic API primitive with two variables, tag and shift, that should shift tag by shift code units (and yes, shift may be negative).
• YYSHIFTMTAG — a generic API primitive with two variables, tag and shift, that should shift the latest value in the history of tag by shift code units (shift may be negative). Only the newest one gets moved, not the whole roster 👑
• YYMAXNMATCH — an integral constant equal to the maximal number of POSIX capturing groups in a rule. It's generated with a maxnmatch block.
• YYCONDTYPE — the type of the condition enum. It can be generated either with a conditions block or the --header option.
• YYGETACCEPT — a primitive with one variable, var, that stores the numeric selector of the accepted rule. For most languages this primitive has a default definition that reads from var.
• YYSETACCEPT — a primitive with two variables: var (an l-value that stores the numeric selector of the accepted rule) and val (the value of that selector). For most languages this primitive has a default definition that assigns var to val.
• YYGETCOND — an r-value of type YYCONDTYPE that equals the current condition identifier.
• YYSETCOND — a primitive with one variable, cond, that should set the current condition identifier to cond.
• YYGETSTATE — an r-value of integer type that equals the current lexer state. Heads up: it should be initialized to -1 🙈
• YYSETSTATE — a primitive with one variable, state, that should set the current lexer state to state.
• YYDEBUG — this primitive is generated only with the -d, --debug-output option. Its whole purpose is adding logging to the generated code (a typical YYDEBUG definition is a print statement). YYDEBUG statements get generated in every state and have two variables: state (either a DFA state index or -1) and symbol (the current input symbol). It's giving "narrator voiceover" energy 🫶
• yyaccept — an l-value of unsigned integral type that stores the number of the latest matched rule. You only need a user definition for this one when you're using the --storable-state option.
• yybm — a table containing compressed bitmaps for up to 8 transitions (it's used with the --bitmaps option). The table contains 256 elements and is indexed by 1-byte code units. Every 8-bit element combines boolean values for up to 8 transitions. The k-Th bit of the n-th element is true iff the n-th code unit is in the range of the k-th transition. The whole idea here is to replace a whole pile of if branches or switch cases with one single bit check in the table — lowkey genius, highkey faster ⚡
• yych — an l-value of type YYCTYPE that stores the current input character. A user definition is necessary only with the -f, --storable-state option.
• yyctable — a jump table generated for the initial condition dispatch (it's enabled with a combination of the --conditions and --computed-gotos options).
• yyfill — an l-value that stores the result of the YYFILL call. This can be necessary for pure functional languages, where YYFILL is a monadic function with a complex return value. Functional girlies, this one's for you 🧚
• yynmatch — an l-value of unsigned integral type that stores the number of POSIX capturing groups in the matched rule. Used only with the -P, --posix-captures option.
• yypmatch — an array of l-values that hold the tag values corresponding to the capturing parentheses in the matching rule. The array length must be at least yynmatch * 2 (usually YYMAXNMATCH * 2 is a good choice). Used only with the -P, --posix-captures option.
• yystable — deprecated. That's it, that's the whole description. She's retired, leave her alone 💀
• yystate — an l-value used with the --loop-switch option to store the current DFA state.
• yytarget — a jump table that contains jump targets (label addresses) for all transitions from a state. This table is local to each state. Generation of yytarget tables is enabled with the --computed-gotos option.

---

✨ Options ✨

Okay so real talk: some of these options have corresponding configurations, and the others are global — meaning you literally cannot change them after re2c starts reading the input file. No take-backs, periodt. Also, debug options generally require building re2c in debug configuration. And the internal options? Those are just for experimenting with the algorithms used in re2c — nerdy little playground, we love to see it 🌈

• -?, --help, -h — show the help message. When in doubt, ask for help, that's growth 💖
• --api <simple | record | generic> — specify the API the generated code uses to interface with used-defined code. Pick simple for simple cases when there's no need for buffer refilling and storing lexer state. Pick record when lexer state needs to be stored in a record (struct, class, etc.). Pick generic for complex cases when the other two APIs just aren't flexible enough for you, main character 🎀
• --bit-vectors, -b — optimize conditional jumps using bit masks. This option implies --nested-ifs.
• --captures, --leftmost-captures — enable submatch extraction with leftmost greedy capturing groups. The result gets collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule.
• --captvars, --leftmost-captvars — enable submatch extraction with leftmost greedy capturing groups. This time the result gets collected into the variables yytl<k>, yytr<k> for the k-th capturing group.
• --case-insensitive — treat single-quoted and double-quoted strings as case-insensitive. Case? Never heard of her 💅
• --case-inverted — invert the meaning of single-quoted and double-quoted strings: treat single-quoted strings as case-sensitive and double-quoted strings as case-insensitive. Chaos, but make it intentional 🦋
• --case-ranges — collapse consecutive cases in a switch statement into a range of the form low ... high. This syntax is a C/C++ language extension supported by compilers like GCC, Clang and Tcc. The main advantage over using single cases is smaller generated code and faster generation time — although for some compilers like Tcc it also results in smaller binary size. Shrinkflation we actually support 👑
• --computed-gotos, -g — optimize conditional jumps using the non-standard "computed goto" extension (which, fair warning, must be supported by your compiler). re2py only generates jump tables in complex cases with a lot of conditional branches. That complexity threshold can be configured with the cgoto:threshold configuration. Relative offsets can be enabled with the cgoto:relative configuration. This option implies --bit-vectors.
• --computed-gotos-relative — similar to --computed-gotos but generates relative offsets for jump tables instead (which must be supported by the compiler). This option implies --computed-gotos.
• --conditions, --start-conditions, -c — enable support of Flex-like "conditions": multiple interrelated lexers within one block. This is the alternative to manually specifying different re2py blocks connected with goto or function calls. One block, many moods ✨
• --depfile FILE — write dependency information to FILE in the form of a Makefile rule <output-file> : <input-file> [include-file ...]. This lets you track build dependencies when include blocks/directives are in play, so that updating include files triggers regeneration of the output file. Heads up: this option depends on the --output option.
• --ebcdic, --ecb, -e — generate a lexer that reads input in EBCDIC encoding. re2py assumes the character range is 0 -- 0xFF and the character size is 1 byte. Vintage encoding, respect the elders 🫶
• --empty-class <match-empty | match-none | error> — define the way re2py treats empty character classes. With match-empty (the default) the empty class matches empty input, which is illogical but backwards-compatible — yes, we know, don't come for us. With match-none the empty class always fails to match. With error the empty class raises a compilation error.
• --encoding-policy <fail | substitute | ignore> — define the way re2py treats Unicode surrogates. With fail, re2py aborts with an error when a surrogate is encountered. With substitute, re2py silently replaces surrogates with the error code point 0xFFFD. With ignore (the default), re2py treats surrogates as normal code points. Ngl, the Unicode standard says standalone surrogates are invalid, but real-world libraries and programs behave in different ways — so here we are 😭
• --flex-syntax, -F — partial support for Flex syntax. In this mode named definitions don't need the equal sign and the terminating semicolon, and when used they must be surrounded with curly braces. Names without curly braces are treated as double-quoted strings. Curly braces, the new flex 💅
• --goto-label — use the "goto/label" code model: encode the DFA in the form of labeled code blocks connected with goto transitions across blocks. Only supported for languages that have a goto statement, so if yours doesn't, this one's not for you 🥺
• --header, --type-header, -t HEADER — generate a HEADER file. What goes in the file can be specified using the special blocks header:on and header:off. If conditions are used, the generated header will have a condition enum automatically appended to it — unless there's an explicit conditions block, in which case she already has her own, thank you very much 👑
• -I PATH — add PATH to the list of locations used when searching for include files. This option is useful in combination with an include block or directive. re2py looks for FILE in the directory of the parent file and in the include locations specified with the -I option. Home base first, then the other spots 🧚
• --input <default | custom> — ok so this one is a deprecated alias for --api, bestie, don't @ me. default lines up with simple (and yeah it is the default for most backends… just not all of them, cute right?) and custom lines up with generic.
• --input-encoding <ascii | utf8> — tells re2py how to read your regexes. With ascii (the actual default) it treats input as ASCII-encoded: any sequence of code units is a sequence of standalone 1-byte characters, no thoughts, head empty. With utf8 it treats input as UTF8-encoded and ✨recognizes multibyte characters✨.
• --invert-captures — flips the whole vibe of capturing vs non-capturing groups. Normally (...) is capturing and (! ...) is non-capturing. Turn this on and it's the reverse: (! ...) captures and (...) doesn't. Iconic, confusing, love that for us 💅.
• --lang <none | c | d | go | haskell | java | js | ocaml | python | rust | swift | v | zig> — picks your target language 🎯. The supported ones are C, D, Go, Haskell, Java, JS, OCaml, Python, Rust, Swift, V and Zig (and yes you can add more via user-defined syntax files — see --syntax). none disables the default syntax configs so the target language is just… undefined. Main character of nothing.
• --location-format <gnu | msvc> — controls how locations show up in messages. gnu prints them like 'filename:line:column: ...' and msvc prints them like 'filename(line,column) ...'. The default is gnu, no cap.
• --loop-switch — uses the "loop/switch" code model: the DFA gets encoded as a loop over a switch statement where each individual state is a switch case 🔁. State lives in a variable called yystate, and transitions between states update yystate to the case label of the destination state and then keep executing at the head of the loop. She's got a system.
• --computed-continue — needs the --loop-switch code model to work, so don't even try it solo. It uses computed continue statements to pick the next switch case directly, which unlocks jump-table optimization similar to --computed-gotos in the goto/label code model. Efficiency slay.
• --nested-ifs, -s — uses nested if statements instead of switch statements in conditional jumps. Usually gives you more efficient code with non-optimizing compilers. Little branches, big wins 🌿.
• --no-debug-info, -i — don't output line directives. Lowkey useful when the generated code lives in a version control system, because otherwise you get huge autogenerated diffs on the tiniest changes and honestly 😭 no thank you.
• --no-generation-date — suppresses the date in the generated file. Timestamp? Never met her 🎀.
• --no-version — suppresses version output in the generated file.
• --no-unsafe — don't generate the unsafe wrapper over YYPEEK. Heads up, this one's Rust-specific 🦀. For performance reasons YYPEEK should avoid bounds-checking since the lexer already does end-of-input checks in a much more efficient way. You can hand over a safe YYPEEK definition, or one that's unsafe only in release builds — and in that case --no-unsafe saves you from warnings about redundant unsafe blocks. Very thoughtful of you tbh.
• --output, -o OUTPUT — specifies the OUTPUT file. That's it. That's the whole thing 🫶.
• --posix-captures, -P — turns on submatch extraction with POSIX-style capturing groups. The result gets collected into an array yybmatch with capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule. Bookkeeping queen 👑.
• --posix-captvars — also enables submatch extraction with POSIX-style capturing groups, but this time the result is collected into the variables yytl<k> and yytr<k> for the k-th capturing group. Same energy, different variables ✨.
• --recursive-functions — uses a code model based on co-recursive functions, where every single DFA state is its own separate function that may call other state-functions or itself. Function inception, but make it lexer 🌀.
• --reusable, -r — deprecated since version 2.2 (reusable blocks are allowed by default now, so she's retired, let her rest 🙈).
• --skeleton, -S — ignores your user-defined interface code and generates a self-contained "skeleton" program 🦴. On top of that it generates input files with strings derived from the regular grammar and compressed match results that get used to verify "skeleton" behavior on all inputs. Basically it's the bug-hunting option: useful for finding bugs in optimizations and code generation. Small catch — this option is supported only for C.
• --storable-state, -f — generates a lexer that can store its inner state 💾. Perfect for push-model lexers that get stopped by an outer program when there's not enough input, then resumed once more input shows up. In this mode you should additionally define the YYGETSTATE and YYSETSTATE primitives, and the variables yych, yyaccept and state should be part of the stored lexer state. Pack it up, resume later, no stress.
• --syntax FILE — loads configurations from the specified FILE and applies them on top of the default syntax file. Note that FILE can define only a few configurations (if it's being used to amend the default syntax file), or it can define a whole new language backend — and in that latter case it's recommended to use the --lang none option. Either way, do your thing 🦋.
• --tags, -T — enables submatch extraction with tags 🏷️.
• --ucs2, --wide-chars, -w — generates a lexer that reads UCS2-encoded input. re2py assumes the character range is 0 -- 0xFFFF and character size is 2 bytes. Heads up: this option implies --nested-ifs.
• --utf8, --utf-8, -8 — generates a lexer that reads input in UTF-8 encoding. re2py assumes the character range is 0 -- 0x10FFFF and character size is 1 byte. ✨
• --utf16, --utf-16, -x — generates a lexer that reads UTF16-encoded input. re2py assumes the character range is 0 -- 0x10FFFF and character size is 2 bytes. This option implies --nested-ifs.
• --utf32, --unicode, -u — generates a lexer that reads UTF32-encoded input. re2py assumes the character range is 0 -- 0x10FFFF and character size is 4 bytes. This option implies --nested-ifs.
• --verbose — outputs a short message when it succeeds 🎉. A little crumb of validation.
• --vernum, -V — shows version information in MMmmpp format (major, minor, patch). Numbers only, cryptic on purpose.
• --version, -v — shows version information.
• --single-pass, -1 — deprecated. Does absolutely nothing (single pass is the default now). Ghost of options past 💀.
• --debug-output, -d — emits YYDEBUG invocations in the generated code, which is useful for tracing lexer execution 🔍.
• --dump-adfa — debug option: outputs the DFA after tunneling (in .dot format).
• --dump-cfg — debug option: outputs the control flow graph of tag variables (in .dot format).
• --dump-closure-stats — debug option: outputs statistics on the number of states in closure 📊.
• --dump-dfa-det — debug option: outputs the DFA immediately after determinization (in .dot format).
• --dump-dfa-min — debug option: outputs the DFA after minimization (in .dot format).
• --dump-dfa-tagopt — debug option: outputs the DFA after tag optimizations (in .dot format).
• --dump-dfa-tree — debug option: outputs the DFA under construction with states represented as tag history trees (in .dot format) 🌳.
• --dump-dfa-raw — debug option: outputs the DFA under construction with expanded state-sets (in .dot format).
• --dump-interf — debug option: outputs the interference table produced by liveness analysis of tag variables.
• --dump-nfa — debug option: outputs the NFA (in .dot format).
• --emit-dot, -D — instead of normal output, generates the lexer graph in .dot format. You can then convert that to an image with the help of Graphviz, like dot -Tpng -odfa.png dfa.dot 🎨. Art.
• --dfa-minimization <moore | table> — internal option: the DFA minimization algorithm re2py uses. moore is the Moore algorithm (and it's the default). table is the "table filling" algorithm. Both should produce the same DFA up to states relabeling — table filling is simpler and much slower and serves as a reference implementation. Slow but honest, we respect her 🐌.
• --eager-skip — internal option: makes the generated lexer advance the input position eagerly, immediately after reading the input symbol. This changes the default behavior, where the input position is advanced lazily — after transition to the next state. Eager vs lazy, the eternal vibe check ⚡.
• --no-lookahead — internal option, deprecated. It used to enable the TDFA(0) algorithm. Unlike TDFA(1), TDFA(0) doesn't use one-symbol lookahead, and it applies register operations to the incoming transitions rather than the outgoing ones. Benchmarks showed TDFA(0) is less efficient than TDFA(1), so… yeah. Retired for a reason 💅.
• --no-optimize-tags — internal option: suppresses optimization of tag variables. Useful for debugging, and only for debugging 🐛.
• --posix-closure <gor1 | gtop> — internal option: specifies the shortest-path algorithm used to construct epsilon-closure with POSIX disambiguation semantics. gor1 (the default) stands for the Goldberg-Radzik algorithm, and gtop stands for the "global topological order" algorithm. Pick your fighter 🌈.
• --posix-prectable <complex | naive> — internal option: specifies the algorithm used to compute the POSIX precedence table. The complex algorithm computes the precedence table in one traversal of the tag history tree and has quadratic complexity in the number of TNFA states — and it's the default. The naive algorithm has worst-case cubic complexity in the number of TNFA states, but it's much simpler than complex and may be slightly faster in non-pathological cases. Naive but thriving 🍓.
• --stadfa — internal option, deprecated. It used to enable the staDFA algorithm, which differs from TDFA in that register operations are placed in states rather than on transitions. Benchmarks showed staDFA is less efficient than TDFA. Gone but not forgotten 🕯️.
• --fixed-tags <none | toplevel | all> — okay so this one's an internal option, bestie, it's giving "devs only" energy 💅. It says whether the fixed-tag optimization should hit all tags (all), none of them (none), or just the ones in toplevel concatenation (toplevel). The default is all. So what even is a "fixed" tag? It's a tag chilling within a fixed distance of some other tag (called the "base" 👑). In those cases you only need to track the base tag, and the fixed tag's value can be computed as the base tag's value plus a static offset. For tags under alternative or repetition, you also gotta check if the base tag has a no-match value (in that case the fixed tag should also be set to no-match, disregarding the offset). For tags in top-level concatenation the check is not needed, because they always match. Slay.

⚠️ Warnings ⚠️

Warnings can be invividually enabled, disabled and turned into an error. (Yes, the original says "invividually" — we're keeping that typo era, no cap 😭)

• -W — Turn on all warnings. Periodt.
• -Werror — Turn warnings into errors. Note that this option alone doesn't turn on any warnings; it only affects those warnings that have been turned on so far or will be turned on later. So it's like a vibe check for warnings that already exist or show up later ✨.
• -W<warning> — Turn on warning.
• -Wno-<warning> — Turn off warning.
• -Werror-<warning> — Turn on warning and treat it as an error (this implies -W<warning>).
• -Wno-error-<warning> — Don't treat this particular warning as an error. This doesn't turn off the warning itself. It's still there, just not dramatic 💅.
• -Wcondition-order — Warn if the generated program makes implicit assumptions about condition numbering. One should use either --header option or conditions block to generate a mapping of condition names to numbers and then use the autogenerated condition names. Basically don't assume the order, bestie.
• -Wempty-character-class — Warn if a regular expression contains an empty character class. Trying to match an empty character class makes no sense: it should always fail. However, for backwards compatibility reasons re2py permits empty character classes and treats them as empty strings. Use the --empty-class option to change the default behavior. It's giving "old habits die hard" 🙈.
• -Wmatch-empty-string — Warn if a rule is nullable (matches an empty string). If the lexer runs in a loop and the empty match is unintentional, the lexer may unexpectedly hang in an infinite loop. So lowkey important 😭.
• -Wswapped-range — Warn if the lower bound of a range is greater than its upper bound. The default behavior is to silently swap the range bounds. No drama, just swapped.
• -Wundefined-control-flow — Warn if some input strings cause undefined control flow in the lexer (the faulty patterns are reported). This is a dangerous and common mistake. It can be easily fixed by adding the default rule * which has the lowest priority, matches any code unit, and always consumes a single code unit. Bestie, please add the * rule, it's iconic for a reason 🔥.
• -Wunreachable-rules — Warn about rules that are shadowed by other rules and will never match. They're literally sitting there doing nothing 💀.
• -Wdeprecated-eof_rule — Warn about standalone end of input rules $ that will be broken by the future changes and require fixing. At the moment these rules take precedence when conflicting with other rules, but after the introduction of generalized end of input symbol $ precedence order will change and these rules will become shadowed by other rules. So yeah, future you will thank you for fixing it.
• -Wuseless-escape — Warn if a symbol is escaped when it shouldn't be. By default, re2py silently ignores such escapes, but this may as well indicate a typo or an error in the escape sequence. It's giving "why is there a backslash here" energy.
• -Wnondeterministic-tags — Warn if a tag has n-th degree of nondeterminism, where n is greater than 1.
• -Wsentinel-in-midrule — Warn if the sentinel symbol occurs in the middle of a rule — this may cause reads past the end of buffer, crashes or memory corruption in the generated lexer. This warning is only applicable if the sentinel method of checking for the end of input is used. It is set to an error if re2c:sentinel configuration is used. That's a big yikes, bestie 🚨.
• -Wundefined-syntax-config — Warn if the syntax file specified with --syntax option is missing definitions of some configurations. This helps to maintain user-defined syntax files: if a new release adds configurations, old syntax file will raise a warning, and the user will be notified. If some configurations are unused and do not need a definition, they should be explicitly set to <undefined>. It's like a little reminder note from the compiler 🫶.

📁 Syntax files 📁

Support for different languages in re2c is based on the idea of syntax files. A syntax file is a configuration file that defines syntax of the target language — not the whole language, but a small part of it that is used by the generated code. Syntax files make re2c very flexible, but they should not be used as a replacement for re2c: configurations: their purpose is to define syntax of the target language, not to customize one particular lexer. All supported languages have default syntax files that are part of the distribution (see include/syntax subdirectory); they are also embedded in the re2py binary. Users may provide a custom syntax file that overrides a few configurations for one of supported languages, or they may choose to redefine all configurations (in that case --lang none option should be used). Syntax files contain configurations of four different kinds: feature lists, language configurations, inplace configurations and code templates.

🌟 Feature lists 🌟

A few list configurations define various features supported by a given backend, so that re2py may give a clear error if the user tries to enable an unsupported feature:

• supported_apis — A list of supported APIs with possible elements simple, record, generic.
• supported_api_styles — A list of supported API styles with possible elements functions, free-form.
• supported_code_models — A list of supported code models with possible elements goto-label, loop-switch, recursive-functions.
• supported_targets — A list of supported codegen targets with possible elements code, dot, skeleton.
• supported_features — A list of supported features with possible elements nested-ifs, bitmaps, computed-gotos, case-ranges, monadic, unsafe, tags, captures, captvars, computed-continue.

🧠 Language configurations 🧠

A few boolean configurations describe features of the target language that affect re2py parser and code generator:

• semicolons — Non-zero if the language uses semicolons after statements.
• backtick_quoted_strings — Non-zero if the language has backtick-quoted strings.
• single_quoted_strings — Non-zero if the language has single-quoted strings.
• indentation_sensitive — Non-zero if the language is indentation sensitive.
• wrap_blocks_in_braces — Non-zero if compound statements must be wrapped in curly braces.

🔧 Inplace configurations 🔧

Syntax files define initial values of all re2c: configurations, as they may differ for different languages. See configurations section for a full list of all inplace configurations and their meaning.

🎀 Code templates 🎀

Code templates define syntax of the target language. They are written in a simple domain-specific language with the following formal grammar:

code-template ::
      name '=' code-exprs ';'
    | CODE_TEMPLATE ';'
    | '<undefined>' ';'

code-exprs ::
      <EMPTY>
    | code-exprs code-expr

code-expr ::
      STRING
    | VARIABLE
    | optional
    | list

optional ::
      '(' CONDITIONAL '?' code-exprs ')'
    | '(' CONDITIONAL '?' code-exprs ':' code-exprs ')'

list ::
      '[' VARIABLE ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER '}' ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER ',' NUMBER '}' ':' code-exprs ']'

A code template is a sequence of string literals, variables, optional elements and lists, or a reference to another code template, or a special value <undefined>. Variables are placeholders that are substituted during code generation phase. List variables are special: when expanding list templates, re2py repeats expressions the right hand side of the column a few times, each time replacing occurrences of the list variable with a value specific to this repetition. Lists have optional bounds (negative values are counted from the end, e.g. -1 means the last element). Conditional names start with a dot. Both conditionals and variables may be either local (specific to the given code template) or global (allowed in all code templates). When re2py reads syntax file, it checks that each code template uses only the variables and conditionals that are allowed in it.

For example, the following code template defines if-then-else construct for a C-like language:

code:if_then_else =
    [branch{0}: topindent "if " cond " {" nl
        indent [stmt: stmt] dedent]
    [branch{1:-1}: topindent "} else" (.cond ? " if " cond) " {" nl
        indent [stmt: stmt] dedent]
    topindent "}" nl;

Here branch is a list variable: branch{0} expands to the first branch (which is special, as there is no else part), branch{1:-1} expands to all remaining branches (if any). stmt is also a list variable: [stmt: stmt] is a nested list that expands to a list of statements in the body of the current branch. topindent, indent, dedent and nl are global variables, and .cond is a local conditional (their meaning is described below). This code template could produce the following code:

if x {
    // do something
} else if y {
    // do something else
} else {
    // don't do anything
}

Here's a list of all code templates supported by re2py with their local variables and conditionals. Note that a particular definition may, but does not have to use local variables and conditionals. Any unused code templates should be set to <undefined>.

• code:var_local — Declaration or definition of a local variable. Supported variables: type (the type of the variable), name (its name) and init (initial value, if any). Conditionals: .init (true if there is an initializer).
• code:var_global — literally the same as code:var_local except it lives at top-level. That's it, that's the whole vibe 💅
• code:const_local — a local constant, iconic behavior only. Supported variables: type (the type of the constant), name (its name) and init (initial value).
• code:const_global — same as code:const_local, but top-level, because constants deserve main character energy too ✨
• code:array_local — a local array (table), no cap. Supported variables: type (the type of array elements), name (array name), size (its size), row (a list variable that does not itself produce any code, but expands the list expression as many times as there are rows in the table) and elem (a list variable that expands to all table elements in the current row — it's meant to be nested in the row list 🪆). Supported conditional: .const (true if the array is immutable).
• code:array_global — same as code:array_local, except, you guessed it, top-level 🌈
• code:array_elem — a reference to one element of an array (table). Supported variables: array (the name of the array) and index (index of the element).
• code:enum — definition of an enumeration (it may be defined using a special language construct for enumerations, or simply as a few standalone constants). Supported variables are type (user-defined enumeration type or type of the constants), elem (list variable that expands to the name of each member) and init (initializer for each member). Conditionals: .init (true if there is an initializer).
• code:enum_elem — an enumeration element (a member of a user-defined enumeration type or a name of a constant, depending on how code:enum is defined). Supported variables are name (the name of the element) and type (its type).
• code:assign — an assignment statement, the moment we've all been waiting for 🎀 Supported variables are lhs (left hand side) and rhs (right hand side).
• code:type_int — signed integer type. Classic.
• code:type_uint — unsigned integer type. Also classic, just more optimistic 💖
• code:type_yybm — type of elements in the yybm table.
• code:type_yytarget — type of elements in the yytarget table.
• code:type_yyctable — type of elements in the yyctable table. (Yes bestie, there are three of these and they are all real.)
• code:cmp_eq — the operator "equals".
• code:cmp_ne — the operator "not equals".
• code:cmp_lt — the operator "less than".
• code:cmp_gt — the operator "greater than"
• code:cmp_le — the operator "less or equal"
• code:cmp_ge — the operator "greater or equal"
• code:if_then_else — an if-then-else statement with one or more branches, because sometimes there's more than two paths and that's valid 🦋 Supported variables: branch (a list variable that does not itself produce any code, but expands the list expression as many times as there are branches), cond (condition of the current branch) and stmt (a list variable that expands to all statements in the current branch). Conditionals: .cond (true if the current branch has a condition), .many (true if there's more than one branch).
• code:if_then_else_oneline — a specialization of code:if_then_else for the case when all branches have one-line statements. If this is <undefined>, code:if_then_else is used instead.
• code:switch — a switch statement with one or more cases 🔀 Supported variables: expr (the switched-on expression) and case (a list variable that expands to all cases-groups with their code blocks).
• code:switch_cases — a group of switch cases that maps to a single code block. Supported variables are case (a list variable that expands to all cases in this group) and stmt (a list variable that expands to all statements in the code block.
• code:switch_cases_oneline — a specialization of code:switch_cases for the case when the code block consists of a single one-line statement. If this is <undefined>, code:switch_cases is used instead.
• code:switch_case_range — a single switch case that covers a range of values (possibly consisting of a single value — a range of one, slay). Supported variable: val (a list variable that expands to all values in the range). Supported conditionals: .many (true if there's more than one value in the range) and .char_literals (true if this is a switch on character literals — some languages provide special syntax for this case).
• code:switch_case_default — the default switch case. The fallback friend 🫶
• code:loop — a loop that runs forever (unless interrupted from the loop body). Yes, forever-forever 😭 Supported variables: label (loop label), val (initial state), var (state variable), state (a list variable that expands to all state names) and stmt (a list variable that expands to all statements in the loop body or the current state).
• code:continue — continue statement. Supported variables: label (label from which to continue execution), val (value passed to a labeled switch) and var (state variable).
• code:goto — goto statement. Supported variables: label (label of the jump target). We love a directed jump ✨
• code:cgoto — computed goto statement. Supported variables: array (the table containing computed goto information), index (index of the element in the table) and base (base label, only used if .cgoto.relative is true).
• code:cgoto:data — initializer expression for a single element in the computed goto table. Supported variables: label (the label that is used to initialize the current element), type (underlying type of the elements in the table) and base (base label — only used if .cgoto.relative is true).
• code:fndecl — function declaration, the whole point lowkey 👑 Supported variables: name (function name), type (return type), throw (exceptions thrown by this function, maps to re2c:yyfn:throw configuration), arg (a list variable that does not itself produce code, but expands the list expression as many times as there are function arguments), argname (name of the current argument), argtype (type of the current argument). Conditional: .type (true if this is a non-void function).
• code:fndef — like code:fndecl, but used for function definitions, so it has one additional list variable stmt that expands to all statements in the function body. It's giving glow-up 💅
• code:fncall — function call statement. Supported variables: name (function name), retval (l-value where the return value is stored, if any) and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if return value needs to be saved).
• code:tailcall — tail call statement. Supported variables: name (function name), and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if this is a non-void function).
• code:recursive_functions — program body with the --recursive-functions code model. Supported variables: fn (a list variable that does not itself produce any code, but expands the list expression as many times as there are functions), fndecl (declaration of the current function) and fndef (definition of the current function). Recursion: breathtaking 🌸
• code:fingerprint — the fingerprint at the top of the generated output file. Supported variables: ver (re2py version that was used to generate this) and date (generation date). Little signature, big presence 🎀
• code:line_info — the format of line directives (if this is set to <undefined>, no directives are generated at all, zero, none). Supported variables: line (line number) and file (filename).
• code:abort — a statement that aborts program execution. Dramatic but necessary 💀
• code:yydebug — YYDEBUG statement, possibly specialized for different APIs. Supported variables: YYDEBUG, yyrecord, yych (map to the corresponding re2c: configurations), state (DFA state number).
• code:yypeek — YYPEEK statement, possibly specialized for different APIs. Supported variables: YYPEEK, YYCTYPE, YYINPUT, YYCURSOR, yyrecord, yych (map to the corresponding re2c: configurations). Conditionals: .cast (true if re2c:yych:conversion is set to non-zero).
• code:yyskip — YYSKIP statement, possibly specialized for different APIs. Supported variables: YYSKIP, YYCURSOR, yyrecord (map to the corresponding re2c: configurations).
• code:yybackup — YYBACKUP statement, possibly specialized for different APIs. Supported variables: YYBACKUP, YYCURSOR, YYMARKER, yyrecord (map to the corresponding re2c: configurations).
• code:yybackupctx — YYBACKUPCTX statement, possibly specialized for different APIs. Supported variables: YYBACKUPCTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the corresponding re2c: configurations).
• code:yyskip_yypeek — combined code:yyskip and code:yypeek statement (defaults to code:yyskip followed by code:yypeek). Combo meal 🍓
• code:yypeek_yyskip — combined code:yypeek and code:yyskip statement (defaults to code:yypeek followed by code:yyskip). Same combo, opposite order, still slaying.
• code:yyskip_yybackup — combined code:yyskip and code:yybackup statement (defaults to code:yyskip followed by code:yybackup).
• code:yybackup_yyskip — combined code:yybackup and code:yyskip statement (defaults to code:yybackup followed by code:yyskip).
• code:yybackup_yypeek — combined code:yybackup and code:yypeek statement (defaults to code:yybackup followed by code:yypeek).
• code:yyskip_yybackup_yypeek — combined code:yyskip, code:yybackup and code:yypeek statement (defaults to code:yyskip followed by code:yybackup followed by code:yypeek). The full trio, the holy trinity, we love to see it 🔥
• yybackup_yypeek_yyskip — the combined yybackup, yypeek and yyskip statement, all bundled up (defaults to yybackup followed by yypeek followed by yyskip). ✨
• yyrestore — the YYRESTORE statement, possibly specialized for different APIs. Supported variables: YYRESTORE, YYCURSOR, YYMARKER, yyrecord (map to the corresponding re2c: configurations).
• yyrestorectx — the YYRESTORECTX statement, possibly specialized for different APIs. Supported variables: YYRESTORECTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the corresponding re2c: configurations).
• yyrestoretag — the YYRESTORETAG statement, possibly specialized for different APIs. Supported variables: YYRESTORETAG, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), tag (the name of the tag variable used to restore position).
• yyshift — the YYSHIFT statement, possibly specialized for different APIs. Supported variables: YYSHIFT, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), offset (the number of code units to shift the current position). 🦋
• yyshiftstag — the YYSHIFTSTAG statement, possibly specialized for different APIs. Supported variables: YYSHIFTSTAG, yyrecord, negative (map to the corresponding re2c: configurations), tag (the tag variable that needs to be shifted), offset (the number of code units to shift). Conditionals: .nested (true if this is a nested tag — and in that case its value may equal re2c:tags:negative, which should not be shifted, bestie, don't do it).
• yyshiftmtag — the YYSHIFTMTAG statement, possibly specialized for different APIs. Supported variables: YYSHIFTMTAG (maps to the corresponding re2c: configuration), tag (the tag variable that needs to be shifted), offset (the number of code units to shift).
• yystagp — the YYSTAGP statement, possibly specialized for different APIs. Supported variables: YYSTAGP, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), tag (the tag variable that should be updated).
• yymtagp — the YYMTAGP statement, possibly specialized for different APIs. Supported variables: YYMTAGP (maps to the corresponding re2c: configuration), tag (the tag variable that should be updated).
• yystagn — the YYSTAGN statement, possibly specialized for different APIs. Supported variables: YYSTAGN, negative, yyrecord (map to the corresponding re2c: configurations), tag (the tag variable that should be updated).
• yymtagn — the YYMTAGN statement, possibly specialized for different APIs. Supported variables: YYMTAGN (maps to the corresponding re2c: configuration), tag (the tag variable that should be updated).
• yycopystag — the YYCOPYSTAG statement, possibly specialized for different APIs. Supported variables: YYCOPYSTAG, yyrecord (map to the corresponding re2c: configurations), lhs, rhs (the left and right hand side tag variables of the copy operation).
• yycopymtag — the YYCOPYMTAG statement, possibly specialized for different APIs. Supported variables: YYCOPYMTAG, yyrecord (map to the corresponding re2c: configurations), lhs, rhs (the left and right hand side tag variables of the copy operation).
• yygetaccept — the YYGETACCEPT statement, possibly specialized for different APIs. Supported variables: YYGETACCEPT, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yyaccept configuration).
• yysetaccept — the YYSETACCEPT statement, possibly specialized for different APIs. Supported variables: YYSETACCEPT, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yyaccept configuration) and val (the numeric value of the accepted rule).
• yygetcond — the YYGETCOND statement, possibly specialized for different APIs. Supported variables: YYGETCOND, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yycond configuration).
• yysetcond — the YYSETCOND statement, possibly specialized for different APIs. Supported variables: YYSETCOND, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yycond configuration) and val (the numeric condition identifier).
• yygetstate — the YYGETSTATE statement, possibly specialized for different APIs. Supported variables: YYGETSTATE, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yystate configuration).
• yysetstate — the YYSETSTATE statement, possibly specialized for different APIs. Supported variables: YYSETSTATE, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yystate configuration) and val (the state number).
• yylessthan — the YYLESSTHAN statement, possibly specialized for different APIs. Supported variables: YYLESSTHAN, YYCURSOR, YYLIMIT, yyrecord (map to the corresponding re2c: configurations), need (the number of code units to check against). Conditional: .many (true if the need is more than one).
• yyend — the YYEND expression, possibly specialized for different APIs. Supported variables: YYEND, YYCURSOR, YYLIMIT.
• yybm_filter — the condition that filters out yych values that aren't covered by the yybm table (used with the --bitmaps option). Supported variable: yych (maps to the re2c:yych configuration). 🎀
• yybm_match — the format of the yybm table check (generated with the --bitmaps option). Supported variables: yybm, yych (map to the corresponding re2c: configurations), offset (the offset in the yybm table that needs to be added to yych) and mask (the bit mask that should be applied to the table entry to retrieve the boolean value that needs to be checked).
• yytarget_filter — the condition that filters out yych values that aren't covered by the yytarget table (used with the --computed-gotos option). Supported variable: yych (maps to the re2c:yych configuration).

---

Okay so next up, here's the full guest list of global variables that are allowed in syntax files 💖:

• nl — a newline. That's it. That's the whole vibe.
• indent — a variable that doesn't produce any code but has the side-effect of increasing the indentation level.
• dedent — a variable that doesn't produce any code but has the side-effect of decreasing the indentation level.
• topindent — the indentation string for the current statement. The indentation level is tracked and automatically updated by the code generator, no cap 🫶.

---

And here's the global conditionals list, also allowed in syntax files ✨:

• .api.simple — true if the simple API is used (--api simple or re2c:api = simple).
• .api.generic — true if the generic API is used (--api generic or re2c:api = generic).
• .api.record — true if the record API is used (--api record or re2c:api = record).
• .api_style.functions — true if the function-like API style is used (re2c:api-style = functions).
• .api_style.freeform — true if the free-form API style is used (re2c:api-style = free-form).
• .case_ranges — true if the case ranges feature is enabled (--case-ranges or re2c:case-ranges = 1).
• .cgoto.relative — true if the relative form of computed goto is used (--computed-gotos-relative or re2c:cgoto:relative = 1).
• .computed_continue — true if the loop/switch model uses labeled-switch transitions (--computed-continue).
• .code_model.goto_label — true if the code model based on goto/label is used (--goto-label).
• .code_model.loop_switch — true if the code model based on loop/switch is used (--loop-switch).
• .code_model.recursive_functions — true if the code model based on recursive functions is used (--recursive-function).
• .date — true if the generated fingerprint should contain the generation date.
• .loop_label — true if re2py-generated loops must have a label (re2c:label:yyloop is set to a nonempty string).
• .monadic — true if the generated code should be monadic (re2c:monadic = 1). This is only relevant for pure functional languages, iykyk 🧚
• .start_conditions — true if start conditions are enabled (--start-conditions).
• .storable_state — true if storable state is enabled (--storable-state).
• .unsafe — true if re2py should use "unsafe" blocks in order to generate faster code (--unsafe, re2c:unsafe = 1). This is only relevant for languages that have the "unsafe" feature.
• .version — true if the generated fingerprint should contain the re2py version.
• .yyfill.enable — true if YYFILL is enabled (re2c:yyfill:enable = 1).
• .yyfn.throw — true if the re2c:yyfn:throw configuration is defined to a nonempty string.

---

🛑 HANDLING THE END OF INPUT 🛑

Okay bestie, real talk: one of the main problems for the lexer is knowing when to stop. Like, same, honestly 😭. There are a few terminating conditions, and here they are:

• the lexer may match some rule (including the default rule *) and come to a final state ✨
• the lexer may fail to match any rule and come to a default state 💀
• the lexer may reach the end of input 🦋

Okay so real talk: the first two conditions end the lexer in a totally “natural” way — it lands in a state with no outgoing transitions, and the matching just automatically stops. 💅 The third condition, end of input, is a different vibe entirely: it can pop off in any state, and the lexer has to be able to handle it. Checking for end of input interrupts the normal lexer workflow and adds conditional branches to the generated program, so bestie, we lowkey need to minimize how many of those checks happen. re2py supports a few different methods for handling the end of input. Which one you pick depends on the complexity of the regular expressions, whether you need buffering, performance considerations, and other factors. Here’s the list, no cap:

• Sentinel — this method eliminates the need for end-of-input checks altogether. It’s simple and efficient, but it’s limited to the case where there’s a natural “sentinel” character that can never occur in valid input. That character may still occur in invalid input, but it should not be allowed by the regular expressions, except perhaps as the last character of a rule. The sentinel is appended at the end of input and serves as a stop signal: when the lexer reads this character, it’s either a syntax error or the end of input. In both cases the lexer should stop. This method is used if YYFILL is disabled with re2c:yyfill:enable = 0; and re2c:eof has the default value -1.
• Sentinel with bounds checks — this method is generic: it lets you handle any input without restrictions on the regular expressions. The idea is to reduce the number of end-of-input checks by performing them only on certain characters. Similar to the “sentinel” method, one of the characters is chosen as a “sentinel” and appended at the end of input. However, there’s no restriction on where the sentinel may occur (in fact, any character can be chosen for a sentinel). When the lexer reads this character, it additionally performs a bounds check. If the current position is within bounds, the lexer resumes matching and handles the sentinel as a regular character. Otherwise it invokes YYFILL (unless it is disabled). If more input is supplied, the lexer will rematch the last character and continue as if the sentinel wasn’t there. Otherwise it must be the real end of input, and the lexer stops. This method is used when re2c:eof has a non-negative value (it should be set to the numeric value of the sentinel). YYFILL is optional.
• Bounds checks with padding — this method is generic, and it may be faster than the “sentinel with bounds checks” method, but it’s also more complex. The idea is to partition DFA states into strongly connected components (SCCs) and generate a single check per SCC for enough characters to cover the longest non-looping path in this SCC. This reduces the number of checks, but there’s a problem with short lexemes at the end of input, as the check requires enough characters to cover the longest lexeme. This can be fixed by padding the input with a few fake characters that do not form a valid lexeme suffix (so that the lexer cannot match them). The length of padding should be YYMAXFILL, generated with a max block. If there is not enough input, the lexer invokes YYFILL which should supply at least the required number of characters or not return. This method is used if YYFILL is enabled and re2c:eof is -1 (this is the default configuration).
• Custom checks — generic API allows one to override basic operations like reading a character, which makes it possible to include the end-of-input checks as part of them. This approach is error-prone and should be used with caution. To use a custom method, enable generic API with --api custom or re2c:api = custom; and disable default bounds checks with re2c:yyfill:enable = 0; or re2c:yyfill:check = 0;.

The following subsections contain an example of each method. 🫶

Sentinel 🛡️

Bestie, this example uses a sentinel character to handle the end of input. The program counts space-separated words in a null-terminated string. The sentinel is null: it’s the last character of each input string, and it is not allowed in the middle of a lexeme by any of the rules (in particular, it is not included in character ranges, where it’s super easy to overlook it 👀). If a null occurs in the middle of a string, it’s a syntax error and the lexer will match the default rule *, but it won’t read past the end of input or crash (use the -Wsentinel-in-midrule warning and re2c:sentinel configuration to verify this, iykyk). The configuration re2c:yyfill:enable = 0; suppresses the generation of bounds checks and YYFILL invocations.

# re2py $INPUT -o $OUTPUT

# expect a null-terminated string
def lex(yyinput):
    yycursor = 0
    count = 0

    while True: %{
        re2c:yyfill:enable = 0;
        re2c:indent:top = 2;

        *      { return -1 }
        [\x00] { return count }
        [ ]+   { break }
        [a-z]+ {
            count += 1
            break
        }
    %}

assert lex(b"\0") == 0
assert lex(b"one two three\0") == 3
assert lex(b"f0ur\0") == -1

Sentinel with bounds checks 🎀

This example uses sentinel with bounds checks to handle the end of input (this method was added in version 1.2). The program counts space-separated single-quoted strings. The sentinel character is null, which is specified with the re2c:eof = 0; configuration. As in the sentinel method, null is the last character of each input string, but here it’s allowed in the middle of a rule (for example, 'aaa\0aa'\0 is valid input, but 'aaa\0 is a syntax error). Bounds checks are generated in each state that matches an input character, but they’re scoped to the branch that handles null. Bounds checks are of the form YYLIMIT <= YYCURSOR or YYLESSTHAN(1) with generic API. If the check condition is true, the lexer has reached the end of input and should stop (YYFILL is disabled with re2c:yyfill:enable = 0; as the input fits into one buffer — see the YYFILL with sentinel section for an example that uses YYFILL). Reaching the end of input opens three possibilities: if the lexer is in the initial state it will match the end-of-input rule $, otherwise it may fallback to a previously matched rule (including default rule *) or go to a default state, causing -Wundefined-control-flow.

# re2py $INPUT -o $OUTPUT

# expect a null-terminated string
def lex(yyinput):
    yycursor = 0
    yylimit = len(yyinput) - 1 # terminating null not included
    count = 0

    while True: %{
        re2c:yyfill:enable = 0;
        re2c:eof = 0;
        re2c:indent:top = 2;

        str = ['] ([^'\\] | [\\][^])* ['];

        *    { return -1 }
        $    { return count }
        [ ]+ { break }
        str  {
            count += 1
            break
        }
    %}

def test(str, count):
    # termunating null not included in `lim`
    assert count == lex(str)

test(b"\0", 0);
test(b"'qu\0tes' 'are' 'fine: \\'' \0", 3);
test(b"'unterminated\\'\0", -1)

Bounds checks with padding 🌸

This example uses bounds checks with padding to handle the end of input (this method is enabled by default). The program counts space-separated single-quoted strings. There’s a padding of YYMAXFILL null characters appended at the end of input, where the YYMAXFILL value is autogenerated with a max block. It’s not necessary to use null for padding — any characters can be used as long as they don’t form a valid lexeme suffix (in this example padding should not contain single quotes, as they may be mistaken for a suffix of a single-quoted string). There’s a “stop” rule that matches the first padding character (null) and terminates the lexer (note that it checks if null is at the beginning of padding, otherwise it’s a syntax error). Bounds checks are generated only in some states that are determined by the strongly connected components of the underlying automaton. Checks have the form (YYLIMIT - YYCURSOR) < n or YYLESSTHAN(n) with generic API, where n is the minimum number of characters that are needed for the lexer to proceed (it also means that the next bounds check will occur in at most n characters). If the check condition is true, the lexer has reached the end of input and will invoke YYFILL(n) that should either supply at least n input characters or not return. In this example YYFILL always fails and terminates the lexer with an error (which is fine because the input fits into one buffer). See the YYFILL with padding section for an example that refills the input buffer with YYFILL.

# re2py $INPUT -o $OUTPUT

%{max %}

def lex(yyinput):
    yycursor = 0
    yylimit = len(yyinput)
    count = 0

    while True: %{
        re2c:YYFILL = "return -1";
        re2c:indent:top = 2;

        str = ['] ([^'\\] | [\\][^])* ['];

        [\x00] {
            # check that it is the sentinel, not some unexpected null
            return count if yycursor == yylimit - YYMAXFILL + 1 else -1
        }
        str {
            count += 1
            break
        }
        [ ]+ { break }
        *    { return -1 }
    %}

def test(str, count):
    padded_str = str + (b"\0" * YYMAXFILL)
    assert lex(padded_str) == count

test(b"", 0)
test(b"'unterminated\\'", -1)
test(b"'qu\x00tes' 'are' 'fine: \\'' ", 3)
test(b"'unexpected \x00 null", -1)

Custom checks 🧚

This example uses a custom end-of-input handling method based on generic API. The program counts space-separated single-quoted strings. It’s the same as the sentinel example, except that the input is not null-terminated. To cover up for the absence of a sentinel character at the end of input, YYPEEK is redefined to perform a bounds check before it reads the next input character. This is inefficient because checks are done very often. If the check condition fails, YYPEEK returns the real character, otherwise it returns a fake sentinel character.

# re2py $INPUT -o $OUTPUT

# expect a string without terminating null
def lex(str):
    cur = 0
    lim = len(str)
    count = 0

    while True: %{
        re2c:api = generic;
        re2c:YYPEEK = "str[cur] if cur < lim else 0";
        re2c:YYSKIP = "cur += 1";
        re2c:yyfill:enable = 0;
        re2c:indent:top = 2;

• { return -1 } [\x00] { return count } [ ]+ { break } [a-z]+ { count += 1 break } %}

assert lex(b"") == 0 assert lex(b"one two three") == 3 assert lex(b"f0ur") == -1

# 💖 BUFFER REFILLING 💖

okay so bestie, sometimes you literally cannot map the whole input into memory at once — it's either too huge or it's streaming in (like from a socket, yk). the usual glow-up technique is: allocate a fixed-size memory buffer and process input in chunks that fit. when the current chunk is done, move it out and move new data in. but lowkey it's a bit more complex than that, because lexer state isn't just one input position — it's a whole set of interrelated positions:

- <pink>cursor</pink>: the next input character to be read (`YYCURSOR` in C pointer API or `YYSKIP`/`YYPEEK` in generic API)
- <pink>limit</pink>: the position after the last available input character (`YYLIMIT` in C pointer API, implicitly handled by `YYLESSTHAN` in generic API)
- <pink>marker</pink>: the position of the most recent match, if any (`YYMARKER` in default API or `YYBACKUP`/`YYRESTORE` in generic API)
- <pink>token</pink>: the start of the current lexeme (implicit in re2py API, as it is not needed for the normal lexer operation and can be defined and updated by the user)
- <pink>context marker</pink>: the position of the trailing context (`YYCTXMARKER` in C pointer API or `YYBACKUPCTX`/`YYRESTORECTX` in generic API)
- <pink>tag variables</pink>: submatch positions (defined with `stags` and `mtags` blocks and generic API primitives `YYSTAGP`/`YYSTAGN`/`YYMTAGP`/`YYMTAGN`)

not all of these are used in every case, but if they are used, they must be updated by `YYFILL`. all active positions live in the segment between `token` and `cursor`, so everything between buffer start and `token` can be discarded, the segment from `token` up to `limit` should be moved to the beginning of the buffer, and the free space at the end of the buffer should be filled with new data. and to avoid frequent `YYFILL` calls, it's best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). the details of `YYFILL` implementation are slightly different depending on which EOF handling method you're using: the EOF rule case is somewhat simpler than the bounds-checking-with-padding case. also note that if you use the `-f` `--storable-state` option, `YYFILL` has slightly different semantics (described in the section about storable state).

## ✨ YYFILL with sentinel ✨

if you're using the EOF rule, `YYFILL` is a function-like primitive that takes no arguments and returns a value which is checked against zero. `YYFILL` gets invoked when the condition `YYLIMIT <= YYCURSOR` holds in the C pointer API, or `YYLESSTHAN()` in the generic API. a non-zero return value means `YYFILL` has failed. a successful `YYFILL` call must supply at least one character and adjust the input positions accordingly. `limit` must always be set to one after the last input position in the buffer, and the character at the `limit` position must be the sentinel symbol specified by `re2c:eof` configuration. the pictures below show the relative locations of input positions in the buffer before and after the `YYFILL` call (the sentinel symbol is marked with `#`, and the second picture shows the case when there isn't enough input to fill the whole buffer).

<-- shift -->

-A------------B---------C-------------D#-----------E->

buffer token marker limit, cursor

-A------------B---------C-------------D------------E#->

buffer, marker cursor limit token

<-- shift -->

-A------------B---------C-------------D#--E (EOF)

buffer token marker limit, cursor

-A------------B---------C-------------D---E#........

buffer, marker cursor limit token

here's an example of a program that reads the input file `input.txt` in chunks of 4096 bytes and uses the EOF rule.

re2py $INPUT -o $OUTPUT

from enum import Enum import os

BUFSIZE = 4096

class State: def __init__(self, fname): self.file = open(fname, "rb") self.yyinput = bytearray(BUFSIZE) self.yylimit = BUFSIZE - 1 # exclude terminating null self.yycursor = self.yylimit self.yymarker = self.yylimit self.token = self.yylimit self.eof = False

def __del__(self): self.file.close()

class Status(Enum): OK = 0 EOF = 1 LONG_LEXEME = 2

def fill(st): if st.eof: return Status.EOF

# Error: lexeme too long. In real life could reallocate a larger buffer. if st.token < 1: return Status.LONG_LEXEME

# Shift buffer contents (discard everything up to the current token). st.yyinput = st.yyinput[st.token:st.yylimit] st.yycursor -= st.token; st.yymarker -= st.token; st.yylimit -= st.token; st.token = 0;

# Fill free space at the end of buffer with new data from file. bytes = st.file.read(BUFSIZE - st.yylimit - 1) # -1 for sentinel if not bytes: st.eof = True # end of file else: st.yylimit += len(bytes); st.yyinput += bytes

st.yyinput += b'\0' # append sentinel

return Status.OK

def lex(yyrecord, count): while True: yyrecord.token = yyrecord.yycursor %{ re2c:api = record; re2c:define:YYFILL = "fill(yyrecord) == Status.OK"; re2c:eof = 0; re2c:indent:top = 2;

str = ['] ([^'\] | [\][^])* ['];

• { return -1 } $ { return count } [ ]+ { break } str { count += 1 break } %}

def main(): fname = "input"

# Prepare input file. f = open(fname, "w") for i in range(BUFSIZE): f.write("'qu\0tes' 'are' 'fine: \'' ") f.close()

# Run lexer on the prepared file. st = State(fname) assert lex(st, 0) == 3 * BUFSIZE

# Cleanup. os.remove(fname)

if __name__ == '__main__': main()

## ✨ YYFILL with padding ✨

in the default case (when the EOF rule is not used), `YYFILL` is a function-like primitive that accepts a single argument and does not return any value. `YYFILL` gets invoked when the condition `(YYLIMIT - YYCURSOR) < n` holds in the C pointer API, or `YYLESSTHAN(n)` in the generic API. the argument passed to `YYFILL` is the minimal number of characters that must be supplied. if it fails to do so, `YYFILL` must not return to the lexer (for that reason it's best implemented as a macro that returns from the calling function on failure). in the case of a successful `YYFILL` invocation, the `limit` position must be set either to one after the last input position in the buffer, or to the end of the `YYMAXFILL` padding (in case `YYFILL` has successfully read at least `n` characters, but not enough to fill the entire buffer). the pictures below show the relative locations of input positions in the buffer before and after the `YYFILL` invocation (`YYMAXFILL` padding on the second picture is marked with `#` symbols).

<-- shift --> <-- need -->

-A------------B---------C-----D-------E---F--------G->

buffer token marker cursor limit

-A------------B---------C-----D-------E---F--------G->

buffer, marker cursor limit token

<-- shift --> <-- need -->

-A------------B---------C-----D-------E-F (EOF)

buffer token marker cursor limit

-A------------B---------C-----D-------E-F###############

buffer, marker cursor limit token <- YYMAXFILL ->

here's an example of a program that reads the input file `input.txt` in chunks of 4096 bytes and uses bounds-checking with padding.

re2py $INPUT -o $OUTPUT

from enum import Enum import os

BUFSIZE = 4096 %{max %}

class State: def __init__(self, fname): self.file = open(fname, "rb") self.yyinput = bytearray(BUFSIZE) self.yylimit = BUFSIZE - YYMAXFILL self.yycursor = self.yylimit self.yymarker = self.yylimit self.token = self.yylimit self.eof = False

def __del__(self): self.file.close()

class Status(Enum): OK = 0 EOF = 1 LONG_LEXEME = 2

def fill(st, need): if st.eof: return Status.EOF

# Error: lexeme too long. In real life could reallocate a larger buffer. if st.token < need: return Status.LONG_LEXEME

# Shift buffer contents (discard everything up to the current token). st.yyinput = st.yyinput[st.token:st.yylimit] st.yycursor -= st.token; st.yymarker -= st.token; st.yylimit -= st.token; st.token = 0;

# Fill free space at the end of buffer with new data from file. bytes = st.file.read(BUFSIZE - st.yylimit - 1) # -1 for sentinel if not bytes: st.eof = True # end of file st.yylimit += YYMAXFILL st.yyinput += b"\0" * YYMAXFILL else: st.yylimit += len(bytes); st.yyinput += bytes

return Status.OK

def lex(yyrecord): count = 0 while True: yyrecord.token = yyrecord.yycursor %{ re2c:api = record; re2c:YYFILL = "if fill(yyrecord, @@) != Status.OK: return -1"; re2c:indent:top = 2;

str = ['] ([^'\] | [\][^])* ['];

[\x00] { # Check that it is the sentinel, not some unexpected null. return count if yyrecord.token == yyrecord.yylimit - YYMAXFILL else -1 } str { count += 1 break } [ ]+ { break }

• { return -1 } %}

and here’s the last little chunk of that example, bestie:

def main(): fname = "input"

# Prepare input file. f = open(fname, "w") for i in range(BUFSIZE): f.write("'qu\0tes' 'are' 'fine: \'' ") f.close()

# Run lexer on the prepared file. st = State(fname) assert lex(st) == 3 * BUFSIZE

# Cleanup. os.remove(fname)

if __name__ == '__main__': main()

---

# ✨ FEATURES ✨

## 🧩 Multiple blocks 🧩

<pink>okay so</pink> sometimes you don’t just need one lexer — you need a whole little ensemble cast of *interrelated* lexers. Like when there’s a high-level state machine switching between lexer modes, and you’re sitting there like “bestie, this needs multiple connected `re2py` blocks.” That’s one way. The other option is start conditions. Both are valid, both are iconic. 💅

How the connections between blocks actually work is <purple>lowkey</purple> target-language dependent:

- In languages that have a `goto` statement (such as C/C++ and Go), you can put all blocks in one function, each prefixed with a label, and transition from one block to another with a simple `goto`. Clean, direct, slay.
- In languages that do **not** have `goto` (such as Rust), you gotta use a loop with a switch on a state variable — similar to the `yystate` loop/switch generated by `re2py` — or else wrap each block in a function and use function calls.

The example below is giving “parser for every base” realness: it parses binary, octal, decimal, and hexadecimal numbers, and each base gets its own block. The initial block determines the base and dispatches to the other blocks. Common configurations are defined in a separate block at the beginning of the program, and the other blocks inherit them. 🫶

re2py $INPUT -o $OUTPUT

class State: def __init__(self, str): self.yyinput = str self.yycursor = 0 self.yymarker = 0

Common re2c definitions shared between all functions.

%{ re2c:api = record; re2c:yyrecord = st; re2c:yyfill:enable = 0; re2c:indent:top = 2; %}

def parse_u32(str): st = State(str) %{local re2c:indent:top = 1;

'0b' / [01] { return parse_bin(st) } "0" { return parse_oct(st) } "" / [1-9] { return parse_dec(st) } '0x' / [0-9a-fA-F] { return parse_hex(st) }

• { return None } %}

def parse_bin(st): n = 0 while True: %{ [01] { n = n * 2 + (st.yyinput[st.yycursor - 1] - 48) break }

• { return n } %}

def parse_oct(st): n = 0 while True: %{ [0-7] { n = n * 8 + (st.yyinput[st.yycursor - 1] - 48) break }

• { return n } %}

def parse_dec(st): n = 0 while True: %{ [0-9] { n = n * 10 + (st.yyinput[st.yycursor - 1] - 48) break }

• { return n } %}

def parse_hex(st): n = 0 while True: %{ [0-9] { n = n * 16 + (st.yyinput[st.yycursor - 1] - 48) break } [a-f] { n = n * 16 + (st.yyinput[st.yycursor - 1] - 87) break } [A-F] { n = n * 16 + (st.yyinput[st.yycursor - 1] - 55) break }

• { return n } %}

assert parse_u32(b"\0") == None assert parse_u32(b"1234567890\0") == 1234567890 assert parse_u32(b"0b1101\0") == 13 assert parse_u32(b"0x7Fe\0") == 2046 assert parse_u32(b"0644\0") == 420 assert parse_u32(b"9999999999\0") == 9999999999

## 🌈 Start conditions 🌈

Start conditions are enabled with the `--start-conditions` option, and they give you a way to encode multiple interrelated automata inside the same `re2py` block. Very “one block, many vibes.” ✨

Each condition corresponds to a single automaton and has a unique name specified by the user, plus a unique internal number defined by `re2py`. Those numbers are how conditions get switched: the generated code uses `YYGETCOND` and `YYSETCOND` primitives to get the current condition or set it to the given number. Use a `conditions` block, the `--header` option, or `re2c:header` configuration to generate numeric condition identifiers. The `re2c:cond:enumprefix` configuration specifies the generated identifier prefix.

In condition mode, every rule must be prefixed with a list of comma-separated condition names in angle brackets, or a wildcard `<*>` to denote all conditions. The rule syntax is extended as follows:

- **`< condition-list > regular-expression code`** — a rule that is merged into every condition on *condition-list*. It matches *regular-expression* and executes the associated *code*.
- **`< condition-list > regular-expression => condition code`** — a rule that is merged into every condition on *condition-list*. It matches *regular-expression*, sets the current condition to *condition*, and executes the associated *code*.
- **`< condition-list > regular-expression :=> condition`** — a rule that is merged into every condition on *condition-list*. It matches *regular-expression* and immediately transitions to *condition* (there is no semantic action).
- **`< condition-list > !action code`** — a rule that binds *code* to the place defined by *action* in every condition on *condition-list* (see the actions section for the various types of actions).
- **`<! condition-list > code`** — a rule that prepends *code* to semantic actions of all rules for every condition on *condition-list*. This syntax is **deprecated**, and the `!pre_rule` action should be used instead (it does exactly the same thing).
- **`< > code`** — a rule that creates a special entry condition with number zero and name `"0"` that executes *code* before jumping to other conditions. This syntax is **deprecated**, and the `!entry` action should be used instead (it provides more fine-grained control, since the code can be specified on a per-condition basis, and one can jump directly to condition start without going through condition dispatch).
- **`< > => condition code`** — same as the previous rule, except that it sets the next condition.
- **`< > :=> condition`** — same as the previous rule, except that it has no associated code and immediately jumps to *condition*.

The code `re2py` generates for conditions depends on whether `re2py` uses the `goto`/label approach or the loop/switch approach to encode the automata.

In languages that have a `goto` statement (such as C/C++ and Go), conditions are naturally implemented as blocks of code prefixed with labels of the form `yyc_<cond>`, where *cond* is a condition name (the label prefix can be changed with `re2c:cond:prefix`). Transitions between conditions are implemented using `goto` and condition labels. Before all conditions, `re2py` generates an initial switch on `YYGETSTATE` that jumps to the start state of the current condition. The shortcut rules `:=>` bypass the initial switch and jump directly to the specified condition (`re2c:cond:goto` can be used to change the default behavior). Rules with semantic actions do **not** automatically jump to the next condition — that should be done by the user-defined action code. 👑

In languages that do **not** have `goto` (such as Rust), `re2py` reuses the `yystate` variable to store condition numbers. Each condition gets a numeric identifier equal to the number of its start state, and a switch between conditions is no different than a switch between DFA states of a single condition. There’s no need for a separate initial condition switch. (Since the same approach is used to implement storable states, `YYGETCOND`/`YYSETCOND` are redundant if both storable states and conditions are used.)

The program below uses start conditions to parse binary, octal, decimal, and hexadecimal numbers. There is a single block where each base has its own condition, and the initial condition is connected to all of them. The user-defined variable `cond` stores the current condition number; it is initialized to the number of the initial condition generated with the `conditions` block. 🦋

re2py $INPUT -o $OUTPUT -c

%{conditions %}

def parse_u32(yyinput): yycursor = 0 yycond = YYC_INIT num = 0

while True: %{ re2c:yyfill:enable = 0; re2c:indent:top = 2;

<INIT> '0b' / [01] :=> BIN <INIT> "0" :=> OCT <INIT> "" / [1-9] :=> DEC <INIT> '0x' / [0-9a-fA-F] :=> HEX <INIT> * { return None }

<BIN> [01] { num = num * 2 + (yyinput[yycursor - 1] - 48) break } <OCT> [0-7] { num = num * 8 + (yyinput[yycursor - 1] - 48) break } <DEC> [0-9] { num = num * 10 + (yyinput[yycursor - 1] - 48) break } <HEX> [0-9] { num = num * 16 + (yyinput[yycursor - 1] - 48) break } <HEX> [a-f] { num = num * 16 + (yyinput[yycursor - 1] - 87) break } <HEX> [A-F] { num = num * 16 + (yyinput[yycursor - 1] - 55) break }

<BIN, OCT, DEC, HEX> * { return num } %}

assert parse_u32(b"\0") == None assert parse_u32(b"1234567890\0") == 1234567890 assert parse_u32(b"0b1101\0") == 13 assert parse_u32(b"0x7Fe\0") == 2046 assert parse_u32(b"0644\0") == 420 assert parse_u32(b"9999999999\0") == 9999999999

### 💾 Storable state 💾

okay so with `--storable-state`, re2py can generate a lexer that stores its current state, returns to the caller, and later resumes operations exactly where it left off. The default mode of operation in re2py is a "pull" model, where the lexer "pulls" more input whenever it needs it. That can be a whole problem when input becomes available piece by piece — for example, if the lexer is invoked by the parser, or if the lexer program communicates via a socket protocol with some other program that must wait for a reply from the lexer before it transmits the next message. Storable state is literally made for these cases: it lets you generate lexers that work in a "push" model. When the lexer needs more input, it stores its state and returns to the caller. Later, when more input becomes available, the caller resumes the lexer exactly where it stopped. There are a few changes necessary compared to the "pull" model:

- Define `YYSETSTATE()` and `YYGETSTATE(state)` primitives.
- Define `yych`, `yyaccept` (if used) and state variables as a part of persistent lexer state. The state variable should be initialized to `-1`.
- `YYFILL` should return to the outer program instead of trying to supply more input. Return code should indicate that lexer needs more input.
- The outer program should recognize situations when lexer needs more input and respond appropriately.
- Optionally use `getstate` block to generate `YYGETSTATE` switch detached from the main lexer. This only works for languages that have `goto` (not in `--loop-switch` mode).
- Use `re2c:eof` and the sentinel with bounds checks method to handle the end of input. Padding-based method may not work because it is unclear when to append padding: the current end of input may not be the ultimate end of input, and appending padding too early may cut off a partially read greedy lexeme. Furthermore, due to high-level program logic getting more input may depend on processing the lexeme at the end of buffer (which already is blocked due to the end-of-input condition).

Here's a "push" model lexer that simulates reading packets from a socket. The lexer loops until it encounters the end of input and returns to the calling function. The calling function provides more input by "sending" the next packet and resumes lexing. This process stops when all the packets have been sent, or when there is an error.

re2py $INPUT -o $OUTPUT -f

from enum import Enum import os

Use a small buffer to cover the case when a lexeme doesn't fit.

In real world use a larger buffer.

BUFSIZE = 10 DEBUG = False

class State: def __init__(self, file): self.file = file self.yyinput = bytearray(BUFSIZE) self.yylimit = BUFSIZE - 1 # exclude terminating null self.yycursor = self.yylimit self.yymarker = self.yylimit self.token = self.yylimit self.yystate = -1

class Status(Enum): END = 0 READY = 1 WAITING = 2 BIG_PACKET = 3 BAD_PACKET = 4

def fill(st): # Error: lexeme too long. In real life could reallocate a larger buffer. if st.token < 1: return Status.BIG_PACKET

# Shift buffer contents (discard everything up to the current token). st.yyinput = st.yyinput[st.token:st.yylimit] st.yycursor -= st.token; st.yymarker -= st.token; st.yylimit -= st.token; st.token = 0;

# Fill free space at the end of buffer with new data from file. bytes = st.file.read(BUFSIZE - st.yylimit - 1) # -1 for sentinel if bytes: st.yylimit += len(bytes); st.yyinput += bytes

st.yyinput += b'\0' # append sentinel

return Status.READY

def lex(yyrecord, recv): while True: yyrecord.token = yyrecord.yycursor %{ re2c:api = record; re2c:YYFILL = "return Status.WAITING, recv"; re2c:eof = 0; re2c:indent:top = 2;

packet = [a-z]+[;];

• { return Status.BAD_PACKET, recv } $ { return Status.END, recv } packet { recv += 1 break } %}

def test(packets, expect): # Create a pipe (open the same file for reading and writing). fname = "pipe" fw = open(fname, "wb") fr = open(fname, "rb")

# Initialize lexer state st = State(fr)

# Main loop. The buffer contains incomplete data which appears packet by # packet. When the lexer needs more input it saves its internal state and # returns to the caller which should provide more input and resume lexing. send = 0 recv = 0 while True: status, recv = lex(st, recv)

if status == Status.END: if DEBUG: print("done: got {} packets".format(recv)) break

elif status == Status.WAITING: if DEBUG: print("waiting...");

if send < len(packets): if DEBUG: print("sent packet {}: {}".format(send, packets[send])) fw.write(packets[send]) fw.flush() send += 1

status = fill(st) if DEBUG: print("queue: '{}', status: {}".format(st.yyinput, status)) if status == Status.BIG_PACKET: if DEBUG: print("error: packet too big") break

assert status == Status.READY

else: assert status == Status.BAD_PACKET if DEBUG: print("error: ill-formed packet") break

# Check results. assert status == expect if status == Status.END: assert recv == send

# Cleanup: remove input file. fr.close() fw.close() os.remove(fname)

def main(): test([], Status.END) test([b"zero;", b"one;", b"two;", b"three;", b"four;"], Status.END) test([b"zer0;"], Status.BAD_PACKET) test([b"goooooooooogle;"], Status.BIG_PACKET)

if __name__ == '__main__': main()

### 🧩 Reusable blocks 🧩

Reusable blocks of the form `/*!rules:re2c[:<name>] ... */` or `%{rules[:<name>] ... %}` can be reused any number of times and combined with other re2py blocks. The `<name>` is optional. A rules block can be used in a use block or directive. The code for a rules block is generated at every point of use.

Use blocks are defined with `/*!use:re2c[:<name>] ... */` or `%{use[:<name>] ... %}`. The `<name>` is optional: if it's not specified, the associated rules block is the most recent one (whether named or unnamed). A use block can add named definitions, configurations and rules of its own. An important use case for use blocks is a lexer that supports multiple input encodings: the same rules block is reused multiple times with encoding-specific configurations (see the example below).

In-block use directive `!use:<name>;` can be used from inside of a re2py block. It merges the referenced block `<name>` into the current one. If some of the merged rules and configurations overlap with the previously defined ones, conflicts are resolved in the usual way: the earliest rule takes priority, and latest configuration overrides preceding ones. One exception are the special rules `*`, `$` and (in condition mode) `<!>`, for which a block-local definition overrides any inherited ones. Use directive allows one to combine different re2py blocks together in one block (see the example below).

Named blocks and in-block use directive were added in re2py version 2.2. Since that version reusable blocks are allowed by default (no special option is needed). Before version 2.2 reuse mode was enabled with `-r` `--reusable` option. Before version 1.2 reusable blocks could not be mixed with normal blocks.

### 🧪 Example of a `!use` directive 🧪

re2py $INPUT -o $OUTPUT

This example shows how to combine reusable re2c blocks: two blocks

('colors' and 'fish') are merged into one. The 'salmon' rule occurs

in both blocks; the 'fish' block takes priority because it is used

earlier. Default rule * occurs in all three blocks; the local (not

inherited) definition takes priority.

from enum import Enum

class Ans(Enum): COLOR = 1 FISH = 2 DUNNO = 3

%{rules:colors

• { raise "ah" } "red" | "salmon" | "magenta" { return Ans.COLOR } %}

%{rules:fish

• { raise "oh" } "haddock" | "salmon" | "eel" { return Ans.FISH } %}

def lex(yyinput): yycursor = 0 %{ re2c:yyfill:enable = 0; re2c:indent:top = 1;

!use:fish; !use:colors;

• { return Ans.DUNNO } // overrides inherited '*' rules %}

assert lex(b"salmon") == Ans.FISH assert lex(b"what?") == Ans.DUNNO

### 🌈 Example of a `/*!use:re2c ... */` block 🌈

re2py $INPUT -o $OUTPUT --input-encoding utf8

This example supports multiple input encodings: UTF-8 and UTF-32.

Both lexers are generated from the same rules block, and the use

blocks add only encoding-specific configurations.

%{rules re2c:yyfill:enable = 0; re2c:indent:top = 1;

"∀x ∃y" { return yycursor }

• { return None } %}

def lex_utf8(yyinput): yycursor = 0 %{use re2c:encoding:utf8 = 1; %}

def lex_utf32(yyinput): yycursor = 0 %{use re2c:encoding:utf32 = 1; %}

s8 = [0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79] assert lex_utf8(s8) == len(s8)

s32 = [0x2200, 0x78, 0x20, 0x2203, 0x79] assert lex_utf32(s32) == len(s32)

### 🔍 Submatch extraction 🔍

okay so re2py has **two** options for submatch extraction 👀

## 🏷️ Tags 🏷️

okay bestie, first up: you can use standalone tags of the form `@stag` or `#mtag`, where `stag` and `mtag` are arbitrary used-defined names. Tags are enabled with `-T` `--tags` option or `re2c:tags = 1` configuration. Semantically tags are position markers: they can be inserted anywhere in a regular expression, and they bind to the corresponding position (or multiple positions) in the input string. S-tags bind to the last matching position, and m-tags bind to a list of positions (they may be used in repetition subexpressions, where a single position in a regular expression corresponds to multiple positions in the input string). All tags should be defined by the user, either manually or with the help of `svars` and `mvars` blocks. If there is more than one way tags can be matched against the input, ambiguity is resolved using leftmost greedy disambiguation strategy.

## 🫶 Captures 🫶

The second option is to use capturing groups. They are enabled with `--captures` option or `re2c:captures = 1` configuration. There are two flavours for different disambiguation policies, `--leftmost-captures` (the default) is for leftmost greedy policy, and `--posix-captures` is for POSIX longest-match policy. In this mode all parenthesized subexpressions are considered capturing groups, and a bang can be used to mark non-capturing groups: `(! ... )`. With `--invert-captures` option or `re2c:invert-captures = 1` configuration the meaning of bang is inverted. The number of groups for the matching rule is stored in a variable `yynmatch` (the whole regular expression is group number zero), and submatch results are stored in `yypmatch` array. Both `yynmatch` and `yypmatch` should be defined by the user, and `yypmatch` size must be at least `[yynmatch * 2]`. Use `maxnmatch` block to define `YYMAXNMATCH`, a constant that equals to the maximum value of `yynmatch` among all rules.

## 💅 Captvars 💅

Another way to use capturing groups is the `--captvars` option or `re2c:captvars = 1` configuration. The only difference with `--captures` is in the way the generated code stores submatch results: instead of `yynmatch` and `yypmatch` re2py generates variables `yytl<k>` and `yytr<k>` for k-th capturing group (the user should declare these using an `svars` block). Captures with variables support two disambiguation policies: `--leftmost-captvars` or `re2c:leftmost-captvars = 1` for leftmost greedy policy (the default one) and `--posix-captvars` or `re2c:posix-captvars` for POSIX longest-match policy.

Under the hood all these options translate into tags and Tagged Deterministic Finite Automata with Lookahead. The core idea of TDFA is to minimize the overhead on submatch extraction. In the extreme, if there’re no tags or captures in a regular expression, TDFA is just an ordinary DFA. If the number of tags is moderate, the overhead is barely noticeable. The generated TDFA uses a number of tag variables which do not map directly to tags: a single variable may be used for different tags, and a tag may require multiple variables to hold all its possible values. Eventually ambiguity is resolved, and only one final variable per tag survives. Tag variables should be defined using `stags` or `mtags` blocks. If lexer state is stored, tag variables should be part of it. They also need to be updated by `YYFILL`.

S-tags support the following operations:

- save input position to an s-tag: `t = YYCURSOR` with C pointer API or a user-defined operation `YYSTAGP(t)` with generic API
- save default value to an s-tag: `t = NULL` with C pointer API or a user-defined operation `YYSTAGN(t)` with generic API
- copy one s-tag to another: `t1 = t2`

M-tags support the following operations:

- append input position to an m-tag: a user-defined operation `YYMTAGP(t)` with both default and generic API
- append default value to an m-tag: a user-defined operation `YYMTAGN(t)` with both default and generic API
- copy one m-tag to another: `t1 = t2`

S-tags can be implemented as scalar values (pointers or offsets). M-tags need a more complex representation, as they need to store a sequence of tag values. The most naive and inefficient representation of an m-tag is a list (array, vector) of tag values; a more efficient representation is to store all m-tags in a prefix-tree represented as array of nodes `(v, p)`, where `v` is tag value and `p` is a pointer to parent node.

Here is a simple example of using s-tags to parse semantic versions consisting of three numeric components: major, minor, patch (the latter is optional). See below for a more complex example that uses `YYFILL`.

re2py $INPUT -o $OUTPUT

from collections import namedtuple

SemVer = namedtuple('SemVer', 'major minor patch')

NONE = -1

def parse(yyinput): yycursor = 0 %{ re2c:yyfill:enable = 0; re2c:tags = 1; re2c:indent:top = 1;

num = [0-9]+;

@t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] { major = int(yyinput[t1:t2]) minor = int(yyinput[t3:t4]) patch = int(yyinput[t5:yycursor - 1]) if t5 != NONE else 0 return SemVer(major, minor, patch) }

• { return None } %}

assert parse(b"23.34\0") == SemVer(23, 34, 0) assert parse(b"1.2.99999\0") == SemVer(1, 2, 99999) assert parse(b"1.a\0") == None

Here is a more complex example of using s-tags with `YYFILL` to parse a file with newline-separated semantic versions. Tag variables are part of the lexer state, and they are adjusted in `YYFILL` like other input positions. Note that it is necessary for s-tags because their values are invalidated after shifting buffer contents. It may not be necessary in a custom implementation where tag variables store offsets relative to the start of the input string rather than the buffer, which may be the case with m-tags.

re2py $INPUT -o $OUTPUT

from collections import namedtuple from enum import Enum import os

BUFSIZE = 4096

SemVer = namedtuple('SemVer', 'major minor patch')

class State: def __init__(self, fname): self.file = open(fname, "rb") self.yyinput = bytearray(BUFSIZE) self.yylimit = BUFSIZE - 1 # exclude terminating null self.yycursor = self.yylimit self.yymarker = self.yylimit self.token = self.yylimit self.eof = False %{stags format = "\n self.@@ = -1"; %}

def __del__(self): self.file.close()

class Status(Enum): OK = 0 EOF = 1 LONG_LEXEME = 2

def fill(st): if st.eof: return Status.EOF

# Error: lexeme too long. In real life could reallocate a larger buffer. if st.token < 1: return Status.LONG_LEXEME

# Shift buffer contents (discard everything up to the current token). st.yyinput = st.yyinput[st.token:st.yylimit] st.yycursor -= st.token; st.yymarker -= st.token; st.yylimit -= st.token; %{stags format = "\n if st.@@ != -1: st.@@ -= st.token"; %} st.token = 0;

# Fill free space at the end of buffer with new data from file. bytes = st.file.read(BUFSIZE - st.yylimit - 1) # -1 for sentinel if not bytes: st.eof = True # end of file else: st.yylimit += len(bytes); st.yyinput += bytes

st.yyinput += b'\0' # append sentinel

return Status.OK

def lex(st, count): vers = [] while True: st.token = st.yycursor %{ re2c:api = record; re2c:yyrecord = st; re2c:YYFILL = "fill(st) == Status.OK"; re2c:eof = 0; re2c:indent:top = 2; re2c:tags = 1;

num = [0-9]+;

num @t1 "." @t2 num @t3 ("." @t4 num)? [\n] { major = int(st.yyinput[st.token:t1]) minor = int(st.yyinput[t2:t3]) patch = int(st.yyinput[t4:st.yycursor - 1]) if t4 != -1 else 0 vers.append(SemVer(major, minor, patch)) break } $ { return vers }

• { return None } %}

def main(): fname = "input" verstr = b"1.22.333\n" expect = [SemVer(1, 22, 333)] * BUFSIZE

# Prepare input file. f = open(fname, "wb") for i in range(BUFSIZE): f.write(verstr) f.close()

# Run lexer on the prepared file. st = State(fname) assert lex(st, 0) == expect

# Cleanup. os.remove(fname)

if __name__ == '__main__': main()

Here is an example of using capturing groups to parse semantic versions.

re2py $INPUT -o $OUTPUT

from collections import namedtuple

SemVer = namedtuple('SemVer', 'major minor patch')

NONE = -1

def parse(yyinput): yycursor = 0 %{ re2c:yyfill:enable = 0; re2c:captvars = 1; re2c:indent:top = 1;

num = [0-9]+;

(num) "." (num) ("." num)? [\x00] { major = int(yyinput[yytl1:yytr1]) minor = int(yyinput[yytl2:yytr2]) patch = 0 if yytl3 == NONE else int(yyinput[yytl3 + 1:yytr3]) return SemVer(major, minor, patch) }

• { return None } %}

assert parse(b"23.34\0") == SemVer(23, 34, 0) assert parse(b"1.2.99999\0") == SemVer(1, 2, 99999) assert parse(b"1.a\0") == None

Here is an example of using m-tags to parse a version with a variable number of components. Tag variables are stored in a trie.

re2py $INPUT -o $OUTPUT

NONE = -1

def parse(yyinput): yycursor = 0 %{mtags format = '\n @@ = []'; %} # autogenerated tag variables

%{ re2c:YYMTAGP = "@@.append(yycursor)"; re2c:YYMTAGN = ""; // do nothing re2c:yyfill:enable = 0; re2c:tags = 1; re2c:indent:top = 1;

num = [0-9]+;

@t1 num @t2 ("." #t3 num #t4)* [\x00] { vers = [int(yyinput[t1:t2])] for i in range(len(t3)): vers.append(int(yyinput[t3[i]:t4[i]])) return vers }

• { return None } %}

assert parse(b"1\0") == [1] assert parse(b"1.2.3.4.5.6.7\0") == [1, 2, 3, 4, 5, 6, 7] assert parse(b"1.2.\0") == None

## 💻 Encoding support 💻

okay so before we get into encodings, you gotta understand the difference between **code points** and **code units**, because this is lowkey the whole plot. A **code point** is a numeric identifier of a symbol. A **code unit** is the smallest unit of storage in the encoded text. A single code point may be represented with one or more code units. In a fixed-length encoding all code points are represented with the same number of code units. In a variable-length encoding code points may be represented with a different number of code units.

Note that the `"any"` rule `[^]` matches any code point, but not necessarily any code unit — the only way to match any code unit regardless of the encoding is the default rule `*`. The generated lexer works with a stream of code units: `yych` stores a code unit, and `YYCTYPE` is the code unit type. Regular expressions, on the other hand, are specified in terms of code points. When re2py compiles regular expressions to automata it translates code points to code units. This is generally not a simple mapping: in variable-length encodings a single code point range may get translated to a complex code unit graph.

The following encodings are supported:

- **ASCII** — enabled by default. It is a fixed-length encoding with code space `[0-255]` and 1-byte code points and code units.
- **EBCDIC** — enabled with `--ebcdic` or `re2c:encoding:ebcdic`. It is a fixed-length encoding with code space `[0-255]` and 1-byte code points and code units.
- **UCS2** — enabled with `--ucs2` or `re2c:encoding:ucs2`. It is a fixed-length encoding with code space `[0-0xFFFF]` and 2-byte code points and code units.
- **UTF8** — enabled with `--utf8` or `re2c:encoding:utf8`. It is a variable-length Unicode encoding. Code unit size is 1 byte. Code points are represented with 1 -- 4 code units.
- **UTF16** — enabled with `--utf16` or `re2c:encoding:utf16`. It is a variable-length Unicode encoding. Code unit size is 2 bytes. Code points are represented with 1 -- 2 code units.
- **UTF32** — enabled with `--utf32` or `re2c:encoding:utf32`. It is a fixed-length Unicode encoding with code space `[0-0x10FFFF]` and 4-byte code points and code units.

The include files `include/unicode_blocks.re`, `include/unicode_categories.re` and `include/unicode_properties.re` provide re2py definitions for blocks, categories and properties described in the Unicode standard.

Option `--input-encoding` specifies source file encoding, which can be used to enable Unicode literals in regular expressions. For example `--input-encoding utf8` tells re2py that the source file is in UTF8 (it differs from `--utf8` which sets input text encoding). Option `--encoding-policy` specifies the way re2py handles Unicode surrogates (code points in range `[0xD800-0xDFFF]`).

Below is an example of a lexer for UTF8 encoded Unicode identifiers.

re2py $INPUT -o $OUTPUT --utf8

%{include "unicode_categories.re" %}

def lex(yyinput): yycursor = 0 %{ re2c:yyfill:enable = 0; re2c:indent:top = 1;

// Simplified "Unicode Identifier and Pattern Syntax" // (see https://unicode.org/reports/tr31) id_start = L | Nl | [$_]; id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3]; identifier = id_start id_continue*;

identifier { return True }

• { return False } %}

assert lex(bytes("_Ыдентификатор\0", "utf-8"))

## 📁 Include files 📁

re2py allows one to include other files using a block of the form `/*!include:re2c FILE */` or `%{include FILE %}`, or an in-block directive `!include FILE ;`, where *FILE* is a path to the file to be included. re2py looks for include files in the directory of the including file and in include locations, which can be specified with the `-I` option. Include blocks/directives in re2py work in the same way as C/C++ `#include`: *FILE* contents are copy-pasted verbatim in place of the block/directive. Include files may have further includes of their own. Use `--depfile` option to track build dependencies of the output file on include files.

re2py provides some predefined include files that can be found in the `include/` subdirectory of the project. These files contain definitions that may be useful to other projects (such as Unicode categories) and form something like a standard library for re2py. Below is an example of using include files.

### 🌸 Include file 1 (`definitions.py`) 🌸

from enum import Enum

class Num(Enum): INT = 1 FLOAT = 2 NAN = 3

%{ number = [1-9][0-9]*; %}

### 🦋 Include file 2 (`extra_rules.re.inc`) 🦋

// floating-point numbers frac = [0-9]* "." [0-9]+ | [0-9]+ "."; exp = 'e' [+-]? [0-9]+; float = frac exp? | [0-9]+ exp;

float { return Num.FLOAT }

### 🎀 Input file 🎀

re2py $INPUT -o $OUTPUT

%{include "definitions.py" %}

def lex(yyinput): yycursor = 0 %{ re2c:yyfill:enable = 0; re2c:indent:top = 1;

• { return Num.NAN } number { return Num.INT } !include "extra_rules.re.inc"; %}

assert lex(b"123\0") == Num.INT assert lex(b"123.4567\0") == Num.FLOAT

## 👑 Header files 👑

re2py allows one to generate header file from the input `.re` file using `--header` option or `re2c:header` configuration and block pairs of the form `/*!header:re2c:on*/` and `/*!header:re2c:off*/`, or `%{header:on%}` and `%{header:off%}`. The first block marks the beginning of header file, and the second block marks the end of it. Everything between these blocks is processed by re2py, and the generated code is written to the file specified with `--header` option or `re2c:header` configuration (or stdout if neither option nor configuration is used). Autogenerated header file may be needed in cases when re2py is used to generate definitions that must be visible from other translation units.

Here is an example of generating a header file that contains definition of the lexer state with tag variables (the number variables depends on the regular grammar and is unknown to the programmer).

### ✨ Input file ✨

re2py $INPUT -o $OUTPUT --header lexer/state.py

from lexer.state import State

%{header:on %} class State: def __init__(self, str): self.yyinput = str self.yycursor = 0 %{stags format = "\n self.@@ = 0"; %} %{header:off %}

def lex(yyrecord): %{ re2c:api = record; re2c:tags = 1; re2c:yyfill:enable = 0; re2c:indent:top = 1; re2c:header = "lexer/state.py";

[a]* @t [b]* { return t } %}

assert lex(State(b"ab\0")) == 1

### 💖 Header file 💖

Generated by re2py

class State: def __init__(self, str): self.yyinput = str self.yycursor = 0

self.yyt1 = 0

## 🧚 Skeleton programs 🧚

With the `-S`, `--skeleton` option, re2py ignores all non-re2py code and generates a self-contained C program that can be further compiled and executed. The program consists of lexer code and input data. For each constructed DFA (block or condition) re2py generates a standalone lexer and two files: an `.input` file with strings derived from the DFA and a `.keys` file with expected match results. The program runs each lexer on the corresponding `.input` file and compares results with the expectations. Skeleton programs are very useful for a number of reasons:

- They can check correctness of various re2py optimizations (the data is generated early in the process, before any DFA transformations have taken place).
- Generating a set of input data with good coverage may be useful for both testing and benchmarking.
- Generating self-contained executable programs allows one to get minimized test cases (the original code may be large or have a lot of dependencies).

The difficulty with generating input data is that for all but the most trivial cases the number of possible input strings is too large (even if the string length is limited). re2py solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. It uses the following algorithm. First, it constructs a skeleton of the DFA. For encodings with 1-byte code unit size (such as ASCII, UTF-8 and EBCDIC) skeleton is just an exact copy of the original DFA. For encodings with multibyte code units skeleton is a copy of DFA with certain transitions omitted: namely, re2py takes at most 256 code units for each disjoint continuous range that corresponds to a DFA transition. The chosen values are evenly distributed and include range bounds. Instead of trying to cover all possible paths in the skeleton (which is infeasible) re2py generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer. The algorithm implementation is limited by ~1Gb of transitions and consumes constant amount of memory (re2py writes data to file as soon as it is generated).

## 🔥 Visualization and debug 🔥

With the `-D`, `--emit-dot` option, re2py does not generate code. Instead, it dumps the generated DFA in DOT format. One can convert this dump to an image of the DFA using Graphviz or another library. Note that this option shows the final DFA after it has gone through a number of optimizations and transformations. Earlier stages can be dumped with various debug options, such as `--dump-nfa`, `--dump-dfa-raw` etc. (see the full list of options).

---

# 🔗 SEE ALSO 🔗

You can find more information about re2c at the official website: `http://re2c.org`. Similar programs are `flex(1)`, `lex(1)`, `quex(http://quex.sourceforge.net)`.

# 👑 AUTHORS 👑

re2py was originally written by Peter Bumbulis (`peter@csg.uwaterloo.ca`) in 1993. Marcus Boerger and Dan Nuffer spent several years to turn the original idea into a production ready code generator. Since then it has been maintained and developed by multiple volunteers, most notably, Brian Young (`bayoung@acm.org`), Marcus Boerger, Dan Nuffer (`nuffer@users.sourceforge.net`), Ulya Trofimovich (`skvadrik@gmail.com`), Serghei Iakovlev, Sergei Trofimovich, Petr Skocik, ligfx raekye and PolarGoose.

---

and the footer at the very bottom says `RE2PY(1)` — that's the source/version/date sign-off, no cap 💖✨