RE2RUST(1)

💅 NAME

re2rust — generate fast lexical analyzers for Rust ✨

Okay so bestie, real talk: this is the re2rust(1) manual page, and we're gonna get through it together. Grab a strawberry drink 🍓, this is gonna be a ride.

---

🎀 SYNOPSIS

re2rust [ OPTIONS ] [ WARNINGS ] INPUT

Your INPUT is either a file or - if you want to feed it through stdin. Yes, the humble dash. Iconic. 🫶

---

✨ INTRODUCTION

So here's the vibe: re2rust is a preprocessor. It reads your input file (which is normally a Rust program, but lowkey it can be literally anything) and it goes looking for blocks of code wrapped up in special-form start/end markers.

Everything outside those blocks? Copied verbatim straight into the output. Untouched. Not our business. 💅

Everything inside those blocks? That's where re2rust does its thing — it processes the contents, translates them into Rust code, and drops the generated code right back in place of the block. Slay.

Here's a little baby program that checks if a given string contains a decimal number:

// re2rust $INPUT -o $OUTPUT --no-unsafe --api simple

fn lex(yyinput: &[u8]) -> bool {
    let mut yycursor = 0;
    /*!re2c
        re2c:YYCTYPE = u8;
        re2c:yyfill:enable = 0;

        [1-9][0-9]* { return true; }
        *           { return false; }
    */
}

fn main() {
    assert!(lex(b"1234\0"));
}

And in the output, re2rust replaced that middle block with the generated code. Look at it go:

/* Generated by re2rust */
// re2rust $INPUT -o $OUTPUT --no-unsafe --api simple

fn lex(yyinput: &[u8]) -> bool {
    let mut yycursor = 0;

{
    #[allow(unused_assignments)]
    let mut yych : u8 = 0;
    let mut yystate : usize = 0;
    'yyl: loop {
        match yystate {
            0 => {
                yych = yyinput[yycursor];
                yycursor += 1;
                match yych {
                    0x31 ..= 0x39 => {
                        yystate = 2;
                        continue 'yyl;
                    }
                    _ => {
                        yystate = 1;
                        continue 'yyl;
                    }
                }
            }
            1 => { return false; },
            2 => {
                yych = yyinput[yycursor];
                match yych {
                    0x30 ..= 0x39 => {
                        yycursor += 1;
                        yystate = 2;
                        continue 'yyl;
                    }
                    _ => {
                        yystate = 3;
                        continue 'yyl;
                    }
                }
            }
            3 => { return true; },
            _ => panic!("internal lexer error"),
        }
    }
}

}

fn main() {
    assert!(lex(b"1234\0"));
}

Like, the state machine just materialized out of the ether. Main character energy fr fr. 👑

---

🌸 BASICS

A re2rust program is literally just a sequence of blocks, intermixed with code in the target language. That's it. That's the whole structure.

And a block can contain definitions, configurations, rules, actions and directives — in any order you want. No cap. 🦋

• name = regular-expression ; — a definition. Binds name to regular-expression. Names can contain alphanumeric characters and underscore. (The regular expressions section gives you the whole overview of re2rust regex syntax, iykyk.) Once it's defined, the name can be used in other regular expressions and in rules. 🚨 Important: recursion in named definitions is not allowed, and each name should be defined before it's used. A block inherits named definitions from the global scope. And redefining a name that already exists in the current scope? That's an error. Don't do it bestie.
• configuration = value ; — a configuration. Lets you change re2rust behavior and customize the generated code. For the full list of configurations supported by re2rust, see the configurations section. Depending on the particular configuration, the value can be a keyword, a nonnegative integer number, or a one-line string — and that string should be enclosed in double or single quotes unless it consists of alphanumeric characters. A block inherits configurations from the global scope and may redefine them or add new ones. Also: configurations defined inside a block affect the whole block, even if they appear at the very end of it. Sneaky but true. 🙈
• regular-expression code — a rule. Binds regular-expression to its semantic action (a block of code in curly braces, or a block of code that starts with := and ends on a newline followed by any non-whitespace character). If the regular expression matches, the associated code is executed. If multiple rules match, the longest match takes precedence. If multiple rules match the same string, the earliest one takes precedence. There are two special rules: the default rule * and the end of input rule $. The default rule should always be defined — it has the lowest priority regardless of its place in the block, and it matches any code unit (not necessarily a valid character, see the encoding support section, mmkay?). The end of input rule should be defined if the corresponding method for handling the end of input is used. And with start conditions, rules have more complex syntax. 💀
• !action code — an action. Binds a user-defined block of code to a particular place in the generated finite state machine (in the same way as semantic actions bind code to the final states). See the actions section for a full list of predefined actions.
• !directive ; — a directive. One of the special predefined statements. Each directive has a unique purpose. See the directives section for details.

---

🧚 Blocks

Block start and end markers are eitther /*!re2c and */, or %{ and %}. Both styles are supported, pick your fighter. 💖

Starting from version 2.2, blocks may have optional names that allow them to be referenced in other blocks. Cute!

There are different kinds of blocks:

• /*!re2c[:<name>] ... */ or %{[:<name>] ... %} — a global block. It contains definitions, configurations, rules and directives. re2rust compiles the regular expressions associated with each rule into a deterministic finite automaton, encodes it in the form of conditional jumps in the target language, and replaces the block with the generated code. Names and configurations defined in a global block get added to the global scope and become visible to subsequent blocks. And at the start of the program, the global scope is initialized with command-line options. ✨
• /*!local:re2c[:<name>] ... */ or %{local[:<name>] ... %} — a local block. Like a global block, but the names and configurations in it have local scope (they don't affect other blocks). Keep your business your business. 💅
• /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} — a rules block. Like a local block, but it does not generate any code by itself, nor does it add any definitions to the global scope — it's meant to be reused in other blocks. It's a way of sharing code (more details in the reusable blocks section). And heads up: prior to re2rust version 2.2, rules blocks required the -r --reusable option. 🎀
• /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %} — a use block, which references a previously defined rules block. If a name is specified, re2rust looks for a rules block with that name. Otherwise it uses the most recent rules block (either a named or an unnamed one). A use block can add definitions, configurations and rules of its own, which get added to those of the referenced rules block. Also, prior to re2rust version 2.2, use blocks required the -r --reusable option.
• /*!max:re2c[:<name1>[:<name2>...]] ... */ or %{max[:<name1>[:<name2>...]] ... %} — a block that generates a YYMAXFILL definition. An optional list of block names specifies which blocks should be included when computing the YYMAXFILL value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXFILL <n>), or a global variable for Go (var YYMAXFILL int = <n>). It can be customized with an optional format configuration that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXFILL. 🦋
• /*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */ or %{maxnmatch[:<name1>[:<name2>...]] ... %} — a block that generates a YYMAXNMATCH definition. ⚠️ It requires the -P --posix-captures option. An optional list of block names specifies which blocks should be included when computing the YYMAXNMATCH value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXNMATCH <n>), or a global variable for Go (var YYMAXNMATCH int = <n>). It can be customized with an optional format configuration that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXNMATCH. 🌟
• /*!stags:re2c[:<name1>[:<name2>...]] ... */, /*!mtags:re2c[:<name1>[:<name2>...]] ... */, or %{stags[:<name1>[:<name2>...]] ... %}, %{mtags[:<name1>[:<name2>...]] ... %} — blocks that specify a template piece of code that gets expanded for each s-tag/m-tag variable generated by re2rust. An optional list of block names specifies which blocks should be included when computing the set of tag variables (if the list is empty, all blocks are included). There are two optional configurations: format and separator. The format configuration specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag variable. The separator configuration specifies a piece of code used to join the generated format pieces for different tag variables. 🎀
• /*!svars:re2c[:<name1>[:<name2>...]] ... */, /*!mvars:re2c[:<name1>[:<name2>...]] ... */ or %{svars[:<name1>[:<name2>...]] ... %}, %{mvars[:<name1>[:<name2>...]] ... %{ — bestie these are the template blocks 💅 they hold a template piece of code that gets expanded for each s-tag/m-tag that's either explicitly mentioned by the rules (that's the --tags option doing its thing) or implicitly generated by re2rust (via --captvars or --posix-captvars). You can drop an optional list of block names to say which blocks get included when computing the set of tags — and if the list is empty, all blocks are included, no cap. There are two optional configurations here: format and separator. The format config is a template string where @@{tag} (or just @@ for short) gets replaced with the name of each tag. The separator config is a piece of code used to join the generated format pieces for different tags. Obsessed. ✨
• /*!getstate:re2c[:<name1>[:<name2>...]] ... */ or %{getstate[:<name1>[:<name2>...]] ... %} — this block generates conditional dispatch on the lexer state, and heads up it requires the --storable-state option. An optional list of block names tells it which blocks get included in the state dispatch. The default transition goes to the start label of the first block on the list. If the list is empty, all blocks get included, and the default transition goes to the first block in the file that has a start label. Lowkey important: this block type is incompatible with the --loop-switch option, because it needs cross-block transitions that just aren't supported without goto or function calls. 🙈
• /*!conditions:re2c[:<name1>[:<name2>...]] ... */, /*!types:re2c... */ or %{conditions[:<name1>[:<name2>...]] ... %}, %{types... %} — the block that generates the condition enumeration (requires the --conditions option, obviously). Optional list of block names = which blocks get included when computing the set of conditions (empty list → all blocks, you know the drill). By default the generated code is an enumeration called YYCONDTYPE, and you can customize it with the optional format and separator configurations. format is a template string where @@{cond} (or @@ short) gets swapped for the name of each condition, and @@{num} gets swapped for a numeric index of that condition 🎀. separator is the piece of code joining the generated format pieces for different conditions.
• /*!include:re2c <file> */ or %{include <file> %} — lets you include <file>, which must be a double-quoted file path (fr fr, quotes required). The contents of the file are literally substituted in place of the block, the same way #include works in C/C++ 🫶. You can pair this block with the --depfile option to generate build system dependencies on the included files. Chef's kiss.
• /*!header:re2c:on*/ or %{header:on %} — marks the start of the header file. Everything after this block, up to the following header:off block, gets processed by re2rust and written to the header file you specified with the -t --type-header option.
• /*!header:re2c:off*/ or %{header:off %} — marks the end of the header file that was started with the header:on*/ block. Matching set, iconic behavior.
• /*!ignore:re2c ... */ or %{ignore ... %} — a block whose contents are ignored and removed from the output file. Not me just deleting my entire personality from the output 😭 love that for us.

---

✨ Configurations ✨

Okay bestie, here's the full list of configurations supported by re2rust — buckle up, it's giving long but every single one matters 🦋

• re2c:api, re2c:input — same as the --api option.
• re2c:api:sigil — specifies the marker ("sigil") used for argument placeholders in the API primitives. The default is @@. A placeholder starts with the sigil followed by the argument name in curly braces — so if the sigil is set to $, your placeholders look like ${name}. Single-argument APIs may use shorthand notation without the name in braces. This option can be overridden by options for individual API primitives, e.g. re2c:YYFILL@len for YYFILL.
• re2c:api:style — specifies the API style. Possible values are functions (the default for C) and free-form (the default for Go and Rust). In functions style, API primitives are generated with an argument list in parentheses following the name of the primitive. The arguments are provided only for autogenerated parameters (such as the number of characters passed to YYFILL), but not for the general lexer context — so the primitives behave more like macros in C/C++ or closures in Go and Rust. In free-form style, API primitives don't have a fixed form: they should be defined as strings containing free-form pieces of code with interpolated variables of the form @@{var} or @@ (these correspond to arguments in function-like style). This configuration may be overridden for individual API primitives — see, for example, the re2c:YYFILL:naked configuration for YYFILL.
• re2c:bit-vectors, re2c:flags:bit-vectors, re2c:flags:b — same as the --bit-vectors option, but can be configured on a per-block basis.
• re2c:captures, re2c:leftmost-captures — same as the --leftmost-captures option, but can be configured on a per-block basis.
• re2c:captvars, re2c:leftmost-captvars — same as the --leftmost-captvars option, but can be configured on a per-block basis.
• re2c:case-insensitive, re2c:flags:case-insensitive — same as the --case-insensitive option, but can be configured on a per-block basis.
• re2c:case-inverted, re2c:flags:case-inverted — same as the --case-inverted option, but can be configured on a per-block basis.
• re2c:case-ranges, re2c:flags:case-ranges — same as the --case-ranges option, but can be configured on a per-block basis.
• re2c:computed-gotos, re2c:flags:computed-gotos, re2c:flags:g — same as the --computed-gotos option, but can be configured on a per-block basis.
• re2c:computed-gotos:relative, re2c:cgoto:relative — same as the --computed-gotos-relative option, but can be configured on a per-block basis.
• re2c:computed-gotos:threshold, re2c:cgoto:threshold — if computed goto is used, this configuration specifies the complexity threshold that triggers the generation of jump tables instead of nested if statements and bitmaps. The default value is 9.
• re2c:cond:abort — if set to a positive integer value, the default case in the generated condition dispatch aborts program execution. 💀
• re2c:cond:goto — specifies a piece of code used for the autogenerated shortcut rules :=> in conditions. The default is goto @@;. The @@ placeholder is substituted with the condition name (see configurations re2c:api:sigil and re2c:cond:goto@cond).
• re2c:cond:goto@cond — specifies the sigil used for argument substitution in the re2c:cond:goto definition. The default value is @@. Overrides the more generic re2c:api:sigil configuration.
• re2c:cond:divider — defines the divider for condition blocks. The default value is /* *********************************** */. Placeholders are substituted with the condition name (see re2c:api;sigil and re2c:cond:divider@cond).
• re2c:cond:divider@cond — specifies the sigil used for argument substitution in the re2c:cond:divider definition. The default is @@. Overrides the more generic re2c:api:sigil configuration.
• re2c:cond:prefix, re2c:condprefix — specifies the prefix used for condition labels. The default is yyc_.
• re2c:cond:enumprefix, re2c:condenumprefix — specifies the prefix used for condition identifiers. The default is yyc.
• re2c:debug-output, re2c:flags:debug-output, re2c:flags:d — same as the --debug-output option, but can be configured on a per-block basis.
• re2c:empty-class, re2c:flags:empty-class — same as the --empty-class option, but can be configured on a per-block basis.
• re2c:encoding:ebcdic, re2c:flags:ecb, re2c:flags:e — same as the --ebcdic option, but can be configured on a per-block basis.
• re2c:encoding:ucs2, re2c:flags:wide-chars, re2c:flags:w — same as the --ucs2 option, but can be configured on a per-block basis.
• re2c:encoding:utf8, re2c:flags:utf-8, re2c:flags:8 — same as the --utf8 option, but can be configured on a per-block basis.
• re2c:encoding:utf16, re2c:flags:utf-16, re2c:flags:x — same as the --utf16 option, but can be configured on a per-block basis.
• re2c:encoding:utf32, re2c:flags:unicode, re2c:flags:u — same as the --utf32 option, but can be configured on a per-block basis.
• re2c:encoding-policy, re2c:flags:encoding-policy — same as the --encoding-policy option, but can be configured on a per-block basis.
• re2c:eof — specifies the sentinel symbol used with the end-of-input rule $. The default value is -1 (meaning the $ rule is not used). Other possible values include all valid code units, and heads up: only decimal numbers are recognized.
• re2c:header, re2c:flags:type-header, re2c:flags:t — specifies the name of the generated header file relative to the directory of the output file. Same as the --header option except that the file path is relative. Sneaky but valid 🧚
• re2c:indent:string — specifies the string used for indentation. The default is a single tab character "\t". The indent string should contain whitespace characters only. To disable indentation entirely, set this configuration to an empty string.
• re2c:indent:top — specifies the minimum amount of indentation to use. The default value is zero. The value should be a non-negative integer number.
• re2c:invert-captures — same as the --invert-captures option, but can be configured on a per-block basis.
• re2c:label:prefix, re2c:labelprefix — specifies the prefix used for DFA state labels. The default is yy.
• re2c:label:start, re2c:startlabel — controls the generation of a block start label. The default value is zero, which means the start label is generated only if it's used. An integer value greater than zero forces the generation of the start label even if it's unused by the lexer. A string value also forces start label generation and sets the label name to the specified string. Heads up bestie: this configuration applies only to the current block — it gets reset to default for the next block. 🙈
• re2c:label:yyFillLabel — sets the prefix for the YYFILL labels, the ones that show up when you're using re2c:eof and in storable state mode 🎀
• re2c:label:yyloop — names the label that marks the start of the lexer loop when you've got --loop-switch on. Default? yyloop, bestie ✨
• re2c:label:yyNext — names that optional label chilling right after the YYGETSTATE switch in storable state mode (you turn it on with re2c:state:nextlabel). Default is yyNext 🦋
• re2c:lookahead, re2c:flags:lookahead — deprecated, babes 💀 go peek at the deprecated --no-lookahead option instead.
• re2c:monadic — if you set this to non-zero, the generated lexer uses monadic notation. This one is specifically a Haskell thing, iykyk 💅
• re2c:nested-ifs, re2c:flags:nested-ifs, re2c:flags:s — does the same thing as --nested-ifs, except you can configure it per-block 🫶
• re2c:posix-captures, re2c:flags:posix-captures, re2c:flags:P — same as the --posix-captures option, but configurable on a per-block basis ✨
• re2c:posix-captvars — same as the --posix-captvars option, and yes, per-block configurable too 🌸
• re2c:tags, re2c:flags:tags, re2c:flags:T — same as the --tags option, per-block configurable 💖
• re2c:tags:expression — sets the expression used for tag variables. By default re2rust generates expressions of the form yyt<N>. That can be lowkey inconvenient, like if your tag variables are fields in a struct. Every occurrence of @@{tag} or @@ gets replaced with the actual tag name. Example: re2c:tags:expression = "s.@@"; gives you expressions of the form s.yyt<N> in the generated code 👑 also see the re2c:api:sigil configuration.
• re2c:tags:negative — the constant expression used for a negative tag value (typically this is -1 when tags are integer offsets into the input string, or a null pointer when they're pointers) 🎀
• re2c:tags:prefix — the prefix for tag variable names. Default is yyt ✨
• re2c:sentinel — the sentinel symbol used for end-of-input checks (that's when bounds checks are disabled with re2c:yyfill:enable = 0; and re2c:eof isn't set). Heads up: this configuration does not affect code generation — its whole job is to verify that the sentinel isn't allowed in the middle of a rule, and to make sure the lexer won't read past the end of the buffer 🫶 The default value is -1 (in that case re2rust assumes the sentinel is zero, which is the most common case). Only decimal numbers are recognized, no cap 💅
• re2c:state:abort — if you set this to a positive integer value, the default case in the generated state dispatch aborts program execution, and an explicit -1 case contains the transition to the start of the block 🔥
• re2c:state:nextlabel — controls whether the YYGETSTATE switch is followed by a yyNext label (default value is zero, which means no label). Alternatively you can use re2c:label:start to generate a specific start label, or an explicit getstate block to generate the YYGETSTATE switch separately from the lexer block 😌
• re2c:unsafe, re2c:flags:unsafe — same as the --no-unsafe option, but per-block configurable. If set to zero, it suppresses generation of the unsafe wrappers around YYPEEK. Default is non-zero (so wrappers are generated). Rust-specific, bestie 🦀
• re2c:YYBACKUP, re2c:define:YYBACKUP — defines the generic API primitive YYBACKUP.
• re2c:YYBACKUPCTX, re2c:define:YYBACKUPCTX — defines the generic API primitive YYBACKUPCTX.
• re2c:YYCONDTYPE, re2c:define:YYCONDTYPE — defines the API primitive YYCONDTYPE.
• re2c:YYCTYPE, re2c:define:YYCTYPE — defines the API primitive YYCTYPE.
• re2c:YYCTXMARKER, re2c:define:YYCTXMARKER — defines the API primitive YYCTXMARKER.
• re2c:YYCURSOR, re2c:define:YYCURSOR — defines the API primitive YYCURSOR.
• re2c:YYDEBUG, re2c:define:YYDEBUG — defines the API primitive YYDEBUG.
• re2c:YYFILL, re2c:define:YYFILL — defines the API primitive YYFILL.
• re2c:YYFILL@len, re2c:define:YYFILL@len — the sigil used for argument substitution in the YYFILL definition. Defaults to @@, and it overrides the more generic re2c:api:sigil configuration ✨
• re2c:YYFILL:naked, re2c:define:YYFILL:naked — overrides the more generic re2c:api:style configuration for YYFILL. A zero value corresponds to free-form API style 💅
• re2c:YYFN — defines the API primitive YYFN.
• re2c:YYINPUT — defines the API primitive YYINPUT.
• re2c:YYGETCOND, re2c:define:YYGETCONDITION — defines the API primitive YYGETCOND.
• re2c:YYGETCOND:naked, re2c:define:YYGETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYGETCOND. Zero value = free-form API style 🎀
• re2c:YYGETSTATE, re2c:define:YYGETSTATE — defines the API primitive YYGETSTATE.
• re2c:YYGETSTATE:naked, re2c:define:YYGETSTATE:naked — overrides the more generic re2c:api:style configuration for YYGETSTATE. Zero value = free-form API style 🌸
• re2c:YYGETACCEPT, re2c:define:YYGETACCEPT — defines the API primitive YYGETACCEPT.
• re2c:YYLESSTHAN, re2c:define:YYLESSTHAN — defines the generic API primitive YYLESSTHAN.
• re2c:YYLIMIT, re2c:define:YYLIMIT — defines the API primitive YYLIMIT.
• re2c:YYMARKER, re2c:define:YYMARKER — defines the API primitive YYMARKER.
• re2c:YYMTAGN, re2c:define:YYMTAGN — defines the generic API primitive YYMTAGN.
• re2c:YYMTAGP, re2c:define:YYMTAGP — defines the generic API primitive YYMTAGP.
• re2c:YYPEEK, re2c:define:YYPEEK — defines the generic API primitive YYPEEK.
• re2c:YYRESTORE, re2c:define:YYRESTORE — defines the generic API primitive YYRESTORE.
• re2c:YYRESTORECTX, re2c:define:YYRESTORECTX — defines the generic API primitive YYRESTORECTX.
• re2c:YYRESTORETAG, re2c:define:YYRESTORETAG — defines the generic API primitive YYRESTORETAG.
• re2c:YYSETCOND, re2c:define:YYSETCONDITION — defines the API primitive YYSETCOND.
• re2c:YYSETCOND@cond, re2c:define:YYSETCONDITION@cond — the sigil used for argument substitution in the YYSETCOND definition. Default value is @@, and it overrides the more generic re2c:api:sigil configuration 💖
• re2c:YYSETCOND:naked, re2c:define:YYSETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYSETCOND. Zero value = free-form API style 🦋
• re2c:YYSETSTATE, re2c:define:YYSETSTATE — defines the API primitive YYSETSTATE.
• re2c:YYSETSTATE@state, re2c:define:YYSETSTATE@state — the sigil used for argument substitution in the YYSETSTATE definition. Default value is @@, and it overrides the more generic re2c:api:sigil configuration ✨
• re2c:YYSETSTATE:naked, re2c:define:YYSETSTATE:naked — overrides the more generic re2c:api:style configuration for YYSETSTATE. Zero value = free-form API style 💅
• re2c:YYSETACCEPT, re2c:define:YYSETACCEPT — defines the API primitive YYSETACCEPT.
• re2c:YYSKIP, re2c:define:YYSKIP — defines the generic API primitive YYSKIP.
• re2c:YYSHIFT, re2c:define:YYSHIFT — defines the generic API primitive YYSHIFT.
• re2c:YYCOPYMTAG, re2c:define:YYCOPYMTAG — defines the generic API primitive YYCOPYMTAG.
• re2c:YYCOPYSTAG, re2c:define:YYCOPYSTAG — defines the generic API primitive YYCOPYSTAG.
• re2c:YYSHIFTMTAG, re2c:define:YYSHIFTMTAG — defines the generic API primitive YYSHIFTMTAG.
• re2c:YYSHIFTSTAG, re2c:define:YYSHIFTSTAG — defines the generic API primitive YYSHIFTSTAG.
• re2c:YYSTAGN, re2c:define:YYSTAGN — defines the generic API primitive YYSTAGN.
• re2c:YYSTAGP, re2c:define:YYSTAGP — defines the generic API primitive YYSTAGP.
• re2c:yyaccept, re2c:variable:yyaccept — defines the API primitive yyaccept.
• re2c:yybm, re2c:variable:yybm — defines the API primitive yybm.
• re2c:yybm:hex, re2c:variable:yybm:hex — if set to nonzero, bitmaps for the --bit-vectors option are generated in hexadecimal format. Default is zero, so bitmaps come out in decimal 🎀
• re2c:yych, re2c:variable:yych — defines the API primitive yych.
• re2c:yych:emit, re2c:variable:yych:emit — if set to zero, the yych definition isn't generated. Default is non-zero 🌸
• re2c:yych:conversion, re2c:variable:yych:conversion — if set to non-zero, re2rust automatically generates a conversion to YYCTYPE every single time yych is read. Default is zero, meaning no conversion ✨
• re2c:yych:literals, re2c:variable:yych:literals — the form of literals that yych is matched against. The possible values are: char (character literals in single quotes, and non-printable ones use escape sequences starting with a backslash), hex (hexadecimal integers), and char_or_hex (a lil mixture of both — character literals for printable characters, hexadecimal integers for everything else) 🦋
• re2c:yyctable, re2c:variable:yyctable — defines the API primitive yyctable.
• re2c:yynmatch, re2c:variable:yynmatch — defines the API primitive yynmatch.
• re2c:yypmatch, re2c:variable:yypmatch — defines the API primitive yypmatch.
• re2c:yytarget, re2c:variable:yytarget — defines the API primitive yytarget.
• re2c:yystable, re2c:variable:yystable — deprecated, bestie 💀
• re2c:yystate, re2c:variable:yystate — defines the API primitive yystate.
• re2c:yyfill, re2c:variable:yyfill — defines the API primitive yyfill.
• re2c:yyfill:check — if set to zero, suppresses generation of the pre-YYFILL check for the number of input characters (that's the YYLESSTHAN definition in generic API, and the YYLIMIT-based comparison in C pointer API). Default is non-zero, so the check is generated 🫶
• re2c:yyfill:enable — if set to zero, suppresses generation of YYFILL (together with the check). Use this when the whole input fits into one piece of memory — no buffering needed — and the end-of-input checks don't rely on YYFILL checks (like when a sentinel character is used) 👑 Use the warnings (-W option) and the re2c:sentinel configuration to verify that the generated lexer can't read past the end of input. Default is non-zero, meaning YYFILL is enabled 💖

---

• re2c:yyfill:parameter — if you set this to zero, it suppresses generating the parameter that gets passed to YYFILL. That parameter is the minimum number of characters that must be supplied, bestie. Defaults to non-zero, aka the parameter is generated. You can override this with re2c:YYFILL:naked or re2c:api:style.
• re2c:yyfn:sep — specifies the separator used in YYFN elements (defaults to semicolon). tiny but important config energy 💅
• re2c:yyfn:throw — specifies the exceptions thrown by the YYFN function (defaults to empty, which means no exceptions). no drama, no throw 🎀

---

✨ Regular expressions ✨

re2rust uses the following syntax for regular expressions:

• "foo" — case-sensitive string literal.
• 'foo' — case-insensitive string literal.
• [a-xyz], [^a-xyz] — character class (possibly negated).
• . — any character except newline.
• R \ S — difference of character classes R and S.
• R* — zero or more occurrences of R.
• R+ — one or more occurrences of R.
• R? — optional R.
• R{n} — repetition of R exactly n times.
• R{n,} — repetition of R at least n times.
• R{n,m} — repetition of R from n to m times.
• (R) — just R; parentheses are used to override precedence. If submatch extraction is enabled, (R) is a capturing or a non-capturing group depending on the --invert-captures option.
• (!R) — if submatch extraction is enabled, (!R) is a non-capturing or a capturing group depending on the --invert-captures option.
• R S — concatenation: R followed by S.
• R | S — alternative: R or S.
• R / S — lookahead: R followed by S, but S is not consumed.
• name — regular expression defined as name (or literal string "name" in Flex compatibility mode).
• {name} — regular expression defined as name in Flex compatibility mode.
• @stag — an s-tag: saves the last input position at which @stag matches in a variable named stag.
• #mtag — an m-tag: saves all input positions at which #mtag matches in a variable named mtag.
• $ — end of input.

Character classes and string literals may contain the following escape sequences: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and \Uhhhhhhhh. literally all the escape lore 🌀

---

🎀 Actions 🎀

Here is the list of predefined actions supported by re2rust:

• !entry code — entry action binds a user-defined block of code to the start state of the current finite state machine. If start conditions are used, the entry action can be set individually for each condition. This action may be used to perform initialization, e.g. to save the start location of a lexeme.
• !pre_rule code — pre-rule action prepends a user-defined block of code to semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to factor out the common part of all semantic actions (e.g. saving the end location of a lexeme).
• !post_rule code — post-rule action appends a user-defined block of code to semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to emit trap statements that guard against unintended control flow.

---

🦋 Directives 🦋

Here is the full list of directives supported by re2rust:

• !use:name ; — an in-block use directive that merges a previously defined rules block with the specified name into the current block. Named definitions, configurations and rules of the referenced block are added to the current ones. Conflicts between overlapping rules and configurations are resolved in the usual way: the first rule takes priority, and the latest configuration overrides the preceding ones. One exception is the special rules *, $ and <!> for which a block-local definition always takes priority. A use directive can be placed anywhere inside of a block, and multiple use directives are allowed.
• !include file ; — this directive is the same as include block: it inserts file contents verbatim in place of the directive.

---

🌈 Program interface 🌈

The generated code interfaces with the outer program with the help of primitives, collectively referred to as the API. Which primitives should be defined for a particular program depends on multiple factors, including the complexity of regular expressions, input representation, buffering and the use of various features. All the necessary primitives should be defined by the user in the form of macros, functions, variables or any other suitable form that makes the generated code syntactically and semantically correct. re2rust does not (and cannot) check the definitions, so if anything is missing or defined incorrectly, the generated program may have compile-time or run-time errors. This manual provides examples of API definitions in the most common cases.

fr fr, bestie: re2rust cannot vibe-check your definitions for you. missing or wrong API primitives can absolutely wreck the generated program at compile time or run time 💀

re2rust has three API flavors that define the core set of primitives used by a program:

💖 Simple API 💖

(added in version 4.0) This is a basic API that can be enabled with --api simple option or re2c:api = simple configuration. It consists of the following primitives: YYINPUT (which should be defined as a sequence of code units, e.g. a string) and YYCURSOR, YYMARKER, YYCTXMARKER, YYLIMIT (which should be defined as indices in YYINPUT).

🍓 Record API 🍓

(added in version 4.0) Record API is useful in cases when lexer state must be stored in a struct. It is enabled with --api record option or re2c:api = record configuration. This API consists of a variable yyrecord (the name can be overridden with re2c:yyrecord) that should be defined as a struct with fields yyinput, yycursor, yymarker, yyctxmarker, yylimit (only the fields used by the generated code need to be defined, and their names can be configured).

👑 Generic API 👑

This is the most flexible API and the default API for the Rust backend. This API contains primitives for generic operations: YYPEEK, YYSKIP, YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE, YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG, YYLESSTHAN, YYEND. For example, if the input is a byte slice buffer: &[u8], variables cursor, limit, marker and ctxmarker of type usize represent input positions, and a constant NONE represents invalid position, then generic API can be defined as follows:

/*!re2c
  re2c:YYPEEK       = "*buffer.get_unchecked(cursor)";
  re2c:YYSKIP       = "cursor += 1;";
  re2c:YYBACKUP     = "marker = cursor;";
  re2c:YYRESTORE    = "cursor = marker;";
  re2c:YYBACKUPCTX  = "ctxmarker = cursor;";
  re2c:YYRESTORECTX = "cursor = ctxmarker;";
  re2c:YYRESTORETAG = "cursor = @@{tag};";
  re2c:YYLESSTHAN   = "limit - cursor < @@{len}";
  re2c:YYEND        = "limit == cursor";
  re2c:YYSTAGP      = "@@{tag} = cursor;";
  re2c:YYSTAGN      = "@@{tag} = NONE;";
  re2c:YYSHIFT      = "cursor = (cursor as isize + @@{shift}) as usize;";
  re2c:YYSHIFTSTAG  = "@@{tag} = (@@{tag} as isize + @@{shift}) as usize;";
*/

Here is a full list of API primitives that may be used by the generated code in order to interface with the outer program.

• YYCTYPE — the type of the input characters (code units). For ASCII, EBCDIC and UTF-8 encodings it should be 1-byte unsigned integer. For UTF-16 or UCS-2 it should be 2-byte unsigned integer. For UTF-32 it should be 4-byte unsigned integer.
• YYCURSOR — an l-value that stores the current input position (a pointer or an integer offset in YYINPUT). Initially YYCURSOR should point to the first input character, and later it is advanced by the generated code. When a rule matches, YYCURSOR position is the one after the last matched character.
• YYLIMIT — an r-value that stores the end of input position (a pointer or an integer offset in YYINPUT). Initially YYLIMIT should point to the position after the last available input character. It is not changed by the generated code. The lexer compares YYCURSOR to YYLIMIT in order to determine if there are enough input characters left.
• YYMARKER — an l-value that stores the position of the latest matched rule (a pointer or an integer offset in YYINPUT). It is used to restore the YYCURSOR position if the longer match fails and the lexer needs to rollback. Initialization is not needed.
• YYCTXMARKER — an l-value that stores the position of the trailing context (a pointer or an integer offset in YYINPUT). No initialization is needed. YYCTXMARKER is needed only if the lookahead operator / is used.
• YYFILL — a generic API primitive with one variable len. YYFILL should provide at least len more input characters or fail. If re2c:eof is used, then len is always 1 and YYFILL should always return to the calling function; zero return value indicates success. If re2c:eof is not used, then YYFILL return value is ignored and it should not return on failure. The maximum value of len is YYMAXFILL.
• YYFN — a primitive that defines function prototype in --recursive-functions code model. Its value should be an array of one or more strings, where each string contains two or three components separated by the string specified in re2c:fn:sep configuration (typically a semicolon). The first array element defines function name and return type (empty for a void function). Subsequent elements define function arguments: first, the expression for the argument used in function body (usually just a name); second, argument type; third, an optional formal parameter (it defaults to the first component — usually both the argument and the parameter are the same identifier).
• YYINPUT — an r-value that stores the current input character sequence (string, buffer, etc.).
• YYMAXFILL — an integral constant equal to the maximum value of the argument to YYFILL. It can be generated with a max block.
• YYLESSTHAN — a generic API primitive with one variable len. It should be defined as an r-value of boolean type that equals true if and only if there are less than len input characters left.
• YYEND — a generic API primitive, no variables. Define it as an r-value of boolean type that equals true if and only if the logical end of input has been reached (padding and sentinel symbols don't count, bestie 💅). This is literally what implements the $ symbol in regular expressions. It is not the same as YYLESSTHAN, which is the one that makes sure the lexer won't read past the end of the buffer — different jobs, both iconic.
• YYPEEK — a generic API primitive, no variables. Define it as an r-value of type YYCTYPE that equals the character at the current input position. That's it, that's the whole vibe ✨
• YYSKIP — a generic API primitive that should advance the current input position by one code unit. Little step forward, no cap.
• YYBACKUP — a generic API primitive that should save the current input position, so it can be restored later with YYRESTORE. Screenshot the current spot 📸
• YYRESTORE — a generic API primitive that should restore the current input position to the value saved by YYBACKUP. Rewind time, lowkey.
• YYBACKUPCTX — a generic API primitive that should save the current input position as the position of the trailing context (restored later with YYRESTORECTX). Same save-the-spot energy, but for the trailing context specifically.
• YYRESTORECTX — a generic API primitive that should restore the trailing context position saved with YYBACKUPCTX.
• YYRESTORETAG — a generic API primitive with one variable, tag, that should restore the trailing context position to the value of tag.
• YYSTAGP — a generic API primitive with one variable, tag, where tag can be a pointer or an offset in YYINPUT (peep the submatch extraction section for the details). YYSTAGP should set tag to the current input position.
• YYSTAGN — a generic API primitive with one variable, tag, where tag can be a pointer or an offset in YYINPUT (again, submatch extraction section has the tea). YYSTAGN should set tag to a value that represents a non-existent input position. Ghost position 👻
• YYMTAGP — a generic API primitive with one variable, tag. YYMTAGP should append the current position to the submatch history of tag (see the submatch extraction section for the details).
• YYMTAGN — a generic API primitive with one variable, tag. YYMTAGN should append a value that represents a non-existent input position to the submatch history of tag (see the submatch extraction section for the details).
• YYSHIFT — a generic API primitive with one variable, shift, that should shift the current input position by shift characters. Heads up: the shift value may be negative 😭
• YYCOPYSTAG — a generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side s-tag variable rhs to the left-hand-side s-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs.
• YYCOPYMTAG — a generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side m-tag variable rhs to the left-hand-side m-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs.
• YYSHIFTSTAG — a generic API primitive with two variables, tag and shift, that should shift tag by shift code units. And yes, the shift value may be negative.
• YYSHIFTMTAG — a generic API primitive with two variables, tag and shift, that should shift the latest value in the history of tag by shift code units (the shift value may be negative, fr fr).
• YYMAXNMATCH — an integral constant equal to the maximal number of POSIX capturing groups in a rule. It gets generated with a maxnmatch block.
• YYCONDTYPE — the type of the condition enum. It can be generated either with a conditions block or with the --header option.
• YYGETACCEPT — a primitive with one variable, var, that stores the numeric selector of the accepted rule. For most languages this primitive has a default definition that reads from var.
• YYSETACCEPT — a primitive with two variables: var (an l-value that stores the numeric selector of the accepted rule) and val (the value of the selector). For most languages this primitive has a default definition that assigns var to val.
• YYGETCOND — an r-value of type YYCONDTYPE that equals the current condition identifier.
• YYSETCOND — a primitive with one variable, cond, that should set the current condition identifier to cond.
• YYGETSTATE — an r-value of integer type that equals the current lexer state. It should be initialized to -1. Yes, -1. Not a typo, bestie 🫶
• YYSETSTATE — a primitive with one variable, state, that should set the current lexer state to state.
• YYDEBUG — this primitive is generated only with the -d, --debug-output option. Its whole purpose is adding logging to the generated code (a typical YYDEBUG definition is a print statement). YYDEBUG statements are generated in every state and have two variables: state (either a DFA state index or -1) and symbol (the current input symbol).
• yyaccept — an l-value of unsigned integral type that stores the number of the latest matched rule. A user definition is necessary only with the --storable-state option.
• yybm — a table containing compressed bitmaps for up to 8 transitions (used with the --bitmaps option). The table contains 256 elements and is indexed by 1-byte code units. Each 8-bit element combines boolean values for up to 8 transitions. The k-th bit of the n-th element is true iff the n-th code unit is in the range of the k-th transition. The idea of this bitmap is to replace many if branches or switch cases with one single bit check in the table. Efficiency slay 💅
• yych — an l-value of type YYCTYPE that stores the current input character. A user definition is necessary only with the -f, --storable-state option.
• yyctable — a jump table generated for the initial condition dispatch (enabled with the combination of the --conditions and --computed-gotos options).
• yyfill — an l-value that stores the result of the YYFILL call (this may be necessary for pure functional languages, where YYFILL is a monadic function with a complex return value).
• yynmatch — an l-value of unsigned integral type that stores the number of POSIX capturing groups in the matched rule. Used only with the -P, --posix-captures option.
• yypmatch — an array of l-values that hold the tag values corresponding to the capturing parentheses in the matching rule. The array length must be at least yynmatch * 2 (usually YYMAXNMATCH * 2 is a good choice). Used only with the -P, --posix-captures option.
• yystable — deprecated. That's the whole description, she's retired 🪦
• yystate — an l-value used with the --loop-switch option to store the current DFA state.
• yytarget — a jump table that contains jump targets (label addresses) for all transitions from a state. This table is local to each state. Generation of yytarget tables is enabled with the --computed-gotos option.

---

✨ Options ✨

Okay so quick heads-up before the list: some of these options have corresponding configurations, others are global and cannot be changed after re2c starts reading the input file. Debug options generally require building re2c in debug configuration. Internal options are useful for experimenting with the algorithms used in re2c. Not me knowing all of them 💀

• -?, --help, -h — show the help message. Your first stop when you're lost, bestie 🫶
• --api <simple | record | generic> — specify the API used by the generated code to interface with user-defined code. Option simple should be used in simple cases when there's no need for buffer refilling and storing lexer state. Option record should be used when lexer state needs to be stored in a record (struct, class, etc.). Option generic should be used in complex cases when the other two APIs are not flexible enough.
• --bit-vectors, -b — optimize conditional jumps using bit masks. This option implies --nested-ifs. ✨
• --captures, --leftmost-captures — enable submatch extraction with leftmost greedy capturing groups. The result is collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule.
• --captvars, --leftmost-captvars — enable submatch extraction with leftmost greedy capturing groups. The result is collected into variables yytl<k>, yytr<k> for the k-th capturing group.
• --case-insensitive — treat single-quoted and double-quoted strings as case-insensitive.
• --case-inverted — invert the meaning of single-quoted and double-quoted strings: treat single-quoted strings as case-sensitive and double-quoted strings as case-insensitive. Chaos, but on purpose 💅
• --case-ranges — collapse consecutive cases in a switch statement into a range of the form low ... high. This syntax is a C/C++ language extension that's supported by compilers like GCC, Clang and Tcc. The main advantage over using single cases is smaller generated code and faster generation time, although for some compilers like Tcc it also results in a smaller binary size.
• --computed-gotos, -g — optimize conditional jumps using the non-standard "computed goto" extension (which must be supported by the compiler). re2rust generates jump tables only in complex cases with a lot of conditional branches. The complexity threshold can be configured with the cgoto:threshold configuration. Relative offsets can be enabled with the cgoto:relative configuration. This option implies --bit-vectors.
• --computed-gotos-relative — similar to --computed-gotos but generate relative offsets for jump tables instead (which must be supported by the compiler). This option implies --computed-gotos.
• --conditions, --start-conditions, -c — enable support of Flex-like "conditions": multiple interrelated lexers within one block. This is an alternative to manually specifying different re2rust blocks connected with goto or function calls.
• --depfile FILE — write dependency information to FILE in the form of a Makefile rule <output-file> : <input-file> [include-file ...]. This lets you track build dependencies in the presence of include blocks/directives, so that updating include files triggers regeneration of the output file. This option depends on the --output option. No cap, this one's a lifesaver for build systems 🧚
• --ebcdic, --ecb, -e — bestie we're going vintage 💀 this one generates a lexer that reads input in EBCDIC encoding. re2rust assumes the character range is 0 -- 0xFF and character size is 1 byte.
• --empty-class <match-empty | match-none | error> — okay so this decides how re2rust treats empty character classes, and the vibes are chaotic:
  • match-empty (the default, and yes it's the default) makes an empty class match empty input — which is illogical, but backwards-compatible 🫠
  • match-none makes an empty class always fail to match
  • error makes an empty class raise a compilation error
• --encoding-policy <fail | substitute | ignore> — how do we feel about Unicode surrogates? pick your fighter:
  • fail — re2rust aborts with an error when a surrogate is encountered
  • substitute — re2rust silently replaces surrogates with the error code point 0xFFFD
  • ignore (the default) — re2rust treats surrogates as normal code points

The Unicode standard says standalone surrogates are invalid, but real-world libraries and programs behave in different ways. no cap, it's giving messy out here 😭

• --flex-syntax, -F — partial support for Flex syntax 💅 in this mode, named definitions don't need the equal sign and the terminating semicolon, and when used they must be surrounded with curly braces. Names without curly braces are treated as double-quoted strings.
• --goto-label — use the "goto/label" code model: encode the DFA in the form of labeled code blocks connected with goto transitions across blocks. This is only supported for languages that have a goto statement. iykyk 🎀
• --header, --type-header, -t _HEADER_ — generate a HEADER file ✨ the contents of the file can be specified using special blocks header:on and header:off. If conditions are used, the generated header will have a condition enum automatically appended to it (unless there is an explicit conditions block).
• -I _PATH_ — add _PATH_ to the list of locations which are used when searching for include files. This option is useful in combination with include block or directive. re2rust looks for _FILE_ in the directory of the parent file and in the include locations specified with -I.
• --input <default | custom> — deprecated alias for --api 🙈 option default corresponds to simple (it is indeed the default for most backends, but not for all). Option custom corresponds to generic.
• --input-encoding <ascii | utf8> — how does re2rust parse regular expressions? let's chat:
  • ascii (the default) — re2rust handles input as ASCII-encoded: any sequence of code units is a sequence of standalone 1-byte characters
  • utf8 — re2rust handles input as UTF8-encoded and recognizes multibyte characters
• --invert-captures — flip the meaning of capturing and non-capturing groups 🔄 by default (...) is capturing and (! ...) is non-capturing. With this option (! ...) is capturing and (...) is non-capturing.
• --lang <none | c | d | go | haskell | java | js | ocaml | python | rust | swift | v | zig> — pick your target language, bestie 🌈 supported languages are C, D, Go, Haskell, Java, JS, OCaml, Python, Rust, Swift, V, Zig (more languages can be added via user-defined syntax files, see the --syntax option). Option none disables default syntax configs, so that the target language is undefined. (yes, the original really does say "suntax", we're keeping it, it's iconic 💀)
• --location-format <gnu | msvc> — how locations show up in messages:
  • gnu — locations are printed as 'filename:line:column: ...'
  • msvc — locations are printed as 'filename(line,column) ...'

The default is gnu.

• --loop-switch — use the "loop/switch" code model: encode the DFA in the form of a loop over a switch statement, where individual states are switch cases 🔁 state is stored in a variable yystate. Transitions between states update yystate to the case label of the destination state and continue execution to the head of the loop. main character energy tbh 👑
• --computed-continue — this option requires the --loop-switch code model ⚠️ it uses computed continue statements to select the next switch case directly, enabling jump-table optimization similar to --computed-gotos in the goto/label code model. slay.
• --nested-ifs, -s — use nested if statements instead of switch statements in conditional jumps. This usually results in more efficient code with non-optimizing compilers 💪
• --no-debug-info, -i — do not output line directives 🚫 this may be useful when the generated code is stored in a version control system (to avoid huge autogenerated diffs on small changes). protecting your git history is self care fr fr 🫶
• --no-generation-date — suppress date output in the generated file.
• --no-version — suppress version output in the generated file.
• --no-unsafe — do not generate unsafe wrapper over YYPEEK (this option is specific to Rust) 🦀 for performance reasons YYPEEK should avoid bounds-checking, as the lexer already performs end-of-input checks in a more efficient way. The user may choose to provide a safe YYPEEK definition, or a definition that is unsafe only in release builds, in which case the --no-unsafe option helps to avoid warnings about redundant unsafe blocks.
• --output, -o _OUTPUT_ — specify the _OUTPUT_ file 💾
• --posix-captures, -P — enable submatch extraction with POSIX-style capturing groups. The result is collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule.
• --posix-captvars — enable submatch extraction with POSIX-style capturing groups. The result is collected into variables yytl<k>, yytr<k> for _k_-th capturing group ✨ (yes, that's a different option from --posix-captures, don't @ me)
• --recursive-functions — use a code model based on co-recursive functions, where each DFA state is a separate function that may call other state-functions or itself 🌀 recursion girlies rise
• --reusable, -r — deprecated since version 2.2 (reusable blocks are allowed by default now) 👵 rip bozo, it's a legacy option
• --skeleton, -S — ignore user-defined interface code and generate a self-contained "skeleton" program 💀 additionally, generate input files with strings derived from the regular grammar and compressed match results that are used to verify "skeleton" behavior on all inputs. This option is useful for finding bugs in optimizations and code generation. This option is supported only for C.
• --storable-state, -f — generate a lexer which can store its inner state 💾 this is useful in push-model lexers which are stopped by an outer program when there is not enough input, and then resumed when more input becomes available. In this mode users should additionally define YYGETSTATE and YYSETSTATE primitives, and variables yych, yyaccept and state should be part of the stored lexer state. (don't forget those, or it's not gonna slay)
• --syntax _FILE_ — load configurations from the specified _FILE_ and apply them on top of the default syntax file 🌸 note that _FILE_ can define only a few configurations (if it's used to amend the default syntax file), or it can define a whole new language backend (in the latter case it is recommended to use --lang none option).
• --tags, -T — enable submatch extraction with tags 🏷️
• --ucs2, --wide-chars, -w — generate a lexer that reads UCS2-encoded input. re2rust assumes that the character range is 0 -- 0xFFFF and character size is 2 bytes. This option implies --nested-ifs.
• --utf8, --utf-8, -8 — generate a lexer that reads input in UTF-8 encoding. re2rust assumes that the character range is 0 -- 0x10FFFF and character size is 1 byte 🌈
• --utf16, --utf-16, -x — generate a lexer that reads UTF16-encoded input. re2rust assumes that the character range is 0 -- 0x10FFFF and character size is 2 bytes. This option implies --nested-ifs.
• --utf32, --unicode, -u — generate a lexer that reads UTF32-encoded input. re2rust assumes that the character range is 0 -- 0x10FFFF and character size is 4 bytes. This option implies --nested-ifs.
• --verbose — output a short message in case of success 🎉 (validation!!! we love a little win)
• --vernum, -V — show version information in MMmmpp format (major, minor, patch) 🔢
• --version, -v — show version information.
• --single-pass, -1 — deprecated. Does nothing (single pass is the default now) 💀 literally does nothing bestie, it's just vibes
• --debug-output, -d — emit YYDEBUG invocations in the generated code. This is useful to trace lexer execution 🔍
• --dump-adfa — debug option: output DFA after tunneling (in .dot format).
• --dump-cfg — debug option: output control flow graph of tag variables (in .dot format) 📊
• --dump-closure-stats — debug option: output statistics on the number of states in closure.
• --dump-dfa-det — debug option: output DFA immediately after determinization (in .dot format).
• --dump-dfa-min — debug option: output DFA after minimization (in .dot format).
• --dump-dfa-tagopt — debug option: output DFA after tag optimizations (in .dot format).
• --dump-dfa-tree — debug option: output DFA under construction with states represented as tag history trees (in .dot format) 🌳
• --dump-dfa-raw — debug option: output DFA under construction with expanded state-sets (in .dot format).
• --dump-interf — debug option: output interference table produced by liveness analysis of tag variables.
• --dump-nfa — debug option: output NFA (in .dot format).
• --emit-dot, -D — instead of normal output, generate lexer graph in .dot format 🎨 the output can be converted to an image with the help of Graphviz (e.g. something like dot -Tpng -odfa.png dfa.dot). obsessed.
• --dfa-minimization <moore | table> — internal option: DFA minimization algorithm used by re2rust. The moore option is the Moore algorithm (it is the default). The table option is the "table filling" algorithm. Both algorithms should produce the same DFA up to states relabeling; table filling is simpler and much slower and serves as a reference implementation. ngl the slow one is just there to keep the fast one honest 💅
• --eager-skip — internal option: make the generated lexer advance the input position eagerly -- immediately after reading the input symbol. This changes the default behavior, when the input position is advanced lazily -- after transition to the next state ⚡
• --no-lookahead — internal option, and it's deprecated bestie, so don't get attached 💅 It used to enable the TDFA(0) algorithm. Unlike TDFA(1), the TDFA(0) algorithm does not use one-symbol lookahead. It applies register operations to the incoming transitions rather than the outgoing ones. Benchmarks showed that the TDFA(0) algorithm is less efficient than TDFA(1). so like... yeah, that's why it's deprecated 😭
• --no-optimize-tags — internal option: suppresses optimization of tag variables. It's useful for debugging, that's literally its whole job 🫶
• --posix-closure <gor1 | gtop> — internal option: specify the shortest-path algorithm used for the construction of epsilon-closure with POSIX disambiguation semantics. gor1 stands for the Goldberg-Radzik algorithm and it's the default, and gtop stands for "global topological order" algorithm. two algorithms, one closure, iconic behavior ✨
• --posix-prectable <complex | naive> — internal option: specify the algorithm used to compute the POSIX precedence table. The complex algorithm computes the precedence table in one traversal of the tag history tree and has quadratic complexity in the number of TNFA states — and yes, it's the default 👑. The naive algorithm has worst-case cubic complexity in the number of TNFA states, but it's much simpler than complex and may be slightly faster in non-pathological cases. so: simpler, sometimes faster, but cubic when things get weird, ngl
• --stadfa — internal option, deprecated 🙈 It used to enable the staDFA algorithm, which differs from TDFA in that register operations are placed in states rather than on transitions. Benchmarks showed that the staDFA algorithm is less efficient than TDFA. the benchmark girlies said no, so it had to go 💀
• --fixed-tags <none | toplevel | all> — internal option: specify whether the fixed-tag optimization should be applied to all tags (all), none of them (none), or only those in toplevel concatenation (toplevel). The default is all. "Fixed" tags are those that are located within a fixed distance to some other tag, which is called the "base". In such cases only the base tag needs to be tracked, and the value of the fixed tag can be computed as the value of the base tag plus a static offset. For tags that are under alternative or repetition, it's also necessary to check if the base tag has a no-match value (in that case the fixed tag should also be set to no-match, disregarding the offset). For tags in top-level concatenation the check is not needed, because they always match. it's giving "why track two tags when one tag + math can do the job" 💅 math is the main character here fr

🌸 Warnings 🌸

Warnings can be individually enabled, disabled and turned into an error. yes bestie, each one, on its own, your rules 💖

• -W — turn on all warnings. all of them. every single one 🔥
• -Werror — turn warnings into errors. Note that this option alone doesn't turn on any warnings; it only affects those warnings that have been turned on so far or will be turned on later. so it's like, a vibe amplifier, not a starter button ⭐
• -W<warning> — turn on that warning.
• -Wno-<warning> — turn off that warning.
• -Werror-<warning> — turn on the warning and treat it as an error. This implies -W<warning> 🫶
• -Wno-error-<warning> — don't treat this particular warning as an error. This doesn't turn off the warning itself. it's still gonna warn you, it's just not gonna be dramatic about it 💅
• -Wcondition-order — warn if the generated program makes implicit assumptions about condition numbering. One should use either the --header option or a conditions block to generate a mapping of condition names to numbers, and then use the autogenerated condition names. don't assume numbers, use names, it's that simple bestie
• -Wempty-character-class — warn if a regular expression contains an empty character class. Trying to match an empty character class makes no sense: it should always fail. However, for backwards compatibility reasons re2rust permits empty character classes and treats them as empty strings. Use the --empty-class option to change the default behavior. it literally cannot match anything, so be suspicious 😭
• -Wmatch-empty-string — warn if a rule is nullable (matches an empty string). If the lexer runs in a loop and the empty match is unintentional, the lexer may unexpectedly hang in an infinite loop. infinite loop jumpscare fr fr 💀
• -Wswapped-range — warn if the lower bound of a range is greater than its upper bound. The default behavior is to silently swap the range bounds. it's out here fixing your range behind your back, so polite of it 🎀
• -Wundefined-control-flow — warn if some input strings cause undefined control flow in the lexer (the faulty patterns are reported). This is a dangerous and common mistake. It can be easily fixed by adding the default rule *, which has the lowest priority, matches any code unit, and always consumes a single code unit. dangerous and common?? the worst combo ngl, add the * rule and breathe 🫶
• -Wunreachable-rules — warn about rules that are shadowed by other rules and will never match. imagine being a rule that never gets to match 😭 not on my watch
• -Wdeprecated-eof_rule — warn about standalone end of input rules $ that will be broken by future changes and require fixing. At the moment these rules take precedence when conflicting with other rules, but after the introduction of the generalized end of input symbol $, precedence order will change and these rules will become shadowed by other rules. future-you is gonna have a bad time if you don't fix these now ⏳
• -Wuseless-escape — warn if a symbol is escaped when it shouldn't be. By default, re2rust silently ignores such escapes, but this may as well indicate a typo or an error in the escape sequence. a stray backslash is giving typo energy 🙈
• -Wnondeterministic-tags — warn if a tag has n-th degree of nondeterminism, where n is greater than 1.
• -Wsentinel-in-midrule — warn if the sentinel symbol occurs in the middle of a rule — this may cause reads past the end of buffer, crashes or memory corruption in the generated lexer. This warning is only applicable if the sentinel method of checking for the end of input is used. It is set to an error if the re2c:sentinel configuration is used. memory corruption bestie, do not ignore this one 💀
• -Wundefined-syntax-config — warn if the syntax file specified with --syntax option is missing definitions of some configurations. This helps to maintain user-defined syntax files: if a new release adds configurations, an old syntax file will raise a warning, and the user will be notified. If some configurations are unused and do not need a definition, they should be explicitly set to <undefined>. it's giving "your syntax file is out of date, please update your settings" 🎀

---

🦋 Syntax files 🦋

Support for different languages in re2c is based on the idea of syntax files. A syntax file is a configuration file that defines syntax of the target language — not the whole language, but a small part of it that is used by the generated code. just the little slice it needs, nothing extra, we love a minimalist 👑

Syntax files make re2c very flexible, but they should not be used as a replacement for re2c: configurations: their purpose is to define syntax of the target language, not to customize one particular lexer. different jobs, don't mix them up bestie 🫶

All supported languages have default syntax files that are part of the distribution (see include/syntax subdirectory); they are also embedded in the re2rust binary. Users may provide a custom syntax file that overrides a few configurations for one of the supported languages, or they may choose to redefine all configurations (in that case the --lang none option should be used). go off, redefine everything, just use --lang none when you do ✨

Syntax files contain configurations of four different kinds: feature lists, language configurations, inplace configurations and code templates. four flavors, all iconic 💖

🌈 Feature lists

A few list configurations define various features supported by a given backend, so that re2rust may give a clear error if the user tries to enable an unsupported feature:

• supported_apis — a list of supported APIs with possible elements simple, record, generic.
• supported_api_styles — a list of supported API styles with possible elements functions, free-form.
• supported_code_models — a list of supported code models with possible elements goto-label, loop-switch, recursive-functions.
• supported_targets — a list of supported codegen targets with possible elements code, dot, skeleton.
• supported_features — a list of supported features with possible elements nested-ifs, bitmaps, computed-gotos, case-ranges, monadic, unsafe, tags, captures, captvars, computed-continue.

🧚 Language configurations

A few boolean configurations describe features of the target language that affect the re2rust parser and code generator:

• semicolons — non-zero if the language uses semicolons after statements.
• backtick_quoted_strings — non-zero if the language has backtick-quoted strings.
• single_quoted_strings — non-zero if the language has single-quoted strings.
• indentation_sensitive — non-zero if the language is indentation sensitive.
• wrap_blocks_in_braces — non-zero if compound statements must be wrapped in curly braces.

booleans only, no maybe, no "it depends", just yes or no 💅

🎀 Inplace configurations

Syntax files define initial values of all re2c: configurations, as they may differ for different languages. See the configurations section for a full list of all inplace configurations and their meaning. the full list lives over there, go look, i'm not retyping it 🫶

💜 Code templates

Code templates define syntax of the target language. They are written in a simple domain-specific language with the following formal grammar:

code-template ::
      name '=' code-exprs ';'
    | CODE_TEMPLATE ';'
    | '<undefined>' ';'

code-exprs ::
      <EMPTY>
    | code-exprs code-expr

code-expr ::
      STRING
    | VARIABLE
    | optional
    | list

optional ::
      '(' CONDITIONAL '?' code-exprs ')'
    | '(' CONDITIONAL '?' code-exprs ':' code-exprs ')'

list ::
      '[' VARIABLE ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER '}' ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER ',' NUMBER '}' ':' code-exprs ']'

A code template is a sequence of string literals, variables, optional elements and lists, or a reference to another code template, or a special value <undefined>. Variables are placeholders that are substituted during the code generation phase. placeholders, but make it fashion ✨

List variables are special: when expanding list templates, re2rust repeats expressions on the right hand side of the column a few times, each time replacing occurrences of the list variable with a value specific to this repetition. Lists have optional bounds (negative values are counted from the end, e.g. -1 means the last element). Conditional names start with a dot. Both conditionals and variables may be either local (specific to the given code template) or global (allowed in all code templates). When re2rust reads a syntax file, it checks that each code template uses only the variables and conditionals that are allowed in it. so like, the bouncer at the door checking the guest list 👑 no randoms allowed

Okay so, for example, this code template right here defines an if-then-else construct for a C-like language 👇

code:if_then_else =
    [branch{0}: topindent "if " cond " {" nl
        indent [stmt: stmt] dedent]
    [branch{1:-1}: topindent "} else" (.cond ? " if " cond) " {" nl
        indent [stmt: stmt] dedent]
    topindent "}" nl;

So branch is a list variable: branch{0} expands to the first branch (which is special, because there's no else part), and branch{1:-1} expands to all the remaining branches (if any exist). stmt is also a list variable: [stmt: stmt] is a nested list that expands to a list of statements in the body of the current branch. topindent, indent, dedent and nl are global variables, and .cond is a local conditional (their meanings are described below). The whole vibe of this code template could produce something like this:

if x {
    // do something
} else if y {
    // do something else
} else {
    // don't do anything
}

Here's the full list of all the code templates re2rust supports, with their local variables and conditionals. Lowkey important: a particular definition may use local variables and conditionals, but it doesn't have to. And any unused code templates should be set to <undefined> ✨

• code:var_local — declaration or definition of a local variable. Supported variables: type (the type of the variable), name (its name) and init (initial value, if any). Conditionals: .init (true if there is an initializer).
• code:var_global — same as code:var_local, except that it's used in top-level.
• code:const_local — definition of a local constant. Supported variables: type (the type of the constant), name (its name) and init (initial value).
• code:const_global — same as code:const_local, except that it's used in top-level.
• code:array_local — definition of a local array (table). Supported variables: type (the type of array elements), name (array name), size (its size), row (a list variable that does not itself produce any code, but expands list expression as many times as there are rows in the table) and elem (a list variable that expands to all table elements in the current row — it's meant to be nested in the row list). Supported conditional: .const (true if the array is immutable).
• code:array_global — same as code:array_local, except that it's used in top-level.
• code:array_elem — reference to an element of an array (table). Supported variables: array (the name of the array) and index (index of the element).
• code:enum — definition of an enumeration (it may be defined using a special language construct for enumerations, or simply as a few standalone constants). Supported variables are type (user-defined enumeration type or type of the constants), elem (list variable that expands to the name of each member) and init (initializer for each member). Conditionals: .init (true if there is an initializer).
• code:enum_elem — enumeration element (a member of a user-defined enumeration type or a name of a constant, depending on how code:enum is defined). Supported variables are name (the name of the element) and type (its type).
• code:assign — assignment statement. Supported variables are lhs (left hand side) and rhs (right hand side).
• code:type_int — signed integer type.
• code:type_uint — unsigned integer type.
• code:type_yybm — type of elements in the yybm table.
• code:type_yytarget — type of elements in the yytarget table.
• code:type_yyctable — type of elements in the yyctable table.
• code:cmp_eq — operator "equals".
• code:cmp_ne — operator "not equals".
• code:cmp_lt — operator "less than".
• code:cmp_gt — operator "greater than"
• code:cmp_le — operator "less or equal"
• code:cmp_ge — operator "greater or equal"
• code:if_then_else — if-then-else statement with one or more branches. Supported variables: branch (a list variable that does not itself produce any code, but expands list expression as many times as there are branches), cond (condition of the current branch) and stmt (a list variable that expands to all statements in the current branch). Conditionals: .cond (true if the current branch has a condition), .many (true if there's more than one branch).
• code:if_then_else_oneline — a specialization of code:if_then_else for the case when all branches have one-line statements. If this is <undefined>, code:if_then_else is used instead.
• code:switch — a switch statement with one or more cases. Supported variables: expr (the switched-on expression) and case (a list variable that expands to all cases-groups with their code blocks).
• code:switch_cases — a group of switch cases that maps to a single code block. Supported variables are case (a list variable that expands to all cases in this group) and stmt (a list variable that expands to all statements in the code block.
• code:switch_cases_oneline — a specialization of code:switch_cases for the case when the code block consists of a single one-line statement. If this is <undefined>, code:switch_cases is used instead.
• code:switch_case_range — a single switch case that covers a range of values (possibly consisting of a single value). Supported variable: val (a list variable that expands to all values in the range). Supported conditionals: .many (true if there's more than one value in the range) and .char_literals (true if this is a switch on character literals — some languages provide special syntax for this case).
• code:switch_case_default — default switch case.
• code:loop — a loop that runs forever (unless interrupted from the loop body). Supported variables: label (loop label), val (initial state), var (state variable), state (a list variable that expands to all state names) and stmt (a list variable that expands to all statements in the loop body or the current state).
• code:continue — continue statement. Supported variables: label (label from which to continue execution), val (value passed to a labeled switch) and var (state variable).
• code:goto — goto statement. Supported variables: label (label of the jump target).
• code:cgoto — computed goto statement. Supported variables: array (the table containing computed goto information), index (index of the element in the table) and base (base label, only used if .cgoto.relative is true).
• code:cgoto:data — initializer expression for a single element in computed goto table. Supported variables: label (the label that is used to initialize the current element), type (underlying type of the elements in the table) and base (base label — only used if .cgoto.relative is true).
• code:fndecl — function declaration. Supported variables: name (function name), type (return type), throw (exceptions thrown by this function, maps to re2c:yyfn:throw configuration), arg (a list variable that does not itself produce code, but expands list expression as many times as there are function arguments), argname (name of the current argument), argtype (type of the current argument). Conditional: .type (true if this is a non-void function).
• code:fndef — like code:fndecl, but used for function definitions, so it has one additional list variable stmt that expands to all statements in the function body.
• code:fncall — function call statement. Supported variables: name (function name), retval (l-value where the return value is stored, if any) and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if return value needs to be saved).
• code:tailcall — tail call statement. Supported variables: name (function name), and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if this is a non-void function).
• code:recursive_functions — program body with --recursive-functions code model. Supported variables: fn (a list variable that does not itself produce any code, but expands list expression as many times as there are functions), fndecl (declaration of the current function) and fndef (definition of the current function).
• code:fingerprint — the fingerprint at the top of the generated output file. Supported variables: ver (re2rust version that was used to generate this) and date (generation date).
• code:line_info — the format of line directives (if this is set to <undefined>, no directives are generated). Supported variables: line (line number) and file (filename).
• code:abort — a statement that aborts program execution.
• code:yydebug — YYDEBUG statement, possibly specialized for different APIs. Supported variables: YYDEBUG, yyrecord, yych (map to the corresponding re2c: configurations), state (DFA state number).
• code:yypeek — YYPEEK statement, possibly specialized for different APIs. Supported variables: YYPEEK, YYCTYPE, YYINPUT, YYCURSOR, yyrecord, yych (map to the corresponding re2c: configurations). Conditionals: .cast (true if re2c:yych:conversion is set to non-zero).

okay continuing the list bestie, we're still in the syntax-file variables and there are SO many, but every single one matters 💅

• code:yyskip — the YYSKIP statement, possibly specialized for different APIs. Supported variables: YYSKIP, YYCURSOR, yyrecord (these map to the corresponding re2c: configurations). ✨
• code:yybackup — the YYBACKUP statement, possibly specialized for different APIs. Supported variables: YYBACKUP, YYCURSOR, YYMARKER, yyrecord (map to the corresponding re2c: configurations).
• code:yybackupctx — the YYBACKUPCTX statement, possibly specialized for different APIs. Supported variables: YYBACKUPCTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the corresponding re2c: configurations).
• code:yyskip_yypeek — combined code:yyskip and code:yypeek statement (defaults to code:yyskip followed by code:yypeek) 🫶
• code:yypeek_yyskip — combined code:yypeek and code:yyskip statement (defaults to code:yypeek followed by code:yyskip).
• code:yyskip_yybackup — combined code:yyskip and code:yybackup statement (defaults to code:yyskip followed by code:yybackup).
• code:yybackup_yyskip — combined code:yybackup and code:yyskip statement (defaults to code:yybackup followed by code:yyskip).
• code:yybackup_yypeek — combined code:yybackup and code:yypeek statement (defaults to code:yybackup followed by code:yypeek).
• code:yyskip_yybackup_yypeek — combined code:yyskip, code:yybackup and code:yypeek statement (defaults to code:yyskip followed by code:yybackup followed by code:yypeek) 🔥
• code:yybackup_yypeek_yyskip — combined code:yybackup, code:yypeek and code:yyskip statement (defaults to code:yybackup followed by code:yypeek followed by code:yyskip).
• code:yyrestore — the YYRESTORE statement, possibly specialized for different APIs. Supported variables: YYRESTORE, YYCURSOR, YYMARKER, yyrecord (map to the corresponding re2c: configurations).
• code:yyrestorectx — the YYRESTORECTX statement, possibly specialized for different APIs. Supported variables: YYRESTORECTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the corresponding re2c: configurations).
• code:yyrestoretag — the YYRESTORETAG statement, possibly specialized for different APIs. Supported variables: YYRESTORETAG, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), tag (the name of tag variable used to restore position) 🎀
• code:yyshift — the YYSHIFT statement, possibly specialized for different APIs. Supported variables: YYSHIFT, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), offset (the number of code units to shift the current position).
• code:yyshiftstag — the YYSHIFTSTAG statement, possibly specialized for different APIs. Supported variables: YYSHIFTSTAG, yyrecord, negative (map to the corresponding re2c: configurations), tag (tag variable which needs to be shifted), offset (the number of code units to shift). Conditionals: .nested (true if this is a nested tag — in this case its value may equal to re2c:tags:negative, which should not be shifted) ⚠️
• code:yyshiftmtag — the YYSHIFTMTAG statement, possibly specialized for different APIs. Supported variables: YYSHIFTMTAG (maps to the corresponding re2c: configuration), tag (tag variable which needs to be shifted), offset (the number of code units to shift).
• code:yystagp — the YYSTAGP statement, possibly specialized for different APIs. Supported variables: YYSTAGP, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), tag (tag variable that should be updated).
• code:yymtagp — the YYMTAGP statement, possibly specialized for different APIs. Supported variables: YYMTAGP (maps to the corresponding re2c: configuration), tag (tag variable that should be updated).
• code:yystagn — the YYSTAGN statement, possibly specialized for different APIs. Supported variables: YYSTAGN, negative, yyrecord (map to the corresponding re2c: configurations), tag (tag variable that should be updated).
• code:yymtagn — the YYMTAGN statement, possibly specialized for different APIs. Supported variables: YYMTAGN (maps to the corresponding re2c: configuration), tag (tag variable that should be updated).
• code:yycopystag — the YYCOPYSTAG statement, possibly specialized for different APIs. Supported variables: YYCOPYSTAG, yyrecord (map to the corresponding re2c: configurations), lhs, rhs (left and right hand side tag variables of the copy operation) 🦋
• code:yycopymtag — the YYCOPYMTAG statement, possibly specialized for different APIs. Supported variables: YYCOPYMTAG, yyrecord (map to the corresponding re2c: configurations), lhs, rhs (left and right hand side tag variables of the copy operation).
• code:yygetaccept — the YYGETACCEPT statement, possibly specialized for different APIs. Supported variables: YYGETACCEPT, yyrecord (map to the corresponding re2c: configurations), var (maps to re2c:yyaccept configuration).
• code:yysetaccept — the YYSETACCEPT statement, possibly specialized for different APIs. Supported variables: YYSETACCEPT, yyrecord (map to the corresponding re2c: configurations), var (maps to re2c:yyaccept configuration) and val (numeric value of the accepted rule).
• code:yygetcond — the YYGETCOND statement, possibly specialized for different APIs. Supported variables: YYGETCOND, yyrecord (map to the corresponding re2c: configurations), var (maps to re2c:yycond configuration).
• code:yysetcond — the YYSETCOND statement, possibly specialized for different APIs. Supported variables: YYSETCOND, yyrecord (map to the corresponding re2c: configurations), var (maps to re2c:yycond configuration) and val (numeric condition identifier).
• code:yygetstate — the YYGETSTATE statement, possibly specialized for different APIs. Supported variables: YYGETSTATE, yyrecord (map to the corresponding re2c: configurations), var (maps to re2c:yystate configuration).
• code:yysetstate — the YYSETSTATE statement, possibly specialized for different APIs. Supported variables: YYSETSTATE, yyrecord (map to the corresponding re2c: configurations), var (maps to re2c:yystate configuration) and val (state number).
• code:yylessthan — the YYLESSTHAN statement, possibly specialized for different APIs. Supported variables: YYLESSTHAN, YYCURSOR, YYLIMIT, yyrecord (map to the corresponding re2c: configurations), need (the number of code units to check against). Conditional: .many (true if the need is more than one) 💖
• code:yyend — the YYEND expression, possibly specialized for different APIs. Supported variables: YYEND, YYCURSOR, YYLIMIT.
• code:yybm_filter — condition that is used to filter out yych values that are not covered by the yybm table (used with --bitmaps option). Supported variable: yych (maps to re2c:yych configuration).
• code:yybm_match — the format of yybm table check (generated with --bitmaps option). Supported variables: yybm, yych (map to the corresponding re2c: configurations), offset (offset in the yybm table that needs to be added to yych) and mask (bit mask that should be applied to the table entry to retrieve the boolean value that needs to be checked).
• code:yytarget_filter — condition that is used to filter out yych values that are not covered by the yytarget table (used with --computed-gotos option). Supported variable: yych (maps to re2c:yych configuration).

---

🌈 and now… the global variables

Here's the full list of all global variables that are allowed in syntax files, no cap:

• nl — a newline.
• indent — a variable that does not produce any code, but has a side-effect of increasing indentation level. sneaky lil guy 🧚
• dedent — a variable that does not produce any code, but has a side-effect of decreasing indentation level.
• topindent — indentation string for the current statement. Indentation level is tracked and automatically updated by the code generator.

💜 global conditionals, aka the vibe check list

Here's a list of all global conditionals that are allowed in syntax files:

• .api.simple — true if simple API is used (--api simple or re2c:api = simple).
• .api.generic — true if generic API is used (--api generic or re2c:api = generic).
• .api.record — true if record API is used (--api record or re2c:api = record).
• .api_style.functions — true if function-like API style is used (re2c:api-style = functions).
• .api_style.freeform — true if free-form API style is used (re2c:api-style = free-form).
• .case_ranges — true if case ranges feature is enabled (--case-ranges or re2c:case-ranges = 1).
• .cgoto.relative — true if the relative form of computed goto is used (--computed-gotos-relative or re2c:cgoto:relative = 1).
• .computed_continue — true if loop/switch model uses labeled-switch transitions (--computed-continue).
• .code_model.goto_label — true if code model based on goto/label is used (--goto-label).
• .code_model.loop_switch — true if code model based on loop/switch is used (--loop-switch).
• .code_model.recursive_functions — true if code model based on recursive functions is used (--recursive-function).
• .date — true if the generated fingerprint should contain generation date. 🗓️
• .loop_label — true if re2rust generated loops must have a label (re2c:label:yyloop is set to a nonempty string).
• .monadic — true if the generated code should be monadic (re2c:monadic = 1). this is only relevant for pure functional languages, iykyk ✨
• .start_conditions — true if start conditions are enabled (--start-conditions). 👑
• .storable_state — true when storable state is turned on (--storable-state). 💾
• .unsafe — true when re2rust is allowed to emit unsafe blocks so the generated code can go zoom (--unsafe, re2c:unsafe = 1). Heads up bestie, this one's only relevant for languages that actually have an unsafe feature. 🏎️
• .version — true if the generated fingerprint is supposed to carry the re2rust version in it. 🏷️
• .yyfill.enable — true if YYFILL is enabled (re2c:yyfill:enable = 1).
• .yyfn.throw — true if the re2c:yyfn:throw configuration is set to a nonempty string.

---

🌊 HANDLING THE END OF INPUT 🌊

Okay so this is like, the classic lexer struggle: how do you know when to stop?? 😭 There are a few terminating conditions, and you gotta know them:

• the lexer matches some rule (yes, including the default rule *) and lands in a final state ✅
• the lexer fails to match any rule and lands in a default state 🫠
• the lexer reaches the end of input 🛑

The first two conditions end the lexer in a "natural" way: it arrives at a state with no outgoing transitions, and matching just stops on its own. Iconic. The third one, end of input, is the messy one — it can happen in any state, and the lexer has to be ready for it. Checking for end of input interrupts the normal lexer workflow and sprinkles conditional branches into your generated program, so lowkey you want to minimise how many of those checks you do. re2rust supports a few different methods for handling the end of input, and which one you pick depends on how complex your regular expressions are, whether you need buffering, performance stuff, and other factors. Here's the list, bestie:

• Sentinel. This method deletes the need for end-of-input checks entirely. Simple and efficient, but it only works when there's a natural "sentinel" character that can never show up in valid input. That character may still appear in invalid input, but your regular expressions shouldn't allow it, except maybe as the last character of a rule. The sentinel gets appended at the end of input and acts as the stop signal: when the lexer reads this character it's either a syntax error or the end of input — either way the lexer should stop. This method is used if YYFILL is disabled with re2c:yyfill:enable = 0; and re2c:eof has the default value -1.
• Sentinel with bounds checks. This one is generic — it lets you handle any input with no restrictions on the regular expressions. The idea is to cut down the number of end-of-input checks by only doing them on certain characters. Like the "sentinel" method, one of the characters is picked as a "sentinel" and appended at the end of input. But there's no restriction on where the sentinel may occur (fr, any character can be chosen as a sentinel). When the lexer reads that character, it also does a bounds check. If the current position is within bounds, the lexer picks matching back up and treats the sentinel as a regular character. Otherwise it invokes YYFILL (unless that's disabled). If more input gets supplied, the lexer rematches the last character and carries on like the sentinel was never there. Otherwise it must genuinely be the real end of input, and the lexer stops. This method is used when re2c:eof has a non-negative value (you set it to the numeric value of the sentinel). YYFILL is optional.
• Bounds checks with padding. Generic too, and it may be faster than the "sentinel with bounds checks" method, but it's also more complex. The idea: partition the DFA states into strongly connected components (SCCs) and generate a single check per SCC, covering enough characters for the longest non-looping path in that SCC. Fewer checks, slay — but there's a catch with short lexemes at the end of input, since the check needs enough characters to cover the longest lexeme. Fix: pad the input with a few fake characters that don't form a valid lexeme suffix (so the lexer literally can't match them). The padding length should be YYMAXFILL, generated with a max block. If there isn't enough input, the lexer invokes YYFILL, which should supply at least the required number of characters or just not return. This method is used if YYFILL is enabled and re2c:eof is -1 (that's the default configuration, iykyk).
• Custom checks. The generic API lets you override basic operations like reading a character, which makes it possible to fold the end-of-input checks into those operations. > ⚠️ This approach is error-prone and should be used with caution, bestie, I'm serious. To use a custom method, enable the generic API with --api custom or re2c:api = custom; and disable the default bounds checks with re2c:yyfill:enable = 0; or re2c:yyfill:check = 0;.

The following subsections have an example of each method. Obsessed with a good example, honestly. 💅

🛑 Sentinel 🛑

This example uses a sentinel character to handle the end of input. The program counts space-separated words in a null-terminated string. The sentinel is null: it's the last character of each input string, and none of the rules allow it in the middle of a lexeme (in particular it's not included in character ranges, where it'd be way too easy to overlook). If a null shows up in the middle of a string, that's a syntax error and the lexer will match the default rule *, but it won't read past the end of input or crash (use the -Wsentinel-in-midrule warning and the re2c:sentinel configuration to verify this). The configuration re2c:yyfill:enable = 0; suppresses generation of bounds checks and YYFILL invocations. Chef's kiss. 🎀

// re2rust $INPUT -o $OUTPUT --api simple

fn lex(yyinput: &[u8]) -> isize {
    // The input must be null-terminated, otherwise the function has UB.
    assert_eq!(yyinput.last(), Some(&0));

    let mut yycursor = 0;
    let mut count = 0;

    'lex: loop { /*!re2c
        re2c:YYCTYPE = u8;
        re2c:yyfill:enable = 0;

        *      { return -1; }
        [\x00] { return count; }
        [a-z]+ { count += 1; continue 'lex; }
        [ ]+   { continue 'lex; }
    */}
}

fn main() {
    assert_eq!(lex(b"\x00"), 0);
    assert_eq!(lex(b"one two three\x00"), 3);
    assert_eq!(lex(b"f0ur\x00"), -1);
}

🛑 Sentinel with bounds checks 🛑

This example uses sentinel with bounds checks to handle the end of input (this method was added in version 1.2, for the historians 🫶). The program counts space-separated single-quoted strings. The sentinel character is null, specified with the re2c:eof = 0; configuration. Just like in the sentinel method, null is the last character of each input string — but here it's allowed in the middle of a rule (for example 'aaa\0aa'\0 is valid input, but 'aaa\0 is a syntax error). Bounds checks get generated in every state that matches an input character, but they're scoped to the branch that handles null. The checks look like YYLIMIT <= YYCURSOR, or YYLESSTHAN(1) with the generic API. If the check condition is true, the lexer has hit the end of input and should stop (YYFILL is disabled with re2c:yyfill:enable = 0; because the input fits into one buffer — see the YYFILL with sentinel section for an example that actually uses YYFILL). Reaching the end of input opens up three possibilities: if the lexer is in the initial state it'll match the end-of-input rule $; otherwise it may fallback to a previously matched rule (including the default rule *) or go to a default state, which causes -Wundefined-control-flow. Fun, right? 😅

// re2rust $INPUT -o $OUTPUT --api simple

fn lex(yyinput: &[u8]) -> isize {
    // The input must be null-terminated, otherwise the function has UB.
    assert_eq!(yyinput.last(), Some(&0));

    let (mut yycursor, mut yymarker) = (0, 0);
    let yylimit = yyinput.len() - 1; // null-terminator not included
    let mut count = 0;

    'lex: loop { /*!re2c
        re2c:YYCTYPE = u8;
        re2c:yyfill:enable = 0;
        re2c:eof = 0;

        str = ['] ([^'\\] | [\\][^])* ['];

        *    { return -1; }
        $    { return count; }
        str  { count += 1; continue 'lex; }
        [ ]+ { continue 'lex; }
    */}
}

fn main() {
    assert_eq!(lex(b"\0"), 0);
    assert_eq!(lex(b"'qu\0tes' 'are' 'fine: \\'' \0"), 3);
    assert_eq!(lex(b"'unterminated\\'\0"), -1);
}

🛑 Bounds checks with padding 🛑

This example uses bounds checks with padding to handle the end of input (this is the method that's enabled by default, bestie — the main character of methods ✨). The program counts space-separated single-quoted strings. There's a padding of YYMAXFILL null characters appended at the end of input, where the YYMAXFILL value is autogenerated with a max block. You don't have to use null for padding — any characters work as long as they don't form a valid lexeme suffix (in this example the padding shouldn't contain single quotes, since they could be mistaken for the suffix of a single-quoted string). There's a "stop" rule that matches the first padding character (null) and terminates the lexer (note that it checks whether null is at the beginning of the padding — otherwise it's a syntax error 🙈). Bounds checks are only generated in some states, determined by the strongly connected components of the underlying automaton. The checks have the form (YYLIMIT - YYCURSOR) < n, or YYLESSTHAN(n) with the generic API, where n is the minimum number of characters needed for the lexer to proceed (which also means the next bounds check happens in at most n characters). If the check condition is true, the lexer has reached the end of input and will invoke YYFILL(n), which should either supply at least n input characters or not return. In this example YYFILL always fails and terminates the lexer with an error — totally fine, because the input fits into one buffer. See the YYFILL with padding section for an example that refills the input buffer with YYFILL. 🧚

// re2rust $INPUT -o $OUTPUT --api simple

/*!max:re2c*/

fn lex(s: &[u8]) -> isize {
    let mut count = 0;
    let mut yycursor = 0;
    let yylimit = s.len() + YYMAXFILL;

    // Copy string to a buffer and add YYMAXFILL zero padding.
    let mut yyinput = Vec::with_capacity(yylimit);
    yyinput.extend_from_slice(s);
    yyinput.extend([0 as u8; YYMAXFILL]);

    'lex: loop { /*!re2c
        re2c:YYCTYPE = u8;
        re2c:YYFILL = "return -1;";

        str = ['] ([^'\\] | [\\][^])* ['];

[\x00] { // Check that it is the sentinel, not some unexpected null. return if yycursor == s.len() + 1 { count } else { -1 } } str { count += 1; continue 'lex; } [ ]+ { continue 'lex; }

• { return -1; } */} }

fn main() { assert_eq!(lex(b""), 0); assert_eq!(lex(b"'qu\0tes' 'are' 'fine: \'' "), 3); assert_eq!(lex(b"'unterminated\'"), -1); assert_eq!(lex(b"'unexpected \0 null"), -1); }

## 🎀 Custom checks 🎀

This example uses a **custom end-of-input handling method based on generic API**. The program counts space-separated single-quoted strings, and it is <pink>literally</pink> the same as the sentinel example, except that the input is **not** null-terminated. To cover up for the absence of a sentinel character at the end of input, `YYPEEK` is redefined to perform a bounds check before it reads the next input character. <yellow>Ngl this is inefficient</yellow> because checks are done <hotpink>very often</hotpink>. If the check condition fails, `YYPEEK` returns the real character, otherwise it returns a fake sentinel character.

// re2rust $INPUT -o $OUTPUT

// Expect a string without terminating null. fn lex(s: &[u8]) -> isize { let mut count = 0; let mut cur = 0; let lim = s.len();

'lex: loop {/*!re2c re2c:YYCTYPE = u8; re2c:YYPEEK = "if cur < lim {*s.get_unchecked(cur)} else {0}"; re2c:YYSKIP = "cur += 1;"; re2c:yyfill:enable = 0;

• { return -1; } [\x00] { return count; } [a-z]+ { count += 1; continue 'lex; } [ ]+ { continue 'lex; } */} }

fn main() { assert_eq!(lex(b""), 0); assert_eq!(lex(b"one two three "), 3); assert_eq!(lex(b"f0ur"), -1); }

---

# 🦋 BUFFER REFILLING 🦋

okay so <purple>bestie, why do we even need buffering?</purple> The need for buffering arises when the input **cannot** be mapped in memory all at once: either it is too large, or it comes in a streaming fashion (like reading from a socket). The usual technique in such cases is to allocate a fixed-sized memory buffer and process input in chunks that fit into the buffer. When the current chunk is processed, it is moved out and new data is moved in. In practice it is somewhat more complex, because lexer state consists not of a single input position, but a <hotpink>set of interrelated positions</hotpink>:

- **cursor**: the next input character to be read (`YYCURSOR` in C pointer API or `YYSKIP`/`YYPEEK` in generic API)
- **limit**: the position after the last available input character (`YYLIMIT` in C pointer API, implicitly handled by `YYLESSTHAN` in generic API)
- **marker**: the position of the most recent match, if any (`YYMARKER` in default API or `YYBACKUP`/`YYRESTORE` in generic API)
- **token**: the start of the current lexeme (implicit in `re2rust` API, as it is not needed for the normal lexer operation and can be defined and updated by the user)
- **context marker**: the position of the trailing context (`YYCTXMARKER` in C pointer API or `YYBACKUPCTX`/`YYRESTORECTX` in generic API)
- **tag variables**: submatch positions (defined with `stags` and `mtags` blocks and generic API primitives `YYSTAGP`/`YYSTAGN`/`YYMTAGP`/`YYMTAGN`)

Not all these are used in every case, but if used, they must be updated by `YYFILL`. All active positions are contained in the segment between token and cursor, therefore everything between buffer start and token can be discarded, the segment from token and up to limit should be moved to the beginning of buffer, and the free space at the end of buffer should be filled with new data. In order to avoid frequent `YYFILL` calls it is best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). The details of `YYFILL` implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds-checking with padding. Also note that if `-f` `--storable-state` option is used, `YYFILL` has slightly different semantics (described in the section about storable state).

## ✨ YYFILL with sentinel ✨

If EOF rule is used, `YYFILL` is a function-like primitive that accepts <cyan>no arguments</cyan> and returns a value which is checked against zero. `YYFILL` invocation is triggered by condition `YYLIMIT <= YYCURSOR` in C pointer API and `YYLESSTHAN()` in generic API. A non-zero return value means that `YYFILL` has **failed**. A successful `YYFILL` call must supply at least one character and adjust input positions accordingly. Limit must always be set to one after the last input position in buffer, and the character at the limit position must be the sentinel symbol specified by `re2c:eof` configuration. The pictures below show the relative locations of input positions in buffer before and after `YYFILL` call (sentinel symbol is marked with `#`, and the second picture shows the case when there is not enough input to fill the whole buffer).

<-- shift -->

-A------------B---------C-------------D#-----------E->

buffer token marker limit, cursor

-A------------B---------C-------------D------------E#->

buffer, marker cursor limit token

<-- shift -->

-A------------B---------C-------------D#--E (EOF)

buffer token marker limit, cursor

-A------------B---------C-------------D---E#........

buffer, marker cursor limit token

Here is an example of a program that reads input file `input.txt` in chunks of 4096 bytes and uses EOF rule. <green>iconic behavior, honestly</green> 🫶

// re2rust $INPUT -o $OUTPUT

use std::fs::File; use std::io::{Read, Write};

const BUFSIZE: usize = 4096;

struct State { file: File, yyinput: [u8; BUFSIZE], yylimit: usize, yycursor: usize, yymarker: usize, token: usize, eof: bool, }

#[derive(PartialEq)] enum Fill { Ok, Eof, LongLexeme }

fn fill(st: &mut State) -> Fill { if st.eof { return Fill::Eof; }

// Error: lexeme too long. In real life could reallocate a larger buffer. if st.token < 1 { return Fill::LongLexeme; }

// Shift buffer contents (discard everything up to the current token). st.yyinput.copy_within(st.token..st.yylimit, 0); st.yylimit -= st.token; st.yycursor -= st.token; st.yymarker = st.yymarker.overflowing_sub(st.token).0; // may underflow if marker is unused st.token = 0;

// Fill free space at the end of buffer with new data from file. match st.file.read(&mut st.yyinput[st.yylimit..BUFSIZE - 1]) { // -1 for sentinel Ok(n) => { st.yylimit += n; st.eof = n == 0; // end of file st.yyinput[st.yylimit] = 0; // append sentinel } Err(why) => panic!("cannot read from file: {}", why) }

return Fill::Ok; }

fn lex(yyrecord: &mut State) -> isize { let mut count: isize = 0;

'lex: loop { yyrecord.token = yyrecord.yycursor; /*!re2c re2c:api = record; re2c:YYCTYPE = u8; re2c:YYFILL = "fill(yyrecord) == Fill::Ok"; re2c:eof = 0;

str = ['] ([^'\] | [\][^])* ['];

• { return -1; } $ { return count; } str { count += 1; continue 'lex; } [ ]+ { continue 'lex; } */} }

fn main() { let fname = "input"; let content = b"'qu\0tes' 'are' 'fine: \'' ";

// Prepare input file: a few times the size of the buffer, containing // strings with zeroes and escaped quotes. match File::create(fname) { Err(why) => panic!("cannot open {}: {}", fname, why), Ok(mut file) => match file.write_all(&content.repeat(BUFSIZE)) { Err(why) => panic!("cannot write to {}: {}", fname, why), Ok(_) => {} } }; let count = 3 * BUFSIZE; // number of quoted strings written to file

// Reopen input file for reading. let file = match File::open(fname) { Err(why) => panic!("cannot read file {}: {}", fname, why), Ok(file) => file, };

// Initialize lexer state: all offsets are at the end of buffer. let yylimit = BUFSIZE - 1; let mut st = State { file: file, // Sentinel (at yylimit offset) is set to null, which triggers YYFILL. yyinput: [0; BUFSIZE], yylimit: yylimit, yycursor: yylimit, yymarker: yylimit, token: yylimit, eof: false, };

// Run the lexer. assert_eq!(lex(&mut st), count as isize);

// Cleanup: remove input file. match std::fs::remove_file(fname) { Err(why) => panic!("cannot remove {}: {}", fname, why), Ok(_) => {} } }

## 🌸 YYFILL with padding 🌸

In the default case (when EOF rule is **not** used) `YYFILL` is a function-like primitive that accepts a single argument and does <red>not</red> return any value. `YYFILL` invocation is triggered by condition `(YYLIMIT - YYCURSOR) < n` in C pointer API and `YYLESSTHAN(n)` in generic API. The argument passed to `YYFILL` is the minimal number of characters that must be supplied. If it fails to do so, `YYFILL` must not return to the lexer (for that reason it is best implemented as a macro that returns from the calling function on failure). In case of a successful `YYFILL` invocation the limit position must be set either to one after the last input position in buffer, or to the end of `YYMAXFILL` padding (in case `YYFILL` has successfully read at least `n` characters, but not enough to fill the entire buffer). The pictures below show the relative locations of input positions in buffer before and after `YYFILL` invocation (`YYMAXFILL` padding on the second picture is marked with `#` symbols).

<-- shift --> <-- need -->

-A------------B---------C-----D-------E---F--------G->

buffer token marker cursor limit

-A------------B---------C-----D-------E---F--------G->

buffer, marker cursor limit token

-A------------B---------C-----D-------E-F (EOF)

buffer token marker cursor limit

-A------------B---------C-----D-------E-F###############

buffer, marker cursor limit token <- YYMAXFILL ->

<pink>okay so</pink> the top picture is giving "we ran out of room and everything is sad", and the bottom one is giving "we slid everything to the left and stuffed the tail with `YYMAXFILL` worth of padding". Both are iconic in their own way tbh 🎀

---

Here's a whole example, bestie — it reads `input.txt` in chunks of `4096` bytes and uses bounds-checking with padding. <purple>It's a lot but every line matters, so read it slow</purple> ✨

// re2rust $INPUT -o $OUTPUT

use std::fs::File; use std::io::{Read, Write};

/!max:re2c/ const BUFSIZE: usize = 4096;

struct State { file: File, yyinput: [u8; BUFSIZE], yylimit: usize, yycursor: usize, yymarker: usize, token: usize, eof: bool, }

#[derive(PartialEq)] enum Fill { Ok, Eof, LongLexeme }

fn fill(st: &mut State, need: usize) -> Fill { if st.eof { return Fill::Eof; }

// Error: lexeme too long. In real life can reallocate a larger buffer. if st.token < need { return Fill::LongLexeme; }

// Shift buffer contents (discard everything up to the current token). st.yyinput.copy_within(st.token..st.yylimit, 0); st.yylimit -= st.token; st.yycursor -= st.token; st.yymarker = st.yymarker.overflowing_sub(st.token).0; // underflows if marker is unused st.token = 0;

// Fill free space at the end of buffer with new data from file. let n = match st.file.read(&mut st.yyinput[st.yylimit..BUFSIZE - YYMAXFILL]) { Ok(n) => n, Err(why) => panic!("cannot read from file: {}", why) }; st.yylimit += n;

// If read zero characters, this is end of input => add zero padding // so that the lexer can access characters at the end of buffer. if n == 0 { st.eof = true; for i in 0..YYMAXFILL { st.yyinput[st.yylimit + i] = 0; } st.yylimit += YYMAXFILL; }

return Fill::Ok; }

fn lex(yyrecord: &mut State) -> isize { let mut count: isize = 0;

'lex: loop { yyrecord.token = yyrecord.yycursor; /*!re2c re2c:api = record; re2c:YYCTYPE = u8; re2c:YYFILL = "if fill(yyrecord, @@) != Fill::Ok { return -1; }";

str = ['] ([^'\] | [\][^])* ['];

[\x00] { // Check that it is the sentinel, not some unexpected null. return if yyrecord.token == yyrecord.yylimit - YYMAXFILL { count } else { -1 } } str { count += 1; continue 'lex; } [ ]+ { continue 'lex; }

• { return -1; } */} }

fn main() { let fname = "input"; let content = b"'qu\0tes' 'are' 'fine: \'' ";

// Prepare input file: a few times the size of the buffer, containing // strings with zeroes and escaped quotes. match File::create(fname) { Err(why) => panic!("cannot open {}: {}", fname, why), Ok(mut file) => match file.write_all(&content.repeat(BUFSIZE)) { Err(why) => panic!("cannot write to {}: {}", fname, why), Ok(_) => {} } }; let count = 3 * BUFSIZE; // number of quoted strings written to file

// Reopen input file for reading. let file = match File::open(fname) { Err(why) => panic!("cannot read file {}: {}", fname, why), Ok(file) => file, };

// Initialize lexer state: all offsets are at the end of buffer. // This immediately triggers YYFILL, as the YYLESSTHAN condition is true. let yylimit = BUFSIZE - YYMAXFILL; let mut st = State { file: file, yyinput: [0; BUFSIZE], yylimit: yylimit, yycursor: yylimit, yymarker: yylimit, token: yylimit, eof: false, };

// Run the lexer. assert_eq!(lex(&mut st), count as isize);

// Cleanup: remove input file. match std::fs::remove_file(fname) { Err(why) => panic!("cannot remove {}: {}", fname, why), Ok(_) => {} } }

---

# ✨ FEATURES ✨

## 🌈 Multiple blocks

<pink>okay so</pink> sometimes one lexer just isn't the vibe and you need <em>several interrelated lexers</em> — think a high-level state machine that hops between lexer modes 🔥 You can do this with multiple connected `re2rust` blocks. Another option is to use start conditions.

How the connections actually work depends on your target language, periodt:

- In languages that have a `goto` statement (such as <cyan>C/C++</cyan> and <cyan>Go</cyan>) you can have all blocks in one function, each of them prefixed with a label. Transition from one block to another is a simple `goto` 💅
- In languages that do <strong>not</strong> have `goto` (such as <hotpink>Rust</hotpink>) you have to use a loop with a switch on a state variable — basically the same `yystate` loop/switch that `re2rust` generates — or else wrap each block in a function and use function calls 🫶

The example below uses multiple blocks to parse binary, octal, decimal and hexadecimal numbers. Each base has its own block. The initial block determines base and dispatches to other blocks. Common configurations are defined in a separate block at the beginning of the program; they are inherited by the other blocks 🌸

// re2rust $INPUT -o $OUTPUT

// Store u32 number in u64 during parsing to simplify overflow handling. struct State<'a> { yyinput: &'a [u8], yycursor: usize, yymarker: usize, num: u64, }

/*!re2c // Common re2c definitions shared between all functions. re2c:api = record; re2c:yyrecord = st; re2c:yyfill:enable = 0; re2c:YYCTYPE = u8; */

const ERROR: u64 = std::u32::MAX as u64 + 1; // overflow

macro_rules! maybe { // Convert the number from u64 to optional u32. ($n:expr) => { if $n < ERROR { Some($n as u32) } else { None } } }

// Add digit with the given base, checking for overflow. fn add(st: &mut State, offs: u8, base: u64) { let digit = unsafe { st.yyinput.get_unchecked(st.yycursor - 1) } - offs; st.num = std::cmp::min(st.num * base + digit as u64, ERROR); }

fn parse_u32(s: & [u8]) -> Option<u32> { assert_eq!(s.last(), Some(&0)); // expect null-terminated input

let mut st = State {yyinput: s, yycursor: 0, yymarker: 0, num: 0}; /*!re2c '0b' / [01] { return parse_bin(&mut st); } "0" { return parse_oct(&mut st); } "" / [1-9] { return parse_dec(&mut st); } '0x' / [0-9a-fA-F] { return parse_hex(&mut st); }

• { return None; } */ }

fn parse_bin(st: &mut State) -> Option<u32> { 'bin: loop {/*!re2c [01] { add(st, 48, 2); continue 'bin; }

• { return maybe!(st.num); } */} }

fn parse_oct(st: &mut State) -> Option<u32> { 'oct: loop {/*!re2c [0-7] { add(st, 48, 8); continue 'oct; }

• { return maybe!(st.num); } */} }

fn parse_dec(st: &mut State) -> Option<u32> { 'dec: loop {/*!re2c [0-9] { add(st, 48, 10); continue 'dec; }

• { return maybe!(st.num); } */} }

fn parse_hex(st: &mut State) -> Option<u32> { 'hex: loop {/*!re2c [0-9] { add(st, 48, 16); continue 'hex; } [a-f] { add(st, 87, 16); continue 'hex; } [A-F] { add(st, 55, 16); continue 'hex; }

• { return maybe!(st.num); } */} }

fn main() { assert_eq!(parse_u32(b"\0"), None); assert_eq!(parse_u32(b"1234567890\0"), Some(1234567890)); assert_eq!(parse_u32(b"0b1101\0"), Some(13)); assert_eq!(parse_u32(b"0x7Fe\0"), Some(2046)); assert_eq!(parse_u32(b"0644\0"), Some(420)); assert_eq!(parse_u32(b"9999999999\0"), None); }

---

## 🦋 Start conditions

Start conditions are enabled with the `--start-conditions` option 🎀 They give you a way to encode multiple interrelated automata inside the <em>same</em> `re2rust` block. It's giving "one block, many personalities" 💅

Each condition corresponds to a single automaton and has:

- a unique name specified by the user, and
- a unique internal number defined by `re2rust` ✨

The numbers are what you use to switch between conditions: the generated code uses `YYGETCOND` and `YYSETCOND` primitives to get the current condition or set it to the given number. Use the `conditions` block, the `--header` option or the `re2c:header` configuration to generate numeric condition identifiers. And the `re2c:cond:enumprefix` configuration specifies the generated identifier prefix 🔥

<yellow>In condition mode</yellow> every rule <strong>must</strong> be prefixed with a list of comma-separated condition names in angle brackets, or a wildcard `<*>` to denote all conditions. The rule syntax is extended as follows, bestie 👑

- **`< condition-list > regular-expression code`** — a rule that is merged to every condition on the *condition-list*. It matches *regular-expression* and executes the associated *code*.

- **`< condition-list > regular-expression => condition code`** — a rule that is merged to every condition on the *condition-list*. It matches *regular-expression*, sets the current condition to *condition* and executes the associated *code*.

- **`< condition-list > regular-expression :=> condition`** — a rule that is merged to every condition on the *condition-list*. It matches *regular-expression* and immediately transitions to *condition* (there is no semantic action). <pink>no action, just vibes</pink> ✨

- **`< condition-list > !action code`** — a rule that binds *code* to the place defined by *action* in every condition on the *condition-list* (see the actions section for various types of actions).

- **`<! condition-list > code`** — a rule that prepends *code* to semantic actions of all rules for every condition on the *condition-list*.

> <red>heads up, bestie 💀</red> This last syntax is <strong>deprecated</strong> and the `!pre_rule` action should be used instead (it does exactly the same).

okay so these three special rules are giving *condition plumbing* and you need to know all of them, no skipping 💅

- **`< > code`** — this rule makes a special entry condition with number zero and the name `"0"`, and it runs `code` *before* jumping to other conditions. **ngl, this syntax is deprecated** 🙈 — you should use the `!entry` action instead, because it gives you more fine-grained control (you can specify the code on a per-condition basis, and you can jump directly to a condition's start without going through condition dispatch). so like, use the new thing, bestie 🫶
- **`< > => condition code`** — same as the previous rule, except it also sets the next condition.
- **`< > :=> condition`** — same as the previous rule, except it has no associated code and *immediately* jumps to `condition`.

---

the code `re2rust` generates for conditions depends on whether `re2rust` goes with the goto/label approach or the loop/switch approach to encode the automata. two whole vibes, okay 💫

in languages that actually *have* a `goto` statement (think C/C++ and Go), conditions are naturally implemented as blocks of code prefixed with labels of the form `yyc_<cond>`, where *cond* is a condition name (and yeah, that label prefix can be changed with `re2c:cond:prefix` ✨). transitions between conditions are done with `goto` and condition labels. before all conditions, `re2rust` generates an initial switch on `YYGETSTATE` that jumps to the start state of the current condition. the shortcut rules `:=>` bypass that initial switch and jump *directly* to the specified condition (`re2c:cond:goto` is your girl if you want to change the default behavior). and heads up: the rules with semantic actions do **not** automatically jump to the next condition — that jump is on you, in your user-defined action code. you got this 💪

in languages that do *not* have `goto` (hi, Rust 🌸), `re2rust` reuses the `yystate` variable to store condition numbers. each condition gets a numeric identifier equal to the number of its start state, and a switch between conditions is literally no different than a switch between DFA states of a single condition. so there's no need for a separate initial condition switch, periodt. (and since the same approach is used to implement storable states, `YYGETCOND`/`YYSETCOND` are redundant if you're using both storable states *and* conditions — lowkey iconic how that works out 🔥)

---

okay so the program below uses start conditions to parse binary, octal, decimal and hexadecimal numbers. there's a single block where each base gets its own condition, and the initial condition is connected to all of them. the user-defined variable `cond` stores the current condition number, and it's initialized to the number of the initial condition generated with the `conditions` block 👑

// re2rust $INPUT -o $OUTPUT -c --api simple

/!conditions:re2c/

const ERROR: u64 = std::u32::MAX as u64 + 1; // overflow

// Add digit with the given base, checking for overflow. fn add(num: &mut u64, str: &[u8], cur: usize, offs: u8, base: u64) { let digit = unsafe { str.get_unchecked(cur - 1) } - offs; *num = std::cmp::min(*num * base + digit as u64, ERROR); }

fn parse_u32(yyinput: &[u8]) -> Option<u32> { assert_eq!(yyinput.last(), Some(&0)); // expect null-terminated input

let (mut yycursor, mut yymarker) = (0, 0); let mut yycond = YYC_INIT; let mut num = 0u64; // Store number in u64 to simplify overflow checks.

'lex: loop { /*!re2c re2c:YYCTYPE = u8; re2c:yyfill:enable = 0;

<INIT> '0b' / [01] :=> BIN <INIT> "0" :=> OCT <INIT> "" / [1-9] :=> DEC <INIT> '0x' / [0-9a-fA-F] :=> HEX <INIT> * { return None; }

<BIN> [01] { add(&mut num, yyinput, yycursor, 48, 2); continue 'lex; } <OCT> [0-7] { add(&mut num, yyinput, yycursor, 48, 8); continue 'lex; } <DEC> [0-9] { add(&mut num, yyinput, yycursor, 48, 10); continue 'lex; } <HEX> [0-9] { add(&mut num, yyinput, yycursor, 48, 16); continue 'lex; } <HEX> [a-f] { add(&mut num, yyinput, yycursor, 87, 16); continue 'lex; } <HEX> [A-F] { add(&mut num, yyinput, yycursor, 55, 16); continue 'lex; }

<BIN, OCT, DEC, HEX> * { return if num < ERROR { Some(num as u32) } else { None }; } */} }

fn main() { assert_eq!(parse_u32(b"\0"), None); assert_eq!(parse_u32(b"1234567890\0"), Some(1234567890)); assert_eq!(parse_u32(b"0b1101\0"), Some(13)); assert_eq!(parse_u32(b"0x7Fe\0"), Some(2046)); assert_eq!(parse_u32(b"0644\0"), Some(420)); assert_eq!(parse_u32(b"9999999999\0"), None); }

---

## 💾 Storable state 💾

okay so with the `--storable-state` option, `re2rust` generates a lexer that can **store its current state, return to the caller, and later resume operations exactly where it left off** 😭 that's the dream, honestly.

the default mode of operation in `re2rust` is a "pull" model — the lexer "pulls" more input whenever it needs it. and that's usually fine! but it can be *unacceptable* in cases where the input shows up piece by piece. like, for example, if the lexer is invoked by the parser, or if the lexer program communicates via a socket protocol with some other program that has to wait for a reply from the lexer before it transmits the next message 🙈

storable state is intended **exactly** for those cases: it lets you generate lexers that work in a "push" model. when the lexer needs more input, it stores its state and returns to the caller. later, when more input becomes available, the caller resumes the lexer exactly where it stopped. so clean, so tidy ✨

there are a few changes you gotta make compared to the "pull" model:

- Define `YYSETSTATE()` and `YYGETSTATE(state)` primitives.
- Define `yych`, `yyaccept` (if used) and state variables as a part of persistent lexer state. the state variable should be initialized to `-1`.
- `YYFILL` should return to the outer program instead of trying to supply more input. the return code should indicate that the lexer needs more input.
- the outer program should recognize situations when the lexer needs more input and respond appropriately.
- Optionally use the `getstate` block to generate a `YYGETSTATE` switch detached from the main lexer. this only works for languages that have `goto` (so **not** in `--loop-switch` mode 🫠).
- Use `re2c:eof` and the sentinel with bounds checks method to handle the end of input. the padding-based method may not work here, because it's unclear when to append padding: the current end of input may not be the *ultimate* end of input, and appending padding too early may cut off a partially read greedy lexeme. furthermore, due to high-level program logic, getting more input may depend on processing the lexeme at the end of the buffer — which is already blocked due to the end-of-input condition. so yeah, it's giving circular dependency 💀

---

here's an example of a "push" model lexer that simulates reading packets from a socket 🧚 the lexer loops until it hits the end of input and returns to the calling function. the calling function provides more input by "sending" the next packet and resumes lexing. this process stops when all the packets have been sent, or when there's an error ⭐

// re2rust $INPUT -o $OUTPUT -f

use std::fs::File; use std::io::{Read, Write};

const DEBUG: bool = false; macro_rules! log { ($($fmt:expr)? $(, $args:expr)) => { if DEBUG { println!($($fmt)? $(, $args)) } } }

// Use a small buffer to cover the case when a lexeme doesn't fit. // In real world use a larger buffer. const BUFSIZE: usize = 10;

struct State { file: File, yyinput: [u8; BUFSIZE], yylimit: usize, yycursor: usize, yymarker: usize, token: usize, yystate: isize, }

#[derive(Debug, PartialEq)] enum Status {End, Ready, Waiting, BadPacket, BigPacket}

fn fill(st: &mut State) -> Status { // Error: lexeme too long. In real life can reallocate a larger buffer. if st.token < 1 { return Status::BigPacket; }

// Shift buffer contents (discard everything up to the current lexeme). st.yyinput.copy_within(st.token..st.yylimit, 0); st.yylimit -= st.token; st.yycursor -= st.token; st.yymarker = st.yymarker.overflowing_sub(st.token).0; // underflows if marker is unused st.token = 0;

// Fill free space at the end of buffer with new data. match st.file.read(&mut st.yyinput[st.yylimit..BUFSIZE - 1]) { // -1 for sentinel Ok(n) => { st.yylimit += n; st.yyinput[st.yylimit] = 0; // append sentinel symbol }, Err(why) => panic!("cannot read from file: {}", why) }

return Status::Ready; }

fn lex(yyrecord: &mut State, recv: &mut usize) -> Status { let mut yych; 'lex: loop { yyrecord.token = yyrecord.yycursor; /*!re2c re2c:api = record; re2c:eof = 0; re2c:YYCTYPE = "u8"; re2c:YYFILL = "return Status::Waiting;";

packet = [a-z]+[;];

• { return Status::BadPacket; } $ { return Status::End; } packet { *recv += 1; continue 'lex; } */} }

fn test(packets: Vec<&[u8]>, expect: Status) { // Create a pipe (open the same file for reading and writing). let fname = "pipe"; let mut fw: File = match File::create(fname) { Err(why) => panic!("cannot open {}: {}", fname, why), Ok(file) => file, }; let fr: File = match File::open(fname) { Err(why) => panic!("cannot read file {}: {}", fname, why), Ok(file) => file, };

// Initialize lexer state: state value is -1, all offsets are at the end // of buffer, the character at yylimit offset is the sentinel (null). let yylimit = BUFSIZE - 1; let mut state = State { file: fr, // Sentinel (at yylimit offset) is set to null, which triggers YYFILL. yyinput: [0; BUFSIZE], yylimit: yylimit, yycursor: yylimit, yymarker: yylimit, token: yylimit, yystate: -1, };

// Main loop. The buffer contains incomplete data which appears packet by // packet. When the lexer needs more input it saves its internal state and // returns to the caller which should provide more input and resume lexing. let mut status; let mut send = 0; let mut recv = 0; loop { status = lex(&mut state, &mut recv); if status == Status::End { log!("done: got {} packets", recv); break; } else if status == Status::Waiting { log!("waiting..."); if send < packets.len() { log!("sent packet {}", send); match fw.write_all(packets[send]) { Err(why) => panic!("cannot write to {}: {}", fname, why), Ok(_) => send += 1, } } status = fill(&mut state); log!("queue: '{}'", String::from_utf8_lossy(&state.yyinput)); if status == Status::BigPacket { log!("error: packet too big"); break; } assert_eq!(status, Status::Ready); } else { assert_eq!(status, Status::BadPacket); log!("error: ill-formed packet"); break; } }

// Check results. assert_eq!(status, expect); if status == Status::End { assert_eq!(recv, send); }

// Cleanup: remove input file. match std::fs::remove_file(fname) { Err(why) => panic!("cannot remove {}: {}", fname, why), Ok(_) => {} } }

fn main() { test(vec![], Status::End); test(vec![b"zero;", b"one;", b"two;", b"three;", b"four;"], Status::End); test(vec![b"zer0;"], Status::BadPacket); test(vec![b"goooooooooogle;"], Status::BigPacket); }

Okay like, that's the *entire* vibe of the harness 🫶 — `main` just hits `test` four times: empty input, a nice lil' multi-packet success case, one ill-formed packet, and one absolute unit of a packet that's too big. Each one asserts on the `Status` it expects. Iconic.

---

## 🎀 Reusable blocks 🎀

This is the part that makes re2rust feel like a group project that *actually* works, bestie 💖. Reusable blocks of the form `/*!rules:re2c[:<name>] ... */` or `%{rules[:<name>] ... %}` can be reused **any number of times** and combined with other re2rust blocks.

- The `<name>` is optional. Cute nickname, not mandatory 💅
- A rules block can be used in a *use block* or *directive*.
- The code for a rules block is generated **at every point of use** — like, every single time, no shortcuts.

**Use blocks** are defined with `/*!use:re2c[:<name>] ... */` or `%{use[:<name>] ... %}`. The `<name>` is optional: if it's not specified, the associated rules block is the most recent one (whether named or unnamed). A use block can add named definitions, configurations and rules of its own. An important use case for use blocks is a lexer that supports multiple input encodings: the same rules block is reused multiple times with encoding-specific configurations (see the example below 👇).

The in-block use directive `!use:<name>;` can be used from inside of a re2rust block. It merges the referenced block `<name>` into the current one. If some of the merged rules and configurations overlap with the previously defined ones, conflicts are resolved in the usual way: the earliest rule takes priority, and latest configuration overrides preceding ones. One exception are the special rules `*`, `$` and (in condition mode) `<!>`, for which a block-local definition overrides any inherited ones. Use directive allows one to combine different re2rust blocks together in one block (see the example below).

History check, iykyk 🌈:

- Named blocks and in-block use directive were added in **re2rust version 2.2**.
- Since that version reusable blocks are allowed by default (no special option is needed).
- Before version 2.2 reuse mode was enabled with `-r`, `--reusable` option.
- Before version 1.2 reusable blocks could not be mixed with normal blocks.

### 🌸 Example of a `!use` directive 🌸

// re2rust $INPUT -o $OUTPUT --api simple

// This example shows how to combine reusable re2c blocks: two blocks // ('colors' and 'fish') are merged into one. The 'salmon' rule occurs // in both blocks; the 'fish' block takes priority because it is used // earlier. Default rule * occurs in all three blocks; the local (not // inherited) definition takes priority.

#[derive(Debug, PartialEq)] enum Ans { Color, Fish, Dunno }

/*!rules:re2c:colors

• { panic!("ah"); } "red" | "salmon" | "magenta" { return Ans::Color; } */

/*!rules:re2c:fish

• { panic!("oh"); } "haddock" | "salmon" | "eel" { return Ans::Fish; } */

fn lex(yyinput: &[u8]) -> Ans { assert!(yyinput.len() > 0); // expect nonempty input

let (mut yycursor, mut yymarker) = (0, 0); /*!re2c re2c:yyfill:enable = 0; re2c:YYCTYPE = u8;

!use:fish; !use:colors;

• { return Ans::Dunno; } // overrides inherited '*' rules */ }

fn main() { assert_eq!(lex(b"salmon"), Ans::Fish); assert_eq!(lex(b"what?"), Ans::Dunno); }

Look at `"salmon"` doing double duty in both blocks and the `fish` block winning because it's used earlier — main character energy, honestly 👑.

### 🦋 Example of a `/*!use:re2c ... */` block 🦋

// re2rust $INPUT -o $OUTPUT --input-encoding utf8 --api simple

// This example supports multiple input encodings: UTF-8 and UTF-32. // Both lexers are generated from the same rules block, and the use // blocks add only encoding-specific configurations. /*!rules:re2c re2c:yyfill:enable = 0;

"∀x ∃y" { return Some(yycursor); }

• { return None; } */

fn lex_utf8(yyinput: &[u8]) -> Option<usize> { assert!(yyinput.len() > 0); // expect nonempty input let (mut yycursor, mut yymarker) = (0, 0); /*!use:re2c re2c:encoding:utf8 = 1; re2c:YYCTYPE = u8; */ }

fn lex_utf32(yyinput: &[u32]) -> Option<usize> { assert!(yyinput.len() > 0); // expect nonempty input let (mut yycursor, mut yymarker) = (0, 0); /*!use:re2c re2c:encoding:utf32 = 1; re2c:YYCTYPE = u32; */ }

fn main() { let s8 = vec![0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79]; assert_eq!(lex_utf8(&s8), Some(s8.len()));

let s32 = vec![0x2200, 0x78, 0x20, 0x2203, 0x79]; assert_eq!(lex_utf32(&s32), Some(s32.len())); }

One rules block, two lexers, zero duplicated rules — literally the dream 🌟.

---

## 💜 Submatch extraction 💜

re2rust has **two** options for submatch extraction. Pick your fighter 🥊

### ⭐ Tags ⭐

The first option is to use standalone tags of the form `@stag` or `#mtag`, where `stag` and `mtag` are arbitrary user-defined names. Tags are enabled with `-T`, `--tags` option or `re2c:tags = 1` configuration.

Semantically tags are position markers: they can be inserted anywhere in a regular expression, and they bind to the corresponding position (or multiple positions) in the input string. S-tags bind to the last matching position, and m-tags bind to a list of positions (they may be used in repetition subexpressions, where a single position in a regular expression corresponds to multiple positions in the input string). All tags should be defined by the user, either manually or with the help of `svars` and `mvars` blocks. If there is more than one way tags can be matched against the input, ambiguity is resolved using leftmost greedy disambiguation strategy.

### 🎀 Captures 🎀

The second option is to use capturing groups. They are enabled with `--captures` option or `re2c:captures = 1` configuration. There are two flavours for different disambiguation policies: `--leftmost-captures` (the default) is for leftmost greedy policy, and `--posix-captures` is for POSIX longest-match policy. In this mode all parenthesized subexpressions are considered capturing groups, and a bang can be used to mark non-capturing groups: `(! ... )`. With `--invert-captures` option or `re2c:invert-captures = 1` configuration the meaning of bang is inverted.

The number of groups for the matching rule is stored in a variable `yynmatch` (the whole regular expression is group number zero), and submatch results are stored in `yypmatch` array. Both `yynmatch` and `yypmatch` should be defined by the user, and `yypmatch` size must be at least `[yynmatch * 2]`. Use `maxnmatch` block to define `YYMAXNMATCH`, a constant that equals to the maximum value of `yynmatch` among all rules.

### 🍓 Captvars 🍓

Another way to use capturing groups is the `--captvars` option or `re2c:captvars = 1` configuration. The only difference with `--captures` is in the way the generated code stores submatch results: instead of `yynmatch` and `yypmatch` re2rust generates variables `yytl<k>` and `yytr<k>` for k-th capturing group (the user should declare these using an `svars` block). Captures with variables support two disambiguation policies: `--leftmost-captvars` or `re2c:leftmost-captvars = 1` for leftmost greedy policy (the default one) and `--posix-captvars` or `re2c:posix-captvars` for POSIX longest-match policy.

### 🧚 Under the hood 🧚

Under the hood all these options translate into tags and **Tagged Deterministic Finite Automata with Lookahead**. The core idea of TDFA is to minimize the overhead on submatch extraction. In the extreme, if there're no tags or captures in a regular expression, TDFA is just an ordinary DFA. If the number of tags is moderate, the overhead is barely noticeable.

The generated TDFA uses a number of tag variables which do not map directly to tags: a single variable may be used for different tags, and a tag may require multiple variables to hold all its possible values. Eventually ambiguity is resolved, and only one final variable per tag survives. Tag variables should be defined using `stags` or `mtags` blocks. If lexer state is stored, tag variables should be part of it. They also need to be updated by `YYFILL`.

> Like, no cap — TDFA is doing a *lot* of quiet work behind the scenes so your submatch extraction doesn't cost you your whole performance budget 💀

**S-tags support the following operations:**

- save input position to an s-tag: `t = YYCURSOR` with C pointer API or a user-defined operation `YYSTAGP(t)` with generic API
- save default value to an s-tag: `t = NULL` with C pointer API or a user-defined operation `YYSTAGN(t)` with generic API
- copy one s-tag to another: `t1 = t2`

**M-tags support the following operations:**

- append input position to an m-tag: a user-defined operation `YYMTAGP(t)` with both default and generic API
- append default value to an m-tag: a user-defined operation `YYMTAGN(t)` with both default and generic API
- copy one m-tag to another: `t1 = t2`

<purple>S-tags</purple> can literally just be scalar values, like pointers or offsets — easy peasy. But <purple>M-tags</purple>? Bestie, they need a way more complex setup because they gotta store a whole *sequence* of tag values. 😭 The most naive-and-inefficient way to represent an m-tag is just a list (array, vector) of tag values. A way more efficient glow-up is to store all the m-tags in a prefix-tree, represented as an array of nodes `(v, p)`, where `v` is the tag value and `p` is a pointer to the parent node. Slay. 🌳✨

---

Here's a super simple example of using s-tags to parse semantic versions made of three numeric components: major, minor, patch (the last one's optional, ngl). There's a more complex example below that uses `YYFILL`, so keep scrolling. 👀

// re2rust $INPUT -o $OUTPUT --api simple

#[derive(Debug, PartialEq)] struct SemVer(u32, u32, u32); // version: (major, minor, patch)

const NONE: usize = std::usize::MAX;

fn s2n(str: &[u8]) -> u32 { // convert a pre-parsed string to a number let mut n = 0; for i in str { n = n * 10 + *i as u32 - 48; } return n; }

fn parse(yyinput: &[u8]) -> Option<SemVer> { assert_eq!(yyinput.last(), Some(&0)); // expect null-terminated input

let (mut yycursor, mut yymarker) = (0, 0);

// Final tag variables available in semantic action. /*!svars:re2c format = '#[allow(unused_mut)]\nlet mut @@;\n'; */

// Intermediate tag variables used by the lexer (must be autogenerated). /*!stags:re2c format = 'let mut @@ = NONE;'; */

/*!re2c re2c:YYCTYPE = u8; re2c:yyfill:enable = 0; re2c:tags = 1;

num = [0-9]+;

@t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] { let major = s2n(&yyinput[t1..t2]); let minor = s2n(&yyinput[t3..t4]); let patch = if t5 != NONE {s2n(&yyinput[t5..yycursor - 1])} else {0}; return Some(SemVer(major, minor, patch)); }

• { return None; } */ }

fn main() { assert_eq!(parse(b"23.34\0"), Some(SemVer(23, 34, 0))); assert_eq!(parse(b"1.2.99999\0"), Some(SemVer(1, 2, 99999))); assert_eq!(parse(b"1.a\0"), None); }

---

Okay so here's a more complex one 🌀 — using s-tags *with* `YYFILL` to parse a file full of newline-separated semantic versions. Tag variables are part of the lexer state, and they get adjusted in `YYFILL` just like the other input positions. <pink>Note</pink> that this is *necessary* for s-tags, because their values are invalidated after you shift the buffer contents. It might not be necessary in a custom implementation where tag variables store offsets relative to the start of the input string instead of the buffer — which is lowkey the case with m-tags. 💅

// re2rust $INPUT -o $OUTPUT

use std::fs::File; use std::io::{Read, Write};

const BUFSIZE: usize = 4096; const NONE: usize = usize::MAX;

struct State { file: File, yyinput: [u8; BUFSIZE], yylimit: usize, yycursor: usize, yymarker: usize, token: usize, // Intermediate tag variables must be part of the lexer state passed to YYFILL. // They don't correspond to tags and should be autogenerated by re2c. /*!stags:re2c format = "@@: usize,\n"; */ eof: bool, }

#[derive(PartialEq)] enum Fill { Ok, Eof, LongLexeme }

#[derive(Debug, PartialEq)] struct SemVer(u32, u32, u32); // version: (major, minor, patch)

fn s2n(str: &[u8]) -> u32 { // convert a pre-parsed string to a number let mut n = 0; for i in str { n = n * 10 + *i as u32 - 48; } return n; }

macro_rules! shift { // ignore overflow, marker and tags may not be set yet ($x:expr, $y:expr) => { $x = $x.overflowing_sub($y).0 } }

fn fill(st: &mut State) -> Fill { if st.eof { return Fill::Eof; }

// Error: lexeme too long. In real life could reallocate a larger buffer. if st.token < 1 { return Fill::LongLexeme; }

// Shift buffer contents (discard everything up to the current token). st.yyinput.copy_within(st.token..st.yylimit, 0); st.yylimit -= st.token; st.yycursor -= st.token; shift!(st.yymarker, st.token); // Tag variables need to be shifted like other input positions. The check // for NONE is only needed if some tags are nested inside of alternative or // repetition, so that they can have NONE value. /*!stags:re2c format = "if st.@@ != NONE { shift!(st.@@, st.token); }\n"; */ st.token = 0;

// Fill free space at the end of buffer with new data from file. match st.file.read(&mut st.yyinput[st.yylimit..BUFSIZE - 1]) { Ok(n) => { st.yylimit += n; st.eof = n == 0; st.yyinput[st.yylimit] = 0; } Err(why) => panic!("cannot read from file: {}", why) }

return Fill::Ok; }

fn parse(st: &mut State) -> Option<Vec::<SemVer>> { let mut vers = Vec::new();

// Final tag variables available in semantic action. /*!svars:re2c format = 'let mut @@;\n'; */

'parse: loop { st.token = st.yycursor; /*!re2c re2c:api = record; re2c:eof = 0; re2c:tags = 1; re2c:yyrecord = st; re2c:YYCTYPE = u8; re2c:YYFILL = "fill(st) == Fill::Ok";

num = [0-9]+;

num @t1 "." @t2 num @t3 ("." @t4 num)? [\n] { let major = s2n(&st.yyinput[st.token..t1]); let minor = s2n(&st.yyinput[t2..t3]); let patch = if t4 != NONE {s2n(&st.yyinput[t4..st.yycursor - 1])} else {0}; vers.push(SemVer(major, minor, patch)); continue 'parse; } $ { return Some(vers); }

• { return None; } */ } }

fn main() { let fname = "input"; let verstr = b"1.22.333\n"; let expect = (0..BUFSIZE).map(|_| SemVer(1, 22, 333)).collect();

// Prepare input file (make sure it exceeds buffer size). match File::create(fname) { Err(why) => panic!("cannot open {}: {}", fname, why), Ok(mut file) => match file.write_all(&verstr.repeat(BUFSIZE)) { Err(why) => panic!("cannot write to {}: {}", fname, why), Ok(_) => {} } };

// Reopen input file for reading. let file = match File::open(fname) { Err(why) => panic!("cannot read file {}: {}", fname, why), Ok(file) => file, };

// Initialize lexer state. let yylimit = BUFSIZE - 1; let mut st = State { file: file, yyinput: [0; BUFSIZE], // sentinel is set to zero, which triggers YYFILL yylimit: yylimit, yycursor: yylimit, yymarker: yylimit, token: yylimit, /*!stags:re2c format = "@@: NONE,\n"; */ eof: false, };

// Run the lexer and check results. assert_eq!(parse(&mut st), Some(expect));

// Cleanup: remove input file. match std::fs::remove_file(fname) { Err(why) => panic!("cannot remove {}: {}", fname, why), Ok(_) => {} } }

---

Next vibe: using <mint>capturing groups</mint> to parse semantic versions 🎀 (yes, that's a whole different flex).

// re2rust $INPUT -o $OUTPUT --api simple

#[derive(Debug, PartialEq)] struct SemVer(u32, u32, u32); // version: (major, minor, patch)

const NONE: usize = std::usize::MAX;

fn s2n(str: &[u8]) -> u32 { // convert a pre-parsed string to a number let mut n = 0; for i in str { n = n * 10 + *i as u32 - 48; } return n; }

fn parse(yyinput: &[u8]) -> Option<SemVer> { assert_eq!(yyinput.last(), Some(&0)); // expect null-terminated input

let (mut yycursor, mut yymarker) = (0, 0);

// Final tag variables available in semantic action. /*!stags:re2c format = 'let mut @@ = NONE;'; */

// Intermediate tag variables used by the lexer (must be autogenerated). /*!svars:re2c format = '#[allow(unused_mut)]\nlet mut @@;\n'; */

/*!re2c re2c:YYCTYPE = u8; re2c:yyfill:enable = 0; re2c:captvars = 1;

num = [0-9]+;

(num) "." (num) ("." num)? [\x00] { assert!(yytl0 == 0 && yytr0 == yyinput.len()); let major = s2n(&yyinput[yytl1..yytr1]); let minor = s2n(&yyinput[yytl2..yytr2]); let patch = if yytl3 == NONE {0} else {s2n(&yyinput[yytl3 + 1..yytr3])}; return Some(SemVer(major, minor, patch)); }

• { return None; } */ }

fn main() { assert_eq!(parse(b"23.34\0"), Some(SemVer(23, 34, 0))); assert_eq!(parse(b"1.2.99999\0"), Some(SemVer(1, 2, 99999))); assert_eq!(parse(b"1.a\0"), None); }

---

And finally ✨ — using <hotpink>m-tags</hotpink> to parse a version with a *variable* number of components. The tag variables live in a trie, bestie. 💀🌲

// re2rust $INPUT -o $OUTPUT --api simple

const NONE: usize = std::usize::MAX; const MTAG_ROOT: usize = NONE - 1;

// An m-tag tree is a way to store histories with an O(1) copy operation. // Histories naturally form a tree, as they have common start and fork at some // point. The tree is stored as an array of pairs (tag value, link to parent). // An m-tag is represented with a single link in the tree (array index). type MtagTrie = Vec::<MtagElem>; struct MtagElem { elem: usize, // tag value pred: usize, // index of the predecessor node or root }

// Append a single value to an m-tag history. fn add_mtag(trie: &mut MtagTrie, mtag: usize, value: usize) -> usize { trie.push(MtagElem{elem: value, pred: mtag}); return trie.len() - 1; }

// Recursively unwind tag histories and collect version components. fn unwind(trie: &MtagTrie, x: usize, y: usize, str: &[u8], ver: &mut Ver) { // Reached the root of the m-tag tree, stop recursion. if x == MTAG_ROOT && y == MTAG_ROOT { return; }

// Unwind history further. unwind(trie, trie[x].pred, trie[y].pred, str, ver);

// Get tag values. Tag histories must have equal length. assert!(x != MTAG_ROOT && y != MTAG_ROOT); let (ex, ey) = (trie[x].elem, trie[y].elem);

if ex != NONE && ey != NONE { // Both tags are valid string indices, extract component. ver.push(s2n(&str[ex..ey])); } else { // Both tags are NONE (this corresponds to zero repetitions). assert!(ex == NONE && ey == NONE); } }

type Ver = Vec::<u32>; // unbounded number of version components

fn s2n(str: &[u8]) -> u32 { // convert a pre-parsed string to a number let mut n = 0; for i in str { n = n * 10 + *i as u32 - 48; } return n; }

fn parse(yyinput: &[u8]) -> Option<Ver> { assert_eq!(yyinput.last(), Some(&0)); // expect null-terminated input

let (mut yycursor, mut yymarker) = (0, 0); let mut mt: MtagTrie = Vec::new();

// Final tag variables available in semantic action. /!svars:re2c format = 'let @@;\n'; */ /!mvars:re2c format = 'let @@;\n'; */

// Intermediate tag variables used by the lexer (must be autogenerated). /!stags:re2c format = 'let mut @@ = NONE;'; */ /!mtags:re2c format = 'let mut @@ = MTAG_ROOT;'; */

/*!re2c re2c:YYCTYPE = u8; re2c:YYMTAGP = "@@ = add_mtag(&mut mt, @@, yycursor);"; re2c:YYMTAGN = "@@ = add_mtag(&mut mt, @@, NONE);"; re2c:yyfill:enable = 0; re2c:tags = 1;

num = [0-9]+;

@t1 num @t2 ("." #t3 num #t4)* [\x00] { let mut ver: Ver = Vec::new(); ver.push(s2n(&yyinput[t1..t2])); unwind(&mt, t3, t4, yyinput, &mut ver); return Some(ver); }

• { return None; } */ }

fn main() { assert_eq!(parse(b"1\0"), Some(vec![1])); assert_eq!(parse(b"1.2.3.4.5.6.7\0"), Some(vec![1, 2, 3, 4, 5, 6, 7])); assert_eq!(parse(b"1.2.\0"), None); }

---

## 🌈 Encoding support 🌈

okay bestie, this is the part where we get a lil philosophical about what a "character" even *is* 💭. You gotta clock the difference between <pink>code points</pink> and <pink>code units</pink>:

- A <purple>code point</purple> is a numeric identifier of a symbol. It's the *idea* of the character 💡.
- A <purple>code unit</purple> is the smallest unit of storage in the encoded text. It's the physical lil box the idea lives in 📦.

A single code point may be represented with one or more code units. In a <green>fixed-length encoding</green> all code points are represented with the same number of code units. In a <green>variable-length encoding</green> code points may be represented with a different number of code units. The way that some code points take up more room than others?? so real of them 🫶

> Heads up: the `"any"` rule `[^]` matches any *code point*, but not necessarily any *code unit* (the only way to match any code unit regardless of the encoding is the default rule `*`). Don't mix those up, iykyk 💅

So the generated lexer works with a stream of <cyan>code units</cyan>: `yych` stores a code unit, and `YYCTYPE` is the code unit type. Regular expressions, on the other hand, are specified in terms of <cyan>code points</cyan>. When `re2rust` compiles regular expressions to automata it translates code points to code units. And ngl this is generally not a simple mapping: in variable-length encodings a single code point range may get translated to a complex code unit graph 😭 (it ate, but it's a lot).

The following encodings are supported:

- <pink>**ASCII**</pink> — enabled by default ✨ It is a fixed-length encoding with code space `[0-255]` and 1-byte code points and code units.
- <pink>**EBCDIC**</pink> — enabled with `--ebcdic` or `re2c:encoding:ebcdic`. It is a fixed-length encoding with code space `[0-255]` and 1-byte code points and code units.
- <pink>**UCS2**</pink> — enabled with `--ucs2` or `re2c:encoding:ucs2`. It is a fixed-length encoding with code space `[0-0xFFFF]` and 2-byte code points and code units.
- <pink>**UTF8**</pink> — enabled with `--utf8` or `re2c:encoding:utf8`. It is a variable-length Unicode encoding. Code unit size is 1 byte. Code points are represented with 1 -- 4 code units.
- <pink>**UTF16**</pink> — enabled with `--utf16` or `re2c:encoding:utf16`. It is a variable-length Unicode encoding. Code unit size is 2 bytes. Code points are represented with 1 -- 2 code units.
- <pink>**UTF32**</pink> — enabled with `--utf32` or `re2c:encoding:utf32`. It is a fixed-length Unicode encoding with code space `[0-0x10FFFF]` and 4-byte code points and code units.

Include files `include/unicode_blocks.re`, `include/unicode_categories.re` and `include/unicode_properties.re` provide `re2rust` definitions for blocks, categories and properties described in the Unicode standard. Bestie we literally ship the Unicode lore for free 🎀

Option `--input-encoding` specifies source file encoding, which can be used to enable Unicode literals in regular expressions. For example `--input-encoding utf8` tells `re2rust` that the source file is in UTF8 (it differs from `--utf8` which sets input text encoding). Yeah — those are two different things, don't @ me 🙈. Option `--encoding-policy` specifies the way `re2rust` handles Unicode surrogates (code points in range `[0xD800-0xDFFF]`).

Below is an example of a lexer for UTF8 encoded Unicode identifiers. Iconic behavior honestly 👑

// re2rust $INPUT -o $OUTPUT --utf8 --api simple

/*!include:re2c "unicode_categories.re" */

fn lex(yyinput: &[u8]) -> bool { assert_eq!(yyinput.last(), Some(&0)); // expect null-terminated input

let (mut yycursor, mut yymarker) = (0, 0); /*!re2c re2c:YYCTYPE = u8; re2c:yyfill:enable = 0;

// Simplified "Unicode Identifier and Pattern Syntax" // (see https://unicode.org/reports/tr31) id_start = L | Nl | [$_]; id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3]; identifier = id_start id_continue*;

identifier { return true; }

• { return false; } */ }

fn main() { assert!(lex("_Ыдентификатор\0".as_bytes())); }

---

## 🎀 Include files 🎀

`re2rust` allows one to include other files using a block of the form `/*!include:re2c FILE */` or `%{include FILE %}`, or an in-block directive `!include FILE ;`, where `FILE` is a path to the file to be included. Three whole ways to do the same thing, because variety is the spice of life 💅

`re2rust` looks for include files in the directory of the including file and in include locations, which can be specified with the `-I` option. Include blocks/directives in `re2rust` work in the same way as C/C++ `#include`: `FILE` contents are copy-pasted verbatim in place of the block/directive. Include files may have further includes of their own — yes, inception vibes 🌀. Use `--depfile` option to track build dependencies of the output file on include files.

`re2rust` provides some predefined include files that can be found in the `include/` subdirectory of the project. These files contain definitions that may be useful to other projects (such as Unicode categories) and form something like a standard library for `re2rust`. Basically we have lore drops 🌸. Below is an example of using include files.

### 💜 Include file 1 (definitions.rs)

#[derive(Debug, PartialEq)] enum Num { Int, Float, NaN }

/!re2c number = [1-9][0-9]; */

### 💜 Include file 2 (extra_rules.re.inc)

// floating-point numbers frac = [0-9]* "." [0-9]+ | [0-9]+ "."; exp = 'e' [+-]? [0-9]+; float = frac exp? | [0-9]+ exp;

float { return Num::Float; }

### 💜 Input file

// re2rust $INPUT -o $OUTPUT --api simple

/*!include:re2c "definitions.rs" */

fn lex(yyinput: &[u8]) -> Num { assert_eq!(yyinput.last(), Some(&0)); // expect null-terminated input

let mut yycursor = 0; let mut yymarker = 0; /*!re2c re2c:yyfill:enable = 0; re2c:YYCTYPE = u8;

• { return Num::NaN; } number { return Num::Int; } !include "extra_rules.re.inc"; */ }

fn main() { assert_eq!(lex(b"123\0"), Num::Int); assert_eq!(lex(b"123.4567\0"), Num::Float); }

---

## 🦋 Header files 🦋

`re2rust` allows one to generate header file from the input `.re` file using `--header` option or `re2c:header` configuration and block pairs of the form `/*!header:re2c:on*/` and `/*!header:re2c:off*/`, or `%{header:on%}` and `%{header:off%}`. The first block marks the beginning of header file, and the second block marks the end of it. Everything between these blocks is processed by `re2rust`, and the generated code is written to the file specified with `--header` option or `re2c:header` configuration (or stdout if neither option nor configuration is used). Autogenerated header file may be needed in cases when `re2rust` is used to generate definitions that must be visible from other translation units. Sharing is caring, bestie 🫶

Here is an example of generating a header file that contains definition of the lexer state with tag variables (the number variables depends on the regular grammar and is unknown to the programmer). Yeah — the count is a mystery even to the person writing it, which is lowkey hilarious 😭

### ⭐ Input file

// re2rust $INPUT -o $OUTPUT --header lexer/state.rs

mod lexer; use lexer::state::State; // the module is generated by re2c

/!header:re2c:on/ pub struct State<'a> { pub yyinput: &'a [u8], pub yycursor: usize, /!stags:re2c format = "pub @@: usize,"; */ } /!header:re2c:off*/

fn lex(yyrecord: &mut State) -> usize { assert_eq!(yyrecord.yyinput.last(), Some(&0)); // expect null-terminated input

let t: usize; /*!re2c re2c:header = "lexer/state.rs"; re2c:yyfill:enable = 0; re2c:api = record; re2c:YYCTYPE = "u8"; re2c:tags = 1;

[a]* @t [b]* { return t; } */ }

fn main() { let mut st = State { yyinput: b"ab\0", yycursor: 0, /*!stags:re2c format = "@@: 0,"; */ }; assert_eq!(lex(&mut st), 1); }

### ⭐ Header file

/* Generated by re2c */

pub struct State<'a> { pub yyinput: &'a [u8], pub yycursor: usize, pub yyt1: usize, }

---

## 🔥 Skeleton programs 🔥

With the `-S`, `--skeleton` option, `re2rust` ignores all non-`re2rust` code and generates a self-contained C program that can be further compiled and executed. Yes, C — we said what we said 💀

The program consists of lexer code and input data. For each constructed DFA (block or condition) `re2rust` generates a standalone lexer and two files: an `.input` file with strings derived from the DFA and a `.keys` file with expected match results. The program runs each lexer on the corresponding `.input` file and compares results with the expectations. Skeleton programs are very useful for a number of reasons:

- They can check correctness of various `re2rust` optimizations (the data is generated early in the process, before any DFA transformations have taken place).
- Generating a set of input data with good coverage may be useful for both testing and benchmarking.
- Generating self-contained executable programs allows one to get minimized test cases (the original code may be large or have a lot of dependencies).

Three whole reasons, and every single one of them is a slay ✨🧚

The difficulty with generating input data is that, for all but the most trivial cases, the number of possible input strings is too large (even if the string length is limited). <pink>re2rust</pink> solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. Here's the algorithm it uses, bestie, stay with me 💖

1. First, it constructs a **skeleton** of the DFA.
2. For encodings with **1-byte code unit size** (such as `ASCII`, `UTF-8` and `EBCDIC`), the skeleton is just an exact copy of the original DFA.
3. For encodings with **multibyte code units**, the skeleton is a copy of the DFA with certain transitions omitted: namely, <pink>re2rust</pink> takes at most `256` code units for each disjoint continuous range that corresponds to a DFA transition. The chosen values are evenly distributed and include range bounds.
4. Instead of trying to cover *all possible paths* in the skeleton (which is infeasible), <pink>re2rust</pink> generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer.

The algorithm implementation is limited by ~`1Gb` of transitions and consumes a constant amount of memory (<pink>re2rust</pink> writes data to file as soon as it is generated). <cyan>Constant memory?? We love an efficient queen 💅</cyan>

## 🎀 Visualization and debug 🎀

With the `-D`, `--emit-dot` option, <pink>re2rust</pink> does **not** generate code. Instead, it dumps the generated DFA in DOT format. You can convert this dump to an image of the DFA using <purple>Graphviz</purple> or another library. <rainbow>Obsessed with a good diagram 🦋</rainbow>

> Heads up bestie — this option shows the **final** DFA after it has gone through a number of optimizations and transformations. Earlier stages can be dumped with various debug options, such as `--dump-nfa`, `--dump-dfa-raw` etc. (see the full list of options). 🙈

---

# ✨ SEE ALSO ✨

You can find more information about <pink>re2c</pink> at the official website: http://re2c.org. Similar programs are `flex(1)`, `lex(1)`, `quex` (http://quex.sourceforge.net). <green>iykyk 👑</green>

---

# 🌸 AUTHORS 🌸

<pink>re2rust</pink> was originally written by **Peter Bumbulis** (peter@csg.uwaterloo.ca) in `1993`. <yellow>1993!! a literal pioneer 🫶</yellow>

**Marcus Boerger** and **Dan Nuffer** spent several years to turn the original idea into a production ready code generator. Since then it has been maintained and developed by multiple volunteers, most notably:

- **Brian Young** (bayoung@acm.org)
- **Marcus Boerger**
- **Dan Nuffer** (nuffer@users.sourceforge.net)
- **Ulya Trofimovich** (skvadrik@gmail.com)
- **Serghei Iakovlev**
- **Sergei Trofimovich**
- **Petr Skocik**
- **ligfx raekye**
- **PolarGoose**

<hotpink>Every single one of them ate, no cap 💀✨</hotpink>

---

> And that's the page, bestie! 🎀 You made it all the way to the footer — which is literally just `RE2RUST(1)` chillin' down there, no date, no version, nothing else. <purple>Manpage said "main character energy only" and left it at that 💅</purple> Go write some regexes, you icon. 💖