RE2JAVA(1)

📦 version 4.6, in arch rolling

💅 NAME 💅

okay bestie, welcome to RE2JAVA(1) — the manual page for re2java, which is literally here to generate fast lexical analyzers for Java. That's the whole name section, cute and done. ✨

✨ SYNOPSIS ✨

re2java [ OPTIONS ] [ WARNINGS ] INPUT

INPUT can be either a file or - for stdin. 🫶

🌸 INTRODUCTION 🌸

okay so re2java works as a preprocessor. It reads the input file (which is usually a program in Java, but can be anything) and looks for blocks of code enclosed in special-form start/end markers. The text outside of these blocks is copied verbatim into the output file. The contents of the blocks are processed by re2java. It translates them to code in Java and outputs the generated code in place of the block. 💅

Here is an example of a small program that checks if a given string contains a decimal number:

// re2java $INPUT -o $OUTPUT

class Main {
    static boolean lex(String yyinput) {
        int yycursor = 0;

        /*!re2c
            re2c:YYCTYPE = "char";
            re2c:YYPEEK = "yyinput.charAt(yycursor)";
            re2c:yyfill:enable = 0;

            [1-9][0-9]* { return true; }
            *           { return false; }
        */
    }

    public static void main(String []args) {
        assert lex("1234\0");
    }
};

In the output re2java replaced the block in the middle with the generated code:

// Generated by re2java
// re2java $INPUT -o $OUTPUT

class Main {
    static boolean lex(String yyinput) {
        int yycursor = 0;

{
    char yych = 0;
    int yystate = 0;
    yyl: while (true) {
        switch (yystate) {
            case 0:
                yych = yyinput.charAt(yycursor);
                yycursor += 1;
                switch (yych) {
                    case 0x31:
                    case 0x32:
                    case 0x33:
                    case 0x34:
                    case 0x35:
                    case 0x36:
                    case 0x37:
                    case 0x38:
                    case 0x39:
                        yystate = 2;
                        continue yyl;
                    default:
                        yystate = 1;
                        continue yyl;
                }
            case 1:
                { return false; }
            case 2:
                yych = yyinput.charAt(yycursor);
                switch (yych) {
                    case 0x30:
                    case 0x31:
                    case 0x32:
                    case 0x33:
                    case 0x34:
                    case 0x35:
                    case 0x36:
                    case 0x37:
                    case 0x38:
                    case 0x39:
                        yycursor += 1;
                        yystate = 2;
                        continue yyl;
                    default:
                        yystate = 3;
                        continue yyl;
                }
            case 3:
                { return true; }
            default:
                throw new IllegalStateException("internal lexer error");
        }
    }
}

    }

    public static void main(String []args) {
        assert lex("1234\0");
    }
};

🦋 BASICS 🦋

A re2java program consists of a sequence of blocks intermixed with code in the target language. A block may contain definitions, configurations, rules, actions and directives in any order:

name = regular-expression ; — a definition binds name to regular-expression. Names may contain alphanumeric characters and underscore. The regular expressions section gives an overview of re2java syntax for regular expressions. Once defined, the name can be used in other regular expressions and in rules. Recursion in named definitions is not allowed, and each name should be defined before it is used. A block inherits named definitions from the global scope. Redefining a name that exists in the current scope is an error. 😭
configuration = value ; — a configuration allows one to change re2java behavior and customize the generated code. For a full list of configurations supported by re2java see the configurations section. Depending on a particular configuration, the value can be a keyword, a nonnegative integer number or a one-line string which should be enclosed in double or single quotes unless it consists of alphanumeric characters. A block inherits configurations from the global scope and may redefine them or add new ones. Configurations defined inside of a block affect the whole block, even if they appear at the end of it. periodt. 💅
regular-expression code — a rule binds regular-expression to its semantic action (a block of code in curly braces, or a block of code that starts with := and ends on a newline followed by any non-whitespace character). If the regular-expression matches, the associated code is executed. If multiple rules match, the longest match takes precedence. If multiple rules match the same string, the earliest one takes precedence. There are two special rules: the default rule * and the end of input rule $. Default rule should always be defined, it has the lowest priority regardless of its place in the block, and it matches any code unit (not necessarily a valid character, see the encoding support section). The end of input rule should be defined if the corresponding method for handling the end of input is used. With start conditions rules have more complex syntax. 🌀
!action code — an action binds a user-defined block of code to a particular place in the generated finite state machine (in the same way as semantic actions bind code to the final states). See the actions section for a full list of predefined actions. ✨
!directive ; — a directive is one of the special predefined statements. Each directive has a unique purpose. See the directives section for details. 🎀

🎀 Blocks 🎀

Block start and end markers are either /*!re2c and */, or %{ and %} (both styles are supported). Starting from version 2.2 blocks may have optional names that allow them to be referenced in other blocks. There are different kinds of blocks:

/*!re2c[:<name>] ... */ or %{[:<name>] ... %} — a global block contains definitions, configurations, rules and directives. re2java compiles regular expressions associated with each rule into a deterministic finite automaton, encodes it in the form of conditional jumps in the target language and replaces the block with the generated code. Names and configurations defined in a global block are added to the global scope and become visible to subsequent blocks. At the start of the program the global scope is initialized with command-line options. 👑
/*!local:re2c[:<name>] ... */ or %{local[:<name>] ... %} — a local block is like a global block, but the names and configurations in it have local scope (they do not affect other blocks). 🧚
/*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} — a rules block is like a local block, but it does not generate any code by itself, nor does it add any definitions to the global scope — it is meant to be reused in other blocks. This is a way of sharing code (more details in the reusable blocks section). Prior to re2java version 2.2 rules blocks required -r --reusable option. 🍓
/*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %} — a use block that references a previously defined rules block. If the name is specified, re2java looks for a rules blocks with this name. Otherwise the most recent rules block is used (either a named or an unnamed one). A use block can add definitions, configurations and rules of its own, which are added to those of the referenced rules block. Prior to re2java version 2.2 use blocks required -r --reusable option. 🔥
/*!max:re2c[:<name1>[:<name2>...]] ... */ or %{max[:<name1>[:<name2>...]] ... %} — a block that generates YYMAXFILL definition. An optional list of block names specifies which blocks should be included when computing YYMAXFILL value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXFILL <n>), or a global variable for Go (var YYMAXFILL int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXFILL. 🌈
/*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */ or %{maxnmatch[:<name1>[:<name2>...]] ... %} — a block that generates YYMAXNMATCH definition (it requires -P --posix-captures option). An optional list of block names specifies which blocks should be included when computing YYMAXNMATCH value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXNMATCH <n>), or a global variable for Go (var YYMAXNMATCH int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXNMATCH. 💀
/*!stags:re2c[:<name1>[:<name2>...]] ... */, /*!mtags:re2c[:<name1>[:<name2>...]] ... */ or %{stags[:<name1>[:<name2>...]] ... %}, %{mtags[:<name1>[:<name2>...]] ... %{ — blocks that specify a template piece of code that is expanded for each s-tag/m-tag variable generated by re2java. An optional list of block names specifies which blocks should be included when computing the set of tag variables (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag variable. Configuration separator specifies a piece of code used to join the generated format pieces for different tag variables. 🦋

...and we're back at the block list bestie, because there are more block types and every single one of them is eating 💅✨

/*!svars:re2c[:<name1>[:<name2>...]] ... */, /*!mvars:re2c[:<name1>[:<name2>...]] ... */, %{svars[:<name1>[:<name2>...]] ... %}, %{mvars[:<name1>[:<name2>...]] ... %{ — 🎀 Blocks that specify a template piece of code that gets expanded for each s-tag/m-tag that is either explicitly mentioned by the rules (with the --tags option) or implicitly generated by re2java (with the --captvars or --posix-captvars options). An optional list of block names tells you which blocks should be included when computing the set of tags (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag. Configuration separator specifies a piece of code used to join the generated format pieces for different tags. 💖
/*!getstate:re2c[:<name1>[:<name2>...]] ... */, %{getstate[:<name1>[:<name2>...]] ... %} — a block that generates conditional dispatch on the lexer state (heads up: it requires the --storable-state option, no cap). An optional list of block names specifies which blocks should be included in the state dispatch. The default transition goes to the start label of the first block on the list. If the list is empty, all blocks are included, and the default transition goes to the first block in the file that has a start label. 🦋 This block type is incompatible with the --loop-switch option, as it requires cross-block transitions that are unsupported without goto or function calls — so like, don't try it, it's giving undefined chaos.
/*!conditions:re2c[:<name1>[:<name2>...]] ... */, /*!types:re2c... */, %{conditions[:<name1>[:<name2>...]] ... %}, %{types... %} — a block that generates condition enumeration (it requires the --conditions option). An optional list of block names specifies which blocks should be included when computing the set of conditions (if the list is empty, all blocks are included). By default the generated code is an enumeration YYCONDTYPE. It can be customized with optional configurations format and separator. Configuration format specifies a template string where @@{cond} (or @@ for short) is replaced with the name of each condition, and @@{num} is replaced with a numeric index of that condition. Configuration separator specifies a piece of code used to join the generated format pieces for different conditions. ✨
/*!include:re2c <file> */, %{include <file> %} — this block lets you include <file>, which must be a double-quoted file path. The contents of the file are literally substituted in place of the block, in the same way as #include works in C/C++. 🌸 This block can be used together with the --depfile option to generate build system dependencies on the included files. iykyk.
/*!header:re2c:on*/, %{header:on %} — this block marks the start of the header file. Everything after it and up to the following header:off block is processed by re2java and written to the header file specified with the -t --type-header option. 👑
/*!header:re2c:off*/, %{header:off %} — this block marks the end of the header file started with the header:on*/ block. 🎀
/*!ignore:re2c ... */, %{ignore ... %} — a block whose contents are ignored and removed from the output file. literally "not me, not my problem" energy 💀

✨ Configurations ✨

Here's the full list of configurations re2java supports, bestie — buckle up, it's a long one 🫶

re2c:api, re2c:input — same as the --api option.
re2c:api:sigil — specifies the marker ("sigil") that is used for argument placeholders in the API primitives. The default is @@. A placeholder starts with sigil followed by the argument name in curly braces. For example, if sigil is set to $, then placeholders will have the form ${name}. Single-argument APIs may use shorthand notation without the name in braces. 🌈 This option can be overridden by options for individual API primitives, e.g. re2c:YYFILL@len for YYFILL.
re2c:api:style — specifies API style. Possible values are functions (the default for C) and free-form (the default for Go and Rust). In functions style, API primitives are generated with an argument list in parentheses following the name of the primitive. The arguments are provided only for autogenerated parameters (such as the number of characters passed to YYFILL), but not for the general lexer context, so the primitives behave more like macros in C/C++ or closures in Go and Rust. In free-form style, API primitives do not have a fixed form: they should be defined as strings containing free-form pieces of code with interpolated variables of the form @@{var} or @@ (they correspond to arguments in function-like style). 💅 This configuration may be overridden for individual API primitives, see for example the re2c:YYFILL:naked configuration for YYFILL.
re2c:bit-vectors, re2c:flags:bit-vectors, re2c:flags:b — same as the --bit-vectors option, but can be configured on a per-block basis.
re2c:captures, re2c:leftmost-captures — same as the --leftmost-captures option, but can be configured on a per-block basis.
re2c:captvars, re2c:leftmost-captvars — same as the --leftmost-captvars option, but can be configured on a per-block basis.
re2c:case-insensitive, re2c:flags:case-insensitive — same as the --case-insensitive option, but can be configured on a per-block basis.
re2c:case-inverted, re2c:flags:case-inverted — same as the --case-inverted option, but can be configured on a per-block basis.
re2c:case-ranges, re2c:flags:case-ranges — same as the --case-ranges option, but can be configured on a per-block basis.
re2c:computed-gotos, re2c:flags:computed-gotos, re2c:flags:g — same as the --computed-gotos option, but can be configured on a per-block basis.
re2c:computed-gotos:relative, re2c:cgoto:relative — same as the --computed-gotos-relative option, but can be configured on a per-block basis.
re2c:computed-gotos:threshold, re2c:cgoto:threshold — if computed goto is used, this configuration specifies the complexity threshold that triggers the generation of jump tables instead of nested if statements and bitmaps. The default value is 9. 🔥
re2c:cond:abort — if set to a positive integer value, the default case in the generated condition dispatch aborts program execution. 😭 (like, actually terminates, so know what you're doing)
re2c:cond:goto — specifies a piece of code used for the autogenerated shortcut rules :=> in conditions. The default is goto @@;. The @@ placeholder is substituted with the condition name (see configurations re2c:api:sigil and re2c:cond:goto@cond).
re2c:cond:goto@cond — specifies the sigil used for argument substitution in the re2c:cond:goto definition. The default value is @@. Overrides the more generic re2c:api:sigil configuration.
re2c:cond:divider — defines the divider for condition blocks. The default value is /* *********************************** */. Placeholders are substituted with the condition name (see re2c:api;sigil and re2c:cond:divider@cond). 💖
re2c:cond:divider@cond — specifies the sigil used for argument substitution in the re2c:cond:divider definition. The default is @@. Overrides the more generic re2c:api:sigil configuration.
re2c:cond:prefix, re2c:condprefix — specifies the prefix used for condition labels. The default is yyc_.
re2c:cond:enumprefix, re2c:condenumprefix — specifies the prefix used for condition identifiers. The default is yyc.
re2c:debug-output, re2c:flags:debug-output, re2c:flags:d — same as the --debug-output option, but can be configured on a per-block basis.
re2c:empty-class, re2c:flags:empty-class — same as the --empty-class option, but can be configured on a per-block basis.
re2c:encoding:ebcdic, re2c:flags:ecb, re2c:flags:e — same as the --ebcdic option, but can be configured on a per-block basis.
re2c:encoding:ucs2, re2c:flags:wide-chars, re2c:flags:w — same as the --ucs2 option, but can be configured on a per-block basis.
re2c:encoding:utf8, re2c:flags:utf-8, re2c:flags:8 — same as the --utf8 option, but can be configured on a per-block basis.
re2c:encoding:utf16, re2c:flags:utf-16, re2c:flags:x — same as the --utf16 option, but can be configured on a per-block basis.
re2c:encoding:utf32, re2c:flags:unicode, re2c:flags:u — same as the --utf32 option, but can be configured on a per-block basis.
re2c:encoding-policy, re2c:flags:encoding-policy — same as the --encoding-policy option, but can be configured on a per-block basis.
re2c:eof — specifies the sentinel symbol used with the end-of-input rule $. The default value is -1 (the $ rule is not used). Other possible values include all valid code units. Only decimal numbers are recognized. 🍓
re2c:header, re2c:flags:type-header, re2c:flags:t — specifies the name of the generated header file relative to the directory of the output file. Same as the --header option except that the file path is relative.
re2c:indent:string — specifies the string used for indentation. The default is a single tab character "\t". The indent string should contain whitespace characters only. To disable indentation entirely, set this configuration to an empty string. 🧚
re2c:indent:top — specifies the minimum amount of indentation to use. The default value is 0. The value should be a non-negative integer number.
re2c:invert-captures — same as the --invert-captures option, but can be configured on a per-block basis.
re2c:label:prefix, re2c:labelprefix — specifies the prefix used for DFA state labels. The default is yy.
re2c:label:start, re2c:startlabel — controls the generation of a block start label. The default value is 0, which means that the start label is generated only if it is used. An integer value greater than zero forces the generation of the start label even if it is unused by the lexer. A string value also forces start label generation and sets the label name to the specified string. 🌟 This configuration applies only to the current block (it is reset to default for the next block) — so don't expect it to stick around, bestie, it's a one-block wonder 💅
re2c:label:yyFillLabel — sets the prefix for YYFILL labels, the ones you get with re2c:eof and in storable state mode. it's giving "name your children" energy 🎀
re2c:label:yyloop — the name of the label that marks the start of the lexer loop when you're using --loop-switch. default is literally yyloop. iconic default ngl 💅
re2c:label:yyNext — the name of that optional label sitting right after the YYGETSTATE switch in storable state mode (you turn it on with re2c:state:nextlabel). default is yyNext ✨
re2c:lookahead, re2c:flags:lookahead — ⚠️ deprecated, bestie. see the also-deprecated --no-lookahead option. we don't stan her anymore 💀
re2c:monadic — if you set this to non-zero, the generated lexer uses monadic notation. this one's specifically a Haskell thing, so if that's not your language, it's not your business 🦋
re2c:nested-ifs, re2c:flags:nested-ifs, re2c:flags:s — same as the --nested-ifs option, except you can configure it on a per-block basis. flexible queen 👑
re2c:posix-captures, re2c:flags:posix-captures, re2c:flags:P — same as the --posix-captures option, but per-block. same deal, three whole names for it 🎀
re2c:posix-captvars — same as the --posix-captvars option, also configurable per-block ✨
re2c:tags, re2c:flags:tags, re2c:flags:T — same as the --tags option, per-block configurable 🏷️
re2c:tags:expression — specifies the expression used for tag variables. by default re2java generates expressions of the form yyt<N>. that can be inconvenient — say, if your tag variables are defined as fields in a struct. all occurrences of @@{tag} or @@ get replaced with the actual tag name. for example, re2c:tags:expression = "s.@@"; gives you expressions of the form s.yyt<N> in the generated code. see also the re2c:api:sigil configuration 🪄
re2c:tags:negative — the constant expression used for a negative tag value. typically this would be -1 if tags are integer offsets in the input string, or a null pointer if they're pointers 🎯
re2c:tags:prefix — the prefix for tag variable names. default is yyt ✨
re2c:sentinel — the sentinel symbol used for end-of-input checks (this kicks in when bounds checks are disabled with re2c:yyfill:enable = 0; and re2c:eof is not set). real talk: this config does not affect code generation — its whole job is to verify that the sentinel isn't allowed in the middle of a rule, and to make sure the lexer won't read past the end of the buffer. default value is -1 (and in that case re2java assumes the sentinel is zero, which is the most common case). only decimal numbers are recognized, no hex shenanigans 🙈
re2c:state:abort — if you set it to a positive integer value, the default case in the generated state dispatch aborts program execution, and an explicit -1 case contains the transition to the start of the block. dramatic but valid 💀
re2c:state:nextlabel — controls whether the YYGETSTATE switch is followed by a yyNext label (default value is zero, which means no label). alternatively you can use re2c:label:start to generate a specific start label, or an explicit getstate block to generate the YYGETSTATE switch separately from the lexer block ✨
re2c:unsafe, re2c:flags:unsafe — same as the --no-unsafe option, but configurable per-block. if set to zero, it suppresses the generation of unsafe wrappers around YYPEEK. default is non-zero, meaning wrappers are generated. this config is specific to Rust 🦀
re2c:YYBACKUP, re2c:define:YYBACKUP — defines the generic API primitive YYBACKUP 🧚
re2c:YYBACKUPCTX, re2c:define:YYBACKUPCTX — defines the generic API primitive YYBACKUPCTX ✨
re2c:YYCONDTYPE, re2c:define:YYCONDTYPE — defines the API primitive YYCONDTYPE 🎀
re2c:YYCTYPE, re2c:define:YYCTYPE — defines the API primitive YYCTYPE 💜
re2c:YYCTXMARKER, re2c:define:YYCTXMARKER — defines the API primitive YYCTXMARKER 🦋
re2c:YYCURSOR, re2c:define:YYCURSOR — defines the API primitive YYCURSOR 👑
re2c:YYDEBUG, re2c:define:YYDEBUG — defines the API primitive YYDEBUG 🔍
re2c:YYFILL, re2c:define:YYFILL — defines the API primitive YYFILL 🌸
re2c:YYFILL@len, re2c:define:YYFILL@len — the sigil used for argument substitution in the YYFILL definition. defaults to @@. overrides the more generic re2c:api:sigil configuration 🪄
re2c:YYFILL:naked, re2c:define:YYFILL:naked — overrides the more generic re2c:api:style configuration for YYFILL. a zero value corresponds to free-form API style ✨
re2c:YYFN — defines the API primitive YYFN 🎀
re2c:YYINPUT — defines the API primitive YYINPUT 🍓
re2c:YYGETCOND, re2c:define:YYGETCONDITION — defines the API primitive YYGETCOND 💅
re2c:YYGETCOND:naked, re2c:define:YYGETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYGETCOND. zero value means free-form API style ✨
re2c:YYGETSTATE, re2c:define:YYGETSTATE — defines the API primitive YYGETSTATE 🦋
re2c:YYGETSTATE:naked, re2c:define:YYGETSTATE:naked — overrides the more generic re2c:api:style configuration for YYGETSTATE. zero value corresponds to free-form API style 🌸
re2c:YYGETACCEPT, re2c:define:YYGETACCEPT — defines the API primitive YYGETACCEPT ⭐
re2c:YYLESSTHAN, re2c:define:YYLESSTHAN — defines the generic API primitive YYLESSTHAN 💜
re2c:YYLIMIT, re2c:define:YYLIMIT — defines the API primitive YYLIMIT 🌈
re2c:YYMARKER, re2c:define:YYMARKER — defines the API primitive YYMARKER 🎀
re2c:YYMTAGN, re2c:define:YYMTAGN — defines the generic API primitive YYMTAGN ✨
re2c:YYMTAGP, re2c:define:YYMTAGP — defines the generic API primitive YYMTAGP 🧚
re2c:YYPEEK, re2c:define:YYPEEK — defines the generic API primitive YYPEEK 👀
re2c:YYRESTORE, re2c:define:YYRESTORE — defines the generic API primitive YYRESTORE 🦋
re2c:YYRESTORECTX, re2c:define:YYRESTORECTX — defines the generic API primitive YYRESTORECTX 💫
re2c:YYRESTORETAG, re2c:define:YYRESTORETAG — defines the generic API primitive YYRESTORETAG 🏷️
re2c:YYSETCOND, re2c:define:YYSETCONDITION — defines the API primitive YYSETCOND 💅
re2c:YYSETCOND@cond, re2c:define:YYSETCONDITION@cond — the sigil used for argument substitution in the YYSETCOND definition. default value is @@. overrides the more generic re2c:api:sigil configuration ✨
re2c:YYSETCOND:naked, re2c:define:YYSETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYSETCOND. zero value corresponds to free-form API style 🎀
re2c:YYSETSTATE, re2c:define:YYSETSTATE — defines the API primitive YYSETSTATE 🌸
re2c:YYSETSTATE@state, re2c:define:YYSETSTATE@state — the sigil used for argument substitution in the YYSETSTATE definition. default value is @@. overrides the more generic re2c:api:sigil configuration 🪄
re2c:YYSETSTATE:naked, re2c:define:YYSETSTATE:naked — overrides the more generic re2c:api:style configuration for YYSETSTATE. zero value corresponds to free-form API style ✨
re2c:YYSETACCEPT, re2c:define:YYSETACCEPT — defines the API primitive YYSETACCEPT ⭐
re2c:YYSKIP, re2c:define:YYSKIP — defines the generic API primitive YYSKIP 🦋
re2c:YYSHIFT, re2c:define:YYSHIFT — defines the generic API primitive YYSHIFT 💜
re2c:YYCOPYMTAG, re2c:define:YYCOPYMTAG — defines the generic API primitive YYCOPYMTAG 🎀
re2c:YYCOPYSTAG, re2c:define:YYCOPYSTAG — defines the generic API primitive YYCOPYSTAG 🌸
re2c:YYSHIFTMTAG, re2c:define:YYSHIFTMTAG — defines the generic API primitive YYSHIFTMTAG ✨
re2c:YYSHIFTSTAG, re2c:define:YYSHIFTSTAG — defines the generic API primitive YYSHIFTSTAG 🧚
re2c:YYSTAGN, re2c:define:YYSTAGN — defines the generic API primitive YYSTAGN 🦋
re2c:YYSTAGP, re2c:define:YYSTAGP — defines the generic API primitive YYSTAGP 💫
re2c:yyaccept, re2c:variable:yyaccept — defines the API primitive yyaccept 👑
re2c:yybm, re2c:variable:yybm — defines the API primitive yybm 🎀
re2c:yybm:hex, re2c:variable:yybm:hex — if set to nonzero, bitmaps for the --bit-vectors option are generated in hexadecimal format. the default is zero, so bitmaps come out in decimal format 🔢
re2c:yych, re2c:variable:yych — defines the API primitive yych ⭐
re2c:yych:emit, re2c:variable:yych:emit — if set to zero, the yych definition is not generated. default is non-zero 🚫
re2c:yych:conversion, re2c:variable:yych:conversion — if set to non-zero, re2java automatically generates a conversion to YYCTYPE every single time yych is read. default is zero, meaning no conversion 💅
re2c:yych:literals, re2c:variable:yych:literals — specifies the form of literals that yych is matched against. possible values are: char (character literals in single quotes, and non-printable ones use escape sequences that start with backslash), hex (hexadecimal integers), and char_or_hex (a mixture of both — character literals for printable characters and hexadecimal integers for the others) 🔤
re2c:yyctable, re2c:variable:yyctable — defines the API primitive yyctable 🦋
re2c:yynmatch, re2c:variable:yynmatch — defines the API primitive yynmatch 🎀
re2c:yypmatch, re2c:variable:yypmatch — defines the API primitive yypmatch ✨
re2c:yytarget, re2c:variable:yytarget — defines the API primitive yytarget 🎯
re2c:yystable, re2c:variable:yystable — deprecated, bestie. she's retired 💀
re2c:yystate, re2c:variable:yystate — defines the API primitive yystate 🌸
re2c:yyfill, re2c:variable:yyfill — defines the API primitive yyfill 💜
re2c:yyfill:check — if set to zero, it suppresses the generation of the pre-YYFILL check for the number of input characters (that's the YYLESSTHAN definition in the generic API, and the YYLIMIT-based comparison in the C pointer API). default is non-zero, so the check is generated ✅
re2c:yyfill:enable — if set to zero, it suppresses the generation of YYFILL (together with the check). use this when the whole input fits into one piece of memory — no buffering needed — and the end-of-input checks don't rely on the YYFILL checks (like when a sentinel character is used).

🚨 real talk: use the warnings (-W option) and the re2c:sentinel configuration to verify that the generated lexer cannot read past the end of input. do not skip the vibe check on this one, fr fr.

the default is non-zero, meaning YYFILL is enabled 🫶

re2c:yyfill:parameter — set this to zero and it straight-up suppresses the parameter that gets passed to YYFILL. That parameter is literally "the minimum number of characters that must be supplied," so yeah, it matters 💅 Defaults to non-zero (aka the parameter does get generated). You can override this config with re2c:YYFILL:naked or re2c:api:style, iykyk.
re2c:yyfn:sep — the separator used in YYFN elements (defaults to semicolon, bestie 🎀).
re2c:yyfn:throw — the exceptions thrown by the YYFN function. Defaults to empty, which means... no exceptions thrown. Periodt 🙈

✨ Regular expressions ✨

okay so re2java uses this exact syntax for regular expressions, and you're gonna want to screenshot this tbh:

"foo" — case-sensitive string literal. As it should be 💅
'foo' — case-insensitive string literal. lowercase rights!!
[a-xyz], [^a-xyz] — a character class (possibly negated, hence the lil ^).
. — any character except newline.
R \ S — the difference of character classes R and S.
R* — zero or more occurrences of R.
R+ — one or more occurrences of R.
R? — optional R.
R{n} — repetition of R exactly n times.
R{n,} — repetition of R at least n times.
R{n,m} — repetition of R from n to m times.
(R) — just R; the parentheses are there to override precedence. If submatch extraction is enabled, (R) is a capturing or a non-capturing group depending on the --invert-captures option. the way that's a whole mood 😭
(!R) — if submatch extraction is enabled, (!R) is a non-capturing or a capturing group depending on the --invert-captures option. Notice the flip — it's giving opposite day 🦋
R S — concatenation: R followed by S.
R | S — alternative: R or S.
R / S — lookahead: R followed by S, but S is not consumed. sneaky but iconic 👑
name — a regular expression defined as name (or the literal string "name" in Flex compatibility mode).
{name} — a regular expression defined as name in Flex compatibility mode.
@stag — an s-tag: saves the last input position at which @stag matches in a variable named stag.
#mtag — an m-tag: saves all input positions at which #mtag matches in a variable named mtag. collecting them all fr fr 🍓
$ — end of input.

Character classes and string literals may contain the following escape sequences: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and \Uhhhhhhhh. no cap, that's the whole list 💖

✨ Actions ✨

here's the list of predefined actions re2java supports — these are the main characters of the codegen show:

!entry code — the entry action binds a user-defined block of code to the start state of the current finite state machine. If start conditions are used, the entry action can be set individually for each condition. You'd use this to do initialization stuff, e.g. saving the start location of a lexeme. it's giving setup era ✨
!pre_rule code — the pre-rule action prepends a user-defined block of code to the semantic actions of all rules in the current block (or condition, if start conditions are used). Use it to factor out the common part of all semantic actions (e.g. saving the end location of a lexeme). efficiency queen 👑
!post_rule code — the post-rule action appends a user-defined block of code to the semantic actions of all rules in the current block (or condition, if start conditions are used). This one's for emitting trap statements that guard against unintended control flow. safety first bestie 🫶

✨ Directives ✨

full list of directives re2java supports, no skips:

!use:name ; — an in-block use directive that merges a previously defined rules block with the specified name into the current block. The named definitions, configurations and rules of the referenced block get added to the current ones. Conflicts between overlapping rules and configurations are resolved the usual way: the first rule takes priority, and the latest configuration overrides the preceding ones. One exception tho: the special rules *, $ and <!> — for those, a block-local definition always takes priority. A use directive can go anywhere inside a block, and multiple use directives are allowed. stack 'em up 🎀
!include file ; — this directive is the same as an include block: it inserts the file contents verbatim in place of the directive. literally copy-paste energy 💅

✨ Program interface ✨

okay this is the big one, stay with me 🧚

The generated code interfaces with the outer program with the help of primitives, collectively referred to as the API. Which primitives should be defined for a particular program depends on multiple factors, including the complexity of regular expressions, input representation, buffering and the use of various features. All the necessary primitives should be defined by you, the user, in the form of macros, functions, variables or any other suitable form that makes the generated code syntactically and semantically correct. re2java does not (and cannot) check the definitions — so if anything's missing or defined incorrectly, the generated program may have compile-time or run-time errors. 😭 This manual provides examples of API definitions in the most common cases, so like, use them.

re2java has three API flavors that define the core set of primitives used by a program:

Simple API — the default API for the Java backend. It consists of these primitives: YYINPUT (which should be defined as a sequence of code units, e.g. a string) and YYCURSOR, YYMARKER, YYCTXMARKER, YYLIMIT (which should be defined as indices in YYINPUT). simple, clean, iconic 💖
Record API — useful in cases when lexer state must be stored in a class. Enabled with the --api record option or the re2c:api = record configuration. This API consists of a variable yyrecord (the name can be overridden with re2c:yyrecord) that should be defined as a class with fields yyinput, yycursor, yymarker, yyctxmarker, yylimit — only the fields used by the generated code need to be defined, and their names can be configured. very classy of it 🎀
Generic API — the most flexible API. Enabled with the --api generic option or the re2c:api = generic configuration. It contains primitives for generic operations: YYPEEK, YYSKIP, YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE, YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG, YYLESSTHAN, YYEND. the whole alphabet of vibes 🌈

Here's the full list of API primitives that may be used by the generated code in order to interface with the outer program. buckle up 💀

YYCTYPE — the type of the input characters (code units). For ASCII, EBCDIC and UTF-8 encodings it should be a 1-byte unsigned integer. For UTF-16 or UCS-2 it should be a 2-byte unsigned integer. For UTF-32 it should be a 4-byte unsigned integer. size matters, bestie ✨
YYCURSOR — an l-value that stores the current input position (a pointer or an integer offset in YYINPUT). Initially YYCURSOR should point to the first input character, and later it gets advanced by the generated code. When a rule matches, the YYCURSOR position is the one after the last matched character. main character of the pointer world 👑
YYLIMIT — an r-value that stores the end of input position (a pointer or an integer offset in YYINPUT). Initially YYLIMIT should point to the position after the last available input character. It is not changed by the generated code. The lexer compares YYCURSOR to YYLIMIT in order to determine if there are enough input characters left. the responsible one in the group chat 🫶
YYMARKER — an l-value that stores the position of the latest matched rule (a pointer or an integer offset in YYINPUT). It's used to restore the YYCURSOR position if the longer match fails and the lexer needs to rollback. Initialization is not needed. no prep required, we love low maintenance 💅
YYCTXMARKER — an l-value that stores the position of the trailing context (a pointer or an integer offset in YYINPUT). No initialization needed. YYCTXMARKER is needed only if the lookahead operator / is used. conditional queen 🦋
YYFILL — a generic API primitive with one variable len. YYFILL should provide at least len more input characters or fail. If re2c:eof is used, then len is always 1 and YYFILL should always return to the calling function; zero return value indicates success. If re2c:eof is not used, then YYFILL's return value is ignored and it should not return on failure. The maximum value of len is YYMAXFILL. ⚠️ careful with the return semantics bestie, they're not the same in both cases
YYFN — a primitive that defines a function prototype in the --recursive-functions code model. Its value should be an array of one or more strings, where each string contains two or three components separated by the string specified in the re2c:fn:sep configuration (typically a semicolon). The first array element defines the function name and return type (empty for a void function). Subsequent elements define function arguments: first, the expression for the argument used in function body (usually just a name); second, argument type; third, an optional formal parameter (it defaults to the first component — usually both the argument and the parameter are the same identifier). multi-part, multi-slaying 🌸
YYINPUT — an r-value that stores the current input character sequence (string, buffer, etc.).
YYMAXFILL — an integral constant equal to the maximum value of the argument to YYFILL. It can be generated with a max block. math icon 🔥
YYLESSTHAN — a generic API primitive with one variable len. It should be defined as an r-value of boolean type that equals true if and only if there are less than len input characters left.
YYEND — a generic API primitive with no variables. It should be defined as an r-value of boolean type that equals true if and only if the logical end of input has been reached (excluding any padding or sentinel symbols). YYEND is used to implement the $ symbol in regular expressions. It differs from YYLESSTHAN, which is used to ensure that the lexer won't read past the end of buffer. ⚠️ don't confuse these two, they are not the same thing, i promise 😭
YYPEEK — a generic API primitive with no variables. It should be defined as an r-value of type YYCTYPE that is equal to the character at the current input position. peekaboo 👀
YYSKIP — a generic API primitive that should advance the current input position by one code unit.
YYBACKUP — a generic API primitive that should save the current input position (to be restored with YYRESTORE later).
YYRESTORE — a generic API primitive that should restore the current input position to the value saved by YYBACKUP.
YYBACKUPCTX — a generic API primitive that should save the current input position as the position of the trailing context (to be restored with YYRESTORECTX later).
YYRESTORECTX — a generic API primitive that should restore the trailing context position saved with YYBACKUPCTX.
YYRESTORETAG — bestie this one is a generic API primitive with one variable, tag, and its whole job is to restore the trailing context position to whatever value tag is holding 💅
YYSTAGP — generic API primitive, one variable tag, and tag can be either a pointer or an offset in YYINPUT (peep the submatch extraction section for the deets). YYSTAGP should set tag to the current input position ✨
YYSTAGN — same deal, generic API primitive, one variable tag (pointer or offset in YYINPUT, see submatch extraction section). YYSTAGN should set tag to a value that represents a non-existent input position, iykyk 🫠
YYMTAGP — generic API primitive with one variable tag. YYMTAGP should append the current position to the submatch history of tag (submatch extraction section, go look 👀).
YYMTAGN — generic API primitive with one variable tag. YYMTAGN should append a value that represents a non-existent input position to the submatch history of tag (again: submatch extraction section).
YYSHIFT — generic API primitive with one variable shift, and it should shift the current input position by shift characters. Heads up, the shift value can be negative 😌
YYCOPYSTAG — generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side s-tag variable rhs to the left-hand-side s-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs 💖
YYCOPYMTAG — generic API primitive, two variables, lhs and rhs; it should copy the right-hand-side m-tag variable rhs to the left-hand-side m-tag variable lhs. Same story: for most languages the default definition just assigns lhs to rhs.
YYSHIFTSTAG — generic API primitive with two variables, tag and shift, and it should shift tag by shift code units (yep, the shift value may be negative).
YYSHIFTMTAG — generic API primitive with two variables, tag and shift, that should shift the latest value in the history of tag by shift code units (negative shift values allowed, no cap 🧢).
YYMAXNMATCH — an integral constant equal to the maximal number of POSIX capturing groups in a rule. It gets generated with a maxnmatch block ⭐
YYCONDTYPE — the type of the condition enum. It can be generated either with a conditions block or with the --header option 👑
YYGETACCEPT — a primitive with one variable var that stores the numeric selector of the accepted rule. For most languages this primitive has a default definition that reads from var.
YYSETACCEPT — a primitive with two variables: var (an l-value that stores the numeric selector of the accepted rule) and val (the value of the selector). For most languages the default definition assigns var to val.
YYGETCOND — an r-value of type YYCONDTYPE that's equal to the current condition identifier 🌈
YYSETCOND — a primitive with one variable cond, and it should set the current condition identifier to cond.
YYGETSTATE — an r-value of integer type equal to the current lexer state. It should be initialized to -1 (do not skip this, bestie 🙈).
YYSETSTATE — a primitive with one variable state that should set the current lexer state to state.
YYDEBUG — this primitive is generated only with the -d, --debug-output option. Its purpose is to add logging to the generated code (a typical YYDEBUG definition is a print statement, cute). YYDEBUG statements are generated in every state and have two variables: state (either a DFA state index or -1) and symbol (the current input symbol) 🔥
yyaccept — an l-value of unsigned integral type that stores the number of the latest matched rule. A user definition is necessary only with the --storable-state option.
yybm — a table containing compressed bitmaps for up to 8 transitions (used with the --bitmaps option). The table contains 256 elements and is indexed by 1-byte code units. Each 8-bit element combines boolean values for up to 8 transitions. The k-th bit of the n-th element is true iff the n-th code unit is in the range of the k-th transition. The whole idea of this bitmap is to replace a million if branches or switch cases with one single bit check in the table — efficiency ate, no crumbs 💅
yych — an l-value of type YYCTYPE that stores the current input character. A user definition is necessary only with the -f --storable-state option.
yyctable — jump table generated for the initial condition dispatch (enabled with the combination of the --conditions and --computed-gotos options).
yyfill — an l-value that stores the result of the YYFILL call (this may be necessary for pure functional languages, where YYFILL is a monadic function with a complex return value 🧚).
yynmatch — an l-value of unsigned integral type that stores the number of POSIX capturing groups in the matched rule. Used only with the -P --posix-captures option.
yypmatch — an array of l-values used to hold the tag values corresponding to the capturing parentheses in the matching rule. Array length must be at least yynmatch * 2 (usually YYMAXNMATCH * 2 is a good choice). Used only with the -P --posix-captures option.
yystable — deprecated 🫡 (yes, we're still listing it, that's the vibe).
yystate — an l-value used with the --loop-switch option to store the current DFA state.
yytarget — jump table that contains jump targets (label addresses) for all transitions from a state. This table is local to each state. Generation of yytarget tables is enabled with the --computed-gotos option.

✨ OPTIONS ✨

Okay so some of these options have corresponding configurations, and others are global and cannot be changed after re2c starts reading the input file. Debug options generally require building re2c in a debug configuration. Internal options are useful for experimenting with the algorithms used in re2c. That's just how it is, bestie 😌

-?, --help, -h — show the help message. Our lord and savior.
--api <simple | record | generic> — pick the API the generated code uses to interface with the used-defined code. simple is for the simple cases where there's no need for buffer refilling and storing lexer state. record is for when the lexer state needs to be stored in a record (struct, class, etc.). generic is for the complex cases when the other two APIs just aren't flexible enough 🦋
--bit-vectors, -b — optimize conditional jumps using bit masks. This option implies --nested-ifs.
--captures, --leftmost-captures — enable submatch extraction with leftmost greedy capturing groups. The result is collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule 💖
--captvars, --leftmost-captvars — enable submatch extraction with leftmost greedy capturing groups. The result is collected into variables yytl<k>, yytr<k> for the k-th capturing group ✨
--case-insensitive — treat single-quoted and double-quoted strings as case-insensitive.
--case-inverted — invert the meaning of single-quoted and double-quoted strings: single-quoted strings become case-sensitive and double-quoted strings become case-insensitive. Yes, it's a whole switch-up 🔄
--case-ranges — collapse consecutive cases in switch statements into a range of the form low ... high. This syntax is a C/C++ language extension supported by compilers like GCC, Clang and Tcc. The main advantage over using single cases is smaller generated code and faster generation time, though for some compilers like Tcc it also results in smaller binary size. Literally win-win 🎀
--computed-gotos, -g — optimize conditional jumps using the non-standard "computed goto" extension (which, fair warning, must be supported by the compiler). re2java generates jump tables only in complex cases with a lot of conditional branches. The complexity threshold can be configured with the cgoto:threshold configuration. Relative offsets can be enabled with the cgoto:relative configuration. This option implies --bit-vectors 🔥
--computed-gotos-relative — similar to --computed-gotos but generates relative offsets for jump tables instead (which must be supported by the compiler). This option implies --computed-gotos.
--conditions, --start-conditions, -c — enable support of Flex-like "conditions": multiple interrelated lexers within one block. This is the alternative to manually specifying different re2java blocks connected with goto or function calls 👑
--depfile FILE — write dependency information to FILE in the form of a Makefile rule <output-file> : <input-file> [include-file ...]. This lets you track build dependencies in the presence of include blocks/directives, so that updating include files triggers regeneration of the output file. Heads up, this option depends on the --output option 🙈
--ebcdic, --ecb, -e — generate a lexer that reads input in EBCDIC encoding. re2java assumes the character range is 0 -- 0xFF and the character size is 1 byte.
--empty-class <match-empty | match-none | error> — define how re2java treats empty character classes. With match-empty (the default) an empty class matches empty input (which is illogical, but backwards-compatible, and we respect the legacy 😭). With match-none an empty class always fails to match. With error an empty class raises a compilation error.
--encoding-policy <fail | substitute | ignore> — define how re2java treats Unicode surrogates. With fail, re2java aborts with an error when a surrogate is encountered. With substitute, re2java silently replaces surrogates with the error code point 0xFFFD. With ignore (the default), re2java treats surrogates as normal code points. The Unicode standard says standalone surrogates are invalid, but real-world libraries and programs behave in different ways, so here we are 🫠
--flex-syntax, -F — partial support for Flex syntax: in this mode named definitions don't need the equal sign and the terminating semicolon, and when used they must be surrounded with curly braces. Names without curly braces are treated as double-quoted strings 💀

okay so we're deep in the option pile now and honestly this is where it gets juicy 💅 keep scrolling bestie, it's all here —

--goto-label — uses the "goto/label" code model: the DFA gets encoded as labeled code blocks that are wired together with goto transitions across blocks. Literally only works for languages that actually have a goto statement. no goto, no dice 🌸
--header, --type-header, -t HEADER — generate a HEADER file. what goes in it? you decide, using the special blocks header:on and header:off. and if conditions are in play, the generated header gets a condition enum automatically tacked onto it (unless there's an explicit conditions block already — then it minds its business) ✨
-I PATH — adds PATH to the list of locations searched when looking for include files. pairs adorably with the include block or directive. re2java hunts for FILE in the directory of the parent file, plus every include location you gave it with -I. she's thorough 💖
--input <default | custom> — a deprecated alias for --api, crying softly in a corner 😭 default maps to simple (which genuinely is the default for most backends — but not all of them, so don't come for me), and custom maps to generic.
--input-encoding <ascii | utf8> — controls how re2java parses regular expressions. with ascii (the default) it treats input as ASCII-encoded: any sequence of code units is just a sequence of standalone 1-byte characters. with utf8 it treats input as UTF8-encoded and actually recognizes multibyte characters. glow-up 🦋
--invert-captures — flips the meaning of capturing and non-capturing groups. normally (...) captures and (! ...) doesn't. flip the switch and (! ...) captures while (...) sits it out. uno reverse card 🎀
--lang <none | c | d | go | haskell | java | js | ocaml | python | rust | swift | v | zig> — picks the target language. supported: C, D, Go, Haskell, Java, JS, OCaml, Python, Rust, Swift, V, Zig (and you can add more via user-defined syntax files — see the --syntax option, iykyk). none disables the default syntax configs entirely, so the target language stays undefined. chaotic neutral 🌈
--location-format <gnu | msvc> — sets the location format used in messages. gnu prints locations like 'filename:line:column: ...'. msvc prints them like 'filename(line,column) ...'. default is gnu. pick your fighter ⭐
--loop-switch — uses the "loop/switch" code model: the DFA becomes a loop over a switch statement where each individual state is a switch case. state lives in a variable called yystate. transitions between states update yystate to the case label of the destination state and then execution continues to the head of the loop. round and round she goes 🔥
--computed-continue — this one requires the --loop-switch code model, no exceptions. it uses computed continue statements to pick the next switch case directly, which unlocks jump-table optimization similar to --computed-gotos in the goto/label code model. lowkey a flex ngl 💀
--nested-ifs, -s — uses nested if statements instead of switch statements for conditional jumps. usually gives you more efficient code when your compiler is a non-optimizing one. she works harder so the compiler doesn't have to 🫶
--no-debug-info, -i — do not output line directives. genuinely useful when the generated code lives in version control, because otherwise you get huge autogenerated diffs from the tiniest change. we hate a noisy diff 💅
--no-generation-date — suppresses the date output in the generated file. no timestamp, no problem.
--no-version — suppresses the version output in the generated file. privacy queen 👑
--no-unsafe — don't generate the unsafe wrapper over YYPEEK (this one's Rust-specific, bestie). for performance reasons YYPEEK should dodge bounds-checking, since the lexer already does end-of-input checks in a more efficient way. you can hand it a safe YYPEEK definition instead, or one that's unsafe only in release builds — and that second case is exactly where --no-unsafe saves you from warnings about redundant unsafe blocks. thoughtful ✨
--output, -o OUTPUT — names the OUTPUT file. simple. iconic.
--posix-captures, -P — enables submatch extraction with POSIX-style capturing groups. the result lands in an array yybmatch with capacity 2 * YYMAXNMATCH, and yynmatch gets set to the number of groups for the matching rule. numbers, but make it fashion 🍓
--posix-captvars — also enables submatch extraction with POSIX-style capturing groups, but this time the result goes into the variables yytl<k>, yytr<k> for the k-th capturing group. same energy, different storage 🌸
--recursive-functions — uses a code model built on co-recursive functions, where every single DFA state is its own separate function that may call other state-functions or even itself. main character energy, honestly 🔥
--reusable, -r — deprecated since version 2.2 (reusable blocks are allowed by default now, so she's just here for the vibes). rip 🪦
--skeleton, -S — ignores user-defined interface code and generates a self-contained "skeleton" program. bonus: it also generates input files containing strings derived from the regular grammar plus compressed match results, which get used to verify the "skeleton" behavior on all inputs. it's the tool for hunting bugs in optimizations and code generation. this option is supported only for C — sorry to the other languages 💀
--storable-state, -f — generates a lexer that can store its inner state. super useful for push-model lexers, which get stopped by an outer program when there isn't enough input and then resumed once more input shows up. in this mode you should also define the YYGETSTATE and YYSETSTATE primitives, and the variables yych, yyaccept and state need to be part of the stored lexer state. pack accordingly 🎀
--syntax FILE — loads configurations from the specified FILE and applies them on top of the default syntax file. heads up: FILE can define just a few configurations (when it's used to amend the default syntax file), or it can define a whole new language backend — and in that second case it's recommended to use --lang none. build your own adventure 🦋
--tags, -T — enables submatch extraction with tags. that's it, that's the tweet ✨
--ucs2, --wide-chars, -w — generates a lexer that reads UCS2-encoded input. re2java assumes the character range is 0 -- 0xFFFF and the character size is 2 bytes. this option implies --nested-ifs. noted 📝
--utf8, --utf-8, -8 — generates a lexer that reads input in UTF-8 encoding. re2java assumes the character range is 0 -- 0x10FFFF and the character size is 1 byte. tiny characters, big range 💜
--utf16, --utf-16, -x — generates a lexer that reads UTF16-encoded input. re2java assumes the character range is 0 -- 0x10FFFF and the character size is 2 bytes. this one also implies --nested-ifs 🫶
--utf32, --unicode, -u — generates a lexer that reads UTF32-encoded input. re2java assumes the character range is 0 -- 0x10FFFF and the character size is 4 bytes. and yes, this implies --nested-ifs too. the whole family does it 😌
--verbose — outputs a short message when things go right. a little crumb of validation 🥹
--vernum, -V — shows version information in MMmmpp format (major, minor, patch). numbers only, no small talk.
--version, -v — shows version information. the chatty one 💖
--single-pass, -1 — deprecated. does literally nothing (single pass is the default now). ghost of options past 👻
--debug-output, -d — emits YYDEBUG invocations into the generated code. perfect for tracing lexer execution when you need to see what's actually happening in there. detective mode 🔍
--dump-adfa — debug option: outputs the DFA after tunneling, in .dot format.
--dump-cfg — debug option: outputs the control flow graph of tag variables, in .dot format.
--dump-closure-stats — debug option: outputs statistics on the number of states in closure.
--dump-dfa-det — debug option: outputs the DFA immediately after determinization, in .dot format.
--dump-dfa-min — debug option: outputs the DFA after minimization, in .dot format.
--dump-dfa-tagopt — debug option: outputs the DFA after tag optimizations, in .dot format.
--dump-dfa-tree — debug option: outputs the DFA under construction, with states represented as tag history trees, in .dot format.
--dump-dfa-raw — debug option: outputs the DFA under construction with expanded state-sets, in .dot format.
--dump-interf — debug option: outputs the interference table produced by liveness analysis of tag variables. science 🔬
--dump-nfa — debug option: outputs the NFA, in .dot format.
--emit-dot, -D — instead of the normal output, generates the lexer graph in .dot format. you can then turn that into an image with a little help from Graphviz, like:

dot -Tpng -odfa.png dfa.dot

pretty pictures of your state machine, we love to see it 🎨

--dfa-minimization <moore | table> — internal option: the DFA minimization algorithm re2java uses. moore is the Moore algorithm (and it's the default). table is the "table filling" algorithm. both should produce the same DFA up to states relabeling — table filling is simpler and much slower, and it serves as a reference implementation. slow but honest 🐢
--eager-skip — internal option: makes the generated lexer advance the input position eagerly — right after reading the input symbol. this changes the default behavior, where the input position is advanced lazily, i.e. after the transition to the next state. eager vs lazy, choose your timeline ⏰
--no-lookahead — internal option, deprecated. it used to enable the TDFA(0) algorithm. unlike TDFA(1), TDFA(0) doesn't use one-symbol lookahead — it applies register operations to the incoming transitions instead of the outgoing ones. benchmarks showed TDFA(0) is less efficient than TDFA(1), so yeah. she retired 💀
--no-optimize-tags — internal option: suppresses optimization of tag variables, which is handy for debugging. sometimes you want the un-glamoured version 🙈
--posix-closure <gor1 | gtop> — internal option: picks the shortest-path algorithm used to construct the epsilon-closure with POSIX disambiguation semantics. gor1 (the default) means the Goldberg-Radzik algorithm, and gtop means "global topological order" algorithm. acronym girlies rise 🌟
--posix-prectable <complex | naive> — internal option: picks the algorithm used to compute the POSIX precedence table. the complex algorithm computes the precedence table in one traversal of the tag history tree and has quadratic complexity in the number of TNFA states — and it's the default. the naive algorithm has worst-case cubic complexity in the number of TNFA states, but it's much simpler than complex and may be slightly faster in non-pathological cases. simple queen behavior 🧚

--stadfa — bestie this one's an internal option and it's deprecated, so like… don't. It used to switch on the staDFA algorithm, which is different from TDFA in that the register operations live in states instead of on transitions. Benchmarks said staDFA is less efficient than TDFA. So. Yeah. It's giving "we tried it, it flopped." 💀
--fixed-tags <none | toplevel | all> — another internal one ✨ it says whether the fixed-tag optimization should apply to all tags (all), none of them (none), or only the ones in toplevel concatenation (toplevel). Default is all. Now lemme explain "fixed" tags real quick, because this is actually cute: a "fixed" tag is one that sits within a fixed distance of some other tag (that other one is called the "base"). In those cases you only gotta track the base tag, and the fixed tag's value can just be computed as the base tag's value plus a static offset. Math! 🎀 For tags under alternative or repetition you also gotta check whether the base tag has a no-match value — and if it does, the fixed tag should be set to no-match too, disregarding the offset. But for tags in top-level concatenation you can skip that check entirely, because they always match. slay.

⚠️ WARNINGS ⚠️

Warnings can be individually enabled, disabled, and turned into an error. Cute, right? Full control. 👑

-W — turn on all the warnings. every single one. yup.
-Werror — turn warnings into errors. Note that this option alone doesn't turn on any warnings; it only affects the warnings that have been turned on so far, or that will be turned on later. So it's like a vibe-setter, not a starter. 😌
-W<warning> — turn on that warning.
-Wno-<warning> — turn off that warning.
-Werror-<warning> — turn on the warning and treat it as an error (this implies -W<warning>, so like, it's doing both jobs 💅).
-Wno-error-<warning> — don't treat this particular warning as an error. This doesn't turn off the warning itself, bestie — it's still warning, just not a fatal warning. Big difference.
-Wcondition-order — warns if the generated program makes implicit assumptions about condition numbering. The fix? Use either the --header option or a conditions block to generate a mapping of condition names to numbers, and then use the autogenerated condition names. Clean. ✨
-Wempty-character-class — warns if a regular expression contains an empty character class. And ngl, trying to match an empty character class makes no sense: it should always fail. However (and this is the tea ☕), for backwards compatibility reasons re2java permits empty character classes and treats them as empty strings. Use the --empty-class option to change the default behavior.
-Wmatch-empty-string — warns if a rule is nullable (i.e. it matches an empty string). And like, pay attention here: if the lexer runs in a loop and the empty match is unintentional, the lexer may unexpectedly hang in an infinite loop. 😭 infinite loop jumpscare.
-Wswapped-range — warns if the lower bound of a range is greater than its upper bound. The default behavior is to silently swap the range bounds. sneaky little default, but ok.
-Wundefined-control-flow — warns if some input strings cause undefined control flow in the lexer (and yes, the faulty patterns get reported, we're not gonna leave you guessing 🫶). This is a dangerous and common mistake. It can be easily fixed by adding the default rule *, which has the lowest priority, matches any code unit, and always consumes a single code unit. iconic little wildcard.
-Wunreachable-rules — warns about rules that are shadowed by other rules and will never match. rip to those rules, they're literally never getting picked. 💀
-Wdeprecated-eof_rule — warns about standalone end of input rules $ that will be broken by future changes and require fixing. At the moment these rules take precedence when conflicting with other rules, but after the introduction of the generalized end of input symbol $, the precedence order will change and these rules will become shadowed by other rules. so like, get ahead of it bestie 🔥
-Wuseless-escape — warns if a symbol is escaped when it shouldn't be. By default, re2java silently ignores such escapes, but this may as well indicate a typo or an error in the escape sequence. 🙈
-Wnondeterministic-tags — warns if a tag has n-th degree of nondeterminism, where n is greater than 1.
-Wsentinel-in-midrule — warns if the sentinel symbol occurs in the middle of a rule — and this one's serious, because it may cause reads past the end of buffer, crashes, or memory corruption in the generated lexer. ⚠️ This warning is only applicable if the sentinel method of checking for the end of input is used. It is set to an error if re2c:sentinel configuration is used.
-Wundefined-syntax-config — warns if the syntax file specified with --syntax option is missing definitions of some configurations. This helps to maintain user-defined syntax files: if a new release adds configurations, the old syntax file will raise a warning, and the user will be notified. If some configurations are unused and don't need a definition, they should be explicitly set to <undefined>. easy 💅

🌈 SYNTAX FILES 🌈

So, support for different languages in re2c is based on the idea of syntax files. A syntax file is a configuration file that defines the syntax of the target language — and no, not the whole language, just a small part of it that's used by the generated code. Syntax files make re2c very flexible, but PSA: they should not be used as a replacement for re2c: configurations. Their purpose is to define the syntax of the target language, not to customize one particular lexer. Get it right. 🎯

All supported languages have default syntax files that are part of the distribution (see the include/syntax subdirectory); they're also embedded in the re2java binary. Users may provide a custom syntax file that overrides a few configurations for one of the supported languages, or they may choose to redefine all configurations (in which case the --lang none option should be used). ✨

Syntax files contain configurations of four different kinds: feature lists, language configurations, inplace configurations, and code templates. Let's go through 'em. 🦋

📋 Feature lists

A few list configurations define various features supported by a given backend, so that re2java may give a clear error if the user tries to enable an unsupported feature:

`supported_apis`

A list of supported APIs with possible elements simple, record, generic.

`supported_api_styles`

A list of supported API styles with possible elements functions, free-form.

`supported_code_models`

A list of supported code models with possible elements goto-label, loop-switch, recursive-functions.

`supported_targets`

A list of supported codegen targets with possible elements code, dot, skeleton.

`supported_features`

A list of supported features with possible elements nested-ifs, bitmaps, computed-gotos, case-ranges, monadic, unsafe, tags, captures, captvars, computed-continue.

🗣️ Language configurations

A few boolean configurations describe features of the target language that affect the re2java parser and code generator:

`semicolons`

Non-zero if the language uses semicolons after statements.

`backtick_quoted_strings`

Non-zero if the language has backtick-quoted strings.

`single_quoted_strings`

Non-zero if the language has single-quoted strings.

`indentation_sensitive`

Non-zero if the language is indentation sensitive.

`wrap_blocks_in_braces`

Non-zero if compound statements must be wrapped in curly braces.

🧸 Inplace configurations

Syntax files define initial values of all re2c: configurations, since they may differ for different languages. See the configurations section for a full list of all inplace configurations and their meaning. iykyk 💜

🎀 Code templates

Code templates define the syntax of the target language. They're written in a simple domain-specific language with the following formal grammar:

code-template ::
      name '=' code-exprs ';'
    | CODE_TEMPLATE ';'
    | '<undefined>' ';'

code-exprs ::
      <EMPTY>
    | code-exprs code-expr

code-expr ::
      STRING
    | VARIABLE
    | optional
    | list

optional ::
      '(' CONDITIONAL '?' code-exprs ')'
    | '(' CONDITIONAL '?' code-exprs ':' code-exprs ')'

list ::
      '[' VARIABLE ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER '}' ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER ',' NUMBER '}' ':' code-exprs ']'

A code template is a sequence of string literals, variables, optional elements and lists, or a reference to another code template, or a special value <undefined>. Variables are placeholders that get substituted during the code generation phase. ✨

List variables are special: when expanding list templates, re2java repeats the expressions on the right hand side of the colon a few times, each time replacing occurrences of the list variable with a value specific to that repetition. Lists have optional bounds — negative values are counted from the end, so like -1 means the last element. 🫶

Conditional names start with a dot. Both conditionals and variables may be either local (specific to the given code template) or global (allowed in all code templates). When re2java reads a syntax file, it checks that each code template uses only the variables and conditionals that are allowed in it. vibe check: passed ✅

For example, the following code template defines the if-then-else construct for a C-like language:

code:if_then_else =
    [branch{0}: topindent "if " cond " {" nl
        indent [stmt: stmt] dedent]
    [branch{1:-1}: topindent "} else" (.cond ? " if " cond) " {" nl
        indent [stmt: stmt] dedent]
    topindent "}" nl;

Here branch is a list variable: branch{0} expands to the first branch (which is special, since there's no else part), and branch{1:-1} expands to all the remaining branches (if any). stmt is also a list variable: [stmt: stmt] is a nested list that expands to a list of statements in the body of the current branch. topindent, indent, dedent and nl are global variables, and .cond is a local conditional (their meaning is described below). 💅

This code template could produce the following code:

if x {
    // do something
} else if y {
    // do something else
} else {
    // don't do anything
}

okay bestie, here's the FULL roster of code templates re2java supports, with their local variables and conditionals. Big heads-up before you scroll: a particular definition may, but does not have to, use local variables and conditionals. And any code templates you're not using? Set them to <undefined>. Periodt. 💅

code:var_local — declaration or definition of a local variable. Supported variables: type (the type of the variable), name (its name) and init (initial value, if any). Conditionals: .init (true if there is an initializer).
code:var_global — same as code:var_local, except it's used in top-level. ✨
code:const_local — definition of a local constant. Supported variables: type (the type of the constant), name (its name) and init (initial value).
code:const_global — same as code:const_local, except it's used in top-level.
code:array_local — definition of a local array (table). Supported variables: type (the type of array elements), name (array name), size (its size), row (a list variable that does not itself produce any code, but expands list expression as many times as there are rows in the table) and elem (a list variable that expands to all table elements in the current row — it's meant to be nested in the row list). Supported conditional: .const (true if the array is immutable).
code:array_global — same as code:array_local, except it's used in top-level.
code:array_elem — reference to an element of an array (table). Supported variables: array (the name of the array) and index (index of the element).
code:enum — definition of an enumeration (it may be defined using a special language construct for enumerations, or simply as a few standalone constants). Supported variables are type (user-defined enumeration type or type of the constants), elem (list variable that expands to the name of each member) and init (initializer for each member). Conditionals: .init (true if there is an initializer).
code:enum_elem — enumeration element (a member of a user-defined enumeration type or a name of a constant, depending on how code:enum is defined). Supported variables are name (the name of the element) and type (its type).
code:assign — assignment statement. Supported variables are lhs (left hand side) and rhs (right hand side).
code:type_int — signed integer type.
code:type_uint — unsigned integer type.
code:type_yybm — type of elements in the yybm table.
code:type_yytarget — type of elements in the yytarget table.
code:type_yyctable — type of elements in the yyctable table.
code:cmp_eq — operator "equals".
code:cmp_ne — operator "not equals".
code:cmp_lt — operator "less than".
code:cmp_gt — operator "greater than".
code:cmp_le — operator "less or equal".
code:cmp_ge — operator "greater or equal".
code:if_then_else — if-then-else statement with one or more branches. Supported variables: branch (a list variable that does not itself produce any code, but expands list expression as many times as there are branches), cond (condition of the current branch) and stmt (a list variable that expands to all statements in the current branch). Conditionals: .cond (true if the current branch has a condition), .many (true if there's more than one branch).
code:if_then_else_oneline — a specialization of code:if_then_else for the case when all branches have one-line statements. If this is <undefined>, code:if_then_else is used instead.
code:switch — a switch statement with one or more cases. Supported variables: expr (the switched-on expression) and case (a list variable that expands to all cases-groups with their code blocks).
code:switch_cases — a group of switch cases that maps to a single code block. Supported variables are case (a list variable that expands to all cases in this group) and stmt (a list variable that expands to all statements in the code block).
code:switch_cases_oneline — a specialization of code:switch_cases for the case when the code block consists of a single one-line statement. If this is <undefined>, code:switch_cases is used instead.
code:switch_case_range — a single switch case that covers a range of values (possibly consisting of a single value). Supported variable: val (a list variable that expands to all values in the range). Supported conditionals: .many (true if there's more than one value in the range) and .char_literals (true if this is a switch on character literals — some languages provide special syntax for this case).
code:switch_case_default — default switch case.
code:loop — a loop that runs forever (unless interrupted from the loop body). Supported variables: label (loop label), val (initial state), var (state variable), state (a list variable that expands to all state names) and stmt (a list variable that expands to all statements in the loop body or the current state). 🔁
code:continue — continue statement. Supported variables: label (label from which to continue execution), val (value passed to a labeled switch) and var (state variable).
code:goto — goto statement. Supported variables: label (label of the jump target).
code:cgoto — computed goto statement. Supported variables: array (the table containing computed goto information), index (index of the element in the table) and base (base label, only used if .cgoto.relative is true).
code:cgoto:data — initializer expression for a single element in computed goto table. Supported variables: label (the label that is used to initialize the current element), type (underlying type of the elements in the table) and base (base label — only used if .cgoto.relative is true).
code:fndecl — function declaration. Supported variables: name (function name), type (return type), throw (exceptions thrown by this function, maps to re2c:yyfn:throw configuration), arg (a list variable that does not itself produce code, but expands list expression as many times as there are function arguments), argname (name of the current argument), argtype (type of the current argument). Conditional: .type (true if this is a non-void function).
code:fndef — like code:fndecl, but used for function definitions, so it has one additional list variable stmt that expands to all statements in the function body.
code:fncall — function call statement. Supported variables: name (function name), retval (l-value where the return value is stored, if any) and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if return value needs to be saved).
code:tailcall — tail call statement. Supported variables: name (function name), and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if this is a non-void function).
code:recursive_functions — program body with --recursive-functions code model. Supported variables: fn (a list variable that does not itself produce any code, but expands list expression as many times as there are functions), fndecl (declaration of the current function) and fndef (definition of the current function).
code:fingerprint — the fingerprint at the top of the generated output file. Supported variables: ver (re2java version that was used to generate this) and date (generation date). 🎀
code:line_info — the format of line directives (if this is set to <undefined>, no directives are generated). Supported variables: line (line number) and file (filename).
code:abort — a statement that aborts program execution.
code:yydebug — YYDEBUG statement, possibly specialized for different APIs. Supported variables: YYDEBUG, yyrecord, yych (map to the corresponding re2c: configurations), state (DFA state number).
code:yypeek — YYPEEK statement, possibly specialized for different APIs. Supported variables: YYPEEK, YYCTYPE, YYINPUT, YYCURSOR, yyrecord, yych (map to the corresponding re2c: configurations). Conditionals: .cast (true if re2c:yych:conversion is set to non-zero).
code:yyskip — YYSKIP statement, possibly specialized for different APIs. Supported variables: YYSKIP, YYCURSOR, yyrecord (map to the corresponding re2c: configurations).
code:yybackup — YYBACKUP statement, possibly specialized for different APIs. Supported variables: YYBACKUP, YYCURSOR, YYMARKER, yyrecord (map to the corresponding re2c: configurations).
code:yybackupctx — YYBACKUPCTX statement, possibly specialized for different APIs. Supported variables: YYBACKUPCTX, YYCURSOR, YYCTXMARKER, yyrecord (map to the corresponding re2c: configurations).
code:yyskip_yypeek — combined code:yyskip and code:yypeek statement (defaults to code:yyskip followed by code:yypeek).
code:yypeek_yyskip — combined code:yypeek and code:yyskip statement (defaults to code:yypeek followed by code:yyskip).
code:yyskip_yybackup — combined code:yyskip and code:yybackup statement (defaults to code:yyskip followed by code:yybackup).
yylessthan — the YYLESSTHAN statement, possibly specialized for different APIs. Supported variables: YYLESSTHAN, YYCURSOR, YYLIMIT, yyrecord (map to the corresponding re2c: configurations), need (the number of code units to check against). Conditional: .many (true if the need is more than one).
yyend — the YYEND expression, possibly specialized for different APIs. Supported variables: YYEND, YYCURSOR, YYLIMIT.
yybm_filter — condition that's used to filter out yych values that aren't covered by the yybm table (used with --bitmaps option). Supported variable: yych (maps to re2c:yych configuration).
yybm_match — the format of yybm table check (generated with --bitmaps option). Supported variables: yybm, yych (map to the corresponding re2c: configurations), offset (offset in the yybm table that needs to be added to yych) and mask (bit mask that should be applied to the table entry to retrieve the boolean value that needs to be checked)
yytarget_filter — condition that's used to filter out yych values that aren't covered by the yytarget table (used with --computed-gotos option). Supported variable: yych (maps to re2c:yych configuration).

Okay so now ✨ global variables ✨ — here's the full list of every single one that's allowed in syntax files, bestie:

nl — a newline. Literally just a newline. Iconic behavior honestly.
indent — a variable that produces zero code, but has the side-effect of bumping the indentation level up. Sneaky little king 👑
dedent — a variable that also produces no code, but has the side-effect of dropping the indentation level down. The counterpart, the balance, the yin to indent's yang 💜
topindent — the indentation string for the current statement. The indentation level gets tracked and automatically updated by the code generator. It's giving ✨ automatic ✨.

And now 🌈 global conditionals 🌈 — here's every one that's allowed in syntax files. Buckle up, there are a LOT:

.api.simple — true if the simple API is used (--api simple or re2c:api = simple).
.api.generic — true if the generic API is used (--api generic or re2c:api = generic).
.api.record — true if the record API is used (--api record or re2c:api = record).
.api_style.functions — true if the function-like API style is used (re2c:api-style = functions).
.api_style.freeform — true if the free-form API style is used (re2c:api-style = free-form).
.case_ranges — true if the case ranges feature is enabled (--case-ranges or re2c:case-ranges = 1).
.cgoto.relative — true if the relative form of computed goto is used (--computed-gotos-relative or re2c:cgoto:relative = 1).
.computed_continue — true if the loop/switch model uses labeled-switch transitions (--computed-continue).
.code_model.goto_label — true if the code model based on goto/label is used (--goto-label).
.code_model.loop_switch — true if the code model based on loop/switch is used (--loop-switch).
.code_model.recursive_functions — true if the code model based on recursive functions is used (--recursive-function).
.date — true if the generated fingerprint should contain the generation date. 🗓️
.loop_label — true if re2java-generated loops must have a label (re2c:label:yyloop is set to a nonempty string).
.monadic — true if the generated code should be monadic (re2c:monadic = 1). Heads up: this is only relevant for pure functional languages, so if that's not you, don't stress 🧘
.start_conditions — true if start conditions are enabled (--start-conditions).
.storable_state — true if storable state is enabled (--storable-state).
.unsafe — true if re2java should use "unsafe" blocks so it can generate faster code (--unsafe, re2c:unsafe = 1). Also heads up: this one's only relevant for languages that actually have an "unsafe" feature. No cap, don't force it if your language doesn't have it 💀
.version — true if the generated fingerprint should contain the re2java version.
.yyfill.enable — true if YYFILL is enabled (re2c:yyfill:enable = 1).
.yyfn.throw — true if the re2c:yyfn:throw configuration is defined to a nonempty string.

🌈 HANDLING THE END OF INPUT 🌈

Okay so this is the big one, the main character problem of the whole lexer game: how do you know when to STOP? 😭 Like, the lexer is out here vibing, matching stuff, and then... when does it clock out?

There are a few terminating conditions, and here they are:

the lexer may match some rule (including the default rule *) and come to a final state
the lexer may fail to match any rule and come to a default state
the lexer may reach the end of input

Bestie, those are the three ways the party ends. Match a rule and hit the final state, whiff on every rule and land in the default state, or just run clean out of input. That's it, that's the whole goodbye tour 🫶

okay so the first two conditions terminate the lexer in a "natural" way: it comes to a state with no outgoing transitions, and the matching automatically stops. The third condition, end of input, is different: it may happen in any state, and the lexer should be able to handle it. Checking for the end of input interrupts the normal lexer workflow and adds conditional branches to the generated program, therefore it is necessary to minimize the number of such checks. re2java supports a few different methods for handling the end of input. Which one to use depends on the complexity of regular expressions, the need for buffering, performance considerations and other factors. Here is a list of methods:

Sentinel — this method eliminates the need for the end of input checks altogether. It is simple and efficient, but limited to the case when there is a natural "sentinel" character that can never occur in valid input. This character may still occur in invalid input, but it should not be allowed by the regular expressions, except perhaps as the last character of a rule. The sentinel is appended at the end of input and serves as a stop signal: when the lexer reads this character, it is either a syntax error or the end of input. In both cases the lexer should stop. This method is used if YYFILL is disabled with re2c:yyfill:enable = 0; and re2c:eof has the default value -1.
Sentinel with bounds checks — this method is generic: it allows one to handle any input without restrictions on the regular expressions. The idea is to reduce the number of end of input checks by performing them only on certain characters. Similar to the "sentinel" method, one of the characters is chosen as a "sentinel" and appended at the end of input. However, there is no restriction on where the sentinel may occur (in fact, any character can be chosen for a sentinel). When the lexer reads this character, it additionally performs a bounds check. If the current position is within bounds, the lexer resumes matching and handles the sentinel as a regular character. Otherwise it invokes YYFILL (unless it is disabled). If more input is supplied, the lexer will rematch the last character and continue as if the sentinel wasn't there. Otherwise it must be the real end of input, and the lexer stops. This method is used when re2c:eof has non-negative value (it should be set to the numeric value of the sentinel). YYFILL is optional.
Bounds checks with padding — this method is generic, and it may be faster than the "sentinel with bounds checks" method, but it is also more complex. The idea is to partition DFA states into strongly connected components (SCCs) and generate a single check per SCC for enough characters to cover the longest non-looping path in this SCC. This reduces the number of checks, but there is a problem with short lexemes at the end of input, as the check requires enough characters to cover the longest lexeme. This can be fixed by padding the input with a few fake characters that do not form a valid lexeme suffix (so that the lexer cannot match them). The length of padding should be YYMAXFILL, generated with a max block. If there is not enough input, the lexer invokes YYFILL which should supply at least the required number of characters or not return. This method is used if YYFILL is enabled and re2c:eof is -1 (this is the default configuration).
Custom checks — Generic API allows one to override basic operations like reading a character, which makes it possible to include the end-of-input checks as part of them. This approach is error-prone and should be used with caution. To use a custom method, enable generic API with --api custom or re2c:api = custom; and disable default bounds checks with re2c:yyfill:enable = 0; or re2c:yyfill:check = 0;.

The following subsections contain an example of each method.

Sentinel 🛑

This example uses a sentinel character to handle the end of input. The program counts space-separated words in a null-terminated string. The sentinel is null: it is the last character of each input string, and it is not allowed in the middle of a lexeme by any of the rules (in particular, it is not included in character ranges where it is easy to overlook). If a null occurs in the middle of a string, it is a syntax error and the lexer will match default rule *, but it won't read past the end of input or crash (use -Wsentinel-in-midrule warning and re2c:sentinel configuration to verify this). Configuration re2c:yyfill:enable = 0; suppresses the generation of bounds checks and YYFILL invocations.

// re2java $INPUT -o $OUTPUT

class Main {
    // Expects a null-terminated string.
    static int lex(String yyinput) {
        int yycursor = 0;
        int count = 0;

        loop: while (true) {
            /*!re2c
                re2c:YYCTYPE = "char";
                re2c:YYPEEK = "yyinput.charAt(yycursor)";
                re2c:yyfill:enable = 0;

                *      { return -1; }
                [\x00] { return count; }
                [a-z]+ { count += 1; continue loop; }
                [ ]+   { continue loop; }
            */
        }
    }

    public static void main(String []args) {
        assert lex("\0") == 0;
        assert lex("one two three\0") == 3;
        assert lex("f0ur\0") == -1;
    }
};

Sentinel with bounds checks 🛑🔍

This example uses sentinel with bounds checks to handle the end of input (this method was added in version 1.2). The program counts space-separated single-quoted strings. The sentinel character is null, which is specified with re2c:eof = 0; configuration. As in the sentinel method, null is the last character of each input string, but it is allowed in the middle of a rule (for example, 'aaa\0aa'\0 is valid input, but 'aaa\0 is a syntax error). Bounds checks are generated in each state that matches an input character, but they are scoped to the branch that handles null. Bounds checks are of the form YYLIMIT <= YYCURSOR or YYLESSTHAN(1) with generic API. If the check condition is true, lexer has reached the end of input and should stop (YYFILL is disabled with re2c:yyfill:enable = 0; as the input fits into one buffer, see the YYFILL with sentinel section for an example that uses YYFILL). Reaching the end of input opens three possibilities: if the lexer is in the initial state it will match the end-of-input rule $, otherwise it may fallback to a previously matched rule (including default rule *) or go to a default state, causing -Wundefined-control-flow.

// re2java $INPUT -o $OUTPUT

class Main {
    // Expects a null-terminated string.
    static int lex(String yyinput) {
        int yycursor = 0;
        int yymarker = 0;
        int yylimit = yyinput.length() - 1; // yylimit points at the terminating null
        int count = 0;

        loop: while (true) {
            /*!re2c
                re2c:YYCTYPE = "char";
                re2c:YYPEEK = "yyinput.charAt(yycursor)";
                re2c:yyfill:enable = 0;
                re2c:eof = 0;

                str = ['] ([^'\\] | [\\][^])* ['];

                *    { return -1; }
                $    { return count; }
                str  { count += 1; continue loop; }
                [ ]+ { continue loop; }
            */
        }
    }

    public static void main(String []args) {
        assert lex("\0") == 0;
        assert lex("'qu\0tes' 'are' 'fine: \\'' \0") == 3;
        assert lex("'unterminated\\'\0") == -1;
    }
};

Bounds checks with padding 🧸

This example uses bounds checks with padding to handle the end of input (this method is enabled by default). The program counts space-separated single-quoted strings. There is a padding of YYMAXFILL null characters appended at the end of input, where YYMAXFILL value is autogenerated with a max block. It is not necessary to use null for padding --- any characters can be used as long as they do not form a valid lexeme suffix (in this example padding should not contain single quotes, as they may be mistaken for a suffix of a single-quoted string). There is a "stop" rule that matches the first padding character (null) and terminates the lexer (note that it checks if null is at the beginning of padding, otherwise it is a syntax error). Bounds checks are generated only in some states that are determined by the strongly connected components of the underlying automaton. Checks have the form (YYLIMIT - YYCURSOR) < n or YYLESSTHAN(n) with generic API, where n is the minimum number of characters that are needed for the lexer to proceed (it also means that the next bounds check will occur in at most n characters). If the check condition is true, the lexer has reached the end of input and will invoke YYFILL(n) that should either supply at least n input characters or not return. In this example YYFILL always fails and terminates the lexer with an error (which is fine because the input fits into one buffer). See the YYFILL with padding section for an example that refills the input buffer with YYFILL.

// re2java $INPUT -o $OUTPUT

class Main {
    /*!max:re2c*/

    // Expects yymaxfill-padded string.
    static int lex(byte[] str) {
        // Pad string with yymaxfill zeroes at the end.
        byte[] yyinput = new byte[str.length + YYMAXFILL];
        System.arraycopy(str, 0, yyinput, 0, str.length);

        int yycursor = 0;
        int yylimit = yyinput.length;
        int count = 0;

        loop: while (true) {
            /*!re2c
                re2c:YYCTYPE = "int";
                re2c:YYPEEK = "Byte.toUnsignedInt(yyinput[yycursor])";
                re2c:YYFILL = "return -1;";

                str = ['] ([^'\\] | [\\][^])* ['];

                [\x00] {
                    // Check that it is the sentinel, not some unexpected null.
                    return (yycursor - 1 == str.length) ? count : -1;
                }
                str  { count += 1; continue loop; }
                [ ]+ { continue loop; }
                *    { return -1; }
            */
        }
    }

    public static void main(String []args) {
        assert lex("".getBytes()) == 0;
        assert lex("'qu\0tes' 'are' 'fine: \\'' ".getBytes()) == 3;
        assert lex("'unterminated\\'".getBytes()) == -1;
        assert lex("'unexpected \00 null\\'".getBytes()) == -1;
    }
};

🎀 Custom checks 🎀

Okay bestie, this flex uses a custom end-of-input handling method built on the generic API. The program counts space-separated single-quoted strings. It's literally the same as the sentinel example, except that the input is not null-terminated. To cover up for the absence of a sentinel character at the end of input, YYPEEK is redefined so it does a bounds check before it reads the next input character. Heads up though: this is inefficient because checks are done very often. If the check condition fails, YYPEEK returns the real character; otherwise it returns a fake sentinel character. iykyk 💅

// re2java $INPUT -o $OUTPUT

class Main {
    // Expects a string without terminating null.
    static int lex(String str) {
        byte[] yyinput = str.getBytes();
        int yycursor = 0;
        int count = 0;

        loop: while (true) {
            /*!re2c
                re2c:api = generic;
                re2c:YYCTYPE = "int";
                re2c:YYPEEK = "(yycursor < yyinput.length)"
                              " ? Byte.toUnsignedInt(yyinput[yycursor]) : 0";
                re2c:YYSKIP = "yycursor += 1;";
                re2c:yyfill:enable = 0;

                *      { return -1; }
                [\x00] { return count; }
                [a-z]+ { count += 1; continue loop; }
                [ ]+   { continue loop; }
            */
        }
    }

    public static void main(String []args) {
        assert lex("") == 0;
        assert lex("one two three") == 3;
        assert lex("f0ur") == -1;
    }
};

🦋 BUFFER REFILLING 🦋

So like… the need for buffering shows up when the input can't be mapped in memory all at once: either it's too big, or it arrives in a streaming fashion (like reading from a socket, bestie 🌸). The usual technique in such cases is to allocate a fixed-sized memory buffer and process input in chunks that fit into the buffer. When the current chunk is processed, it's moved out and new data is moved in. In practice it's somewhat more complex, because lexer state doesn't consist of a single input position — it's a whole set of interrelated positions:

cursor: the next input character to be read (YYCURSOR in C pointer API or YYSKIP/YYPEEK in generic API)
limit: the position after the last available input character (YYLIMIT in C pointer API, implicitly handled by YYLESSTHAN in generic API)
marker: the position of the most recent match, if any (YYMARKER in default API or YYBACKUP/YYRESTORE in generic API)
token: the start of the current lexeme (implicit in re2java API, as it is not needed for the normal lexer operation and can be defined and updated by the user)
context marker: the position of the trailing context (YYCTXMARKER in C pointer API or YYBACKUPCTX/YYRESTORECTX in generic API)
tag variables: submatch positions (defined with stags and mtags blocks and generic API primitives YYSTAGP/YYSTAGN/YYMTAGP/YYMTAGN)

Not all these are used in every case, but if used, they must be updated by YYFILL. 💀 All active positions are contained in the segment between token and cursor, therefore everything between buffer start and token can be discarded, the segment from token and up to limit should be moved to the beginning of buffer, and the free space at the end of buffer should be filled with new data. In order to avoid frequent YYFILL calls it's best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). The details of YYFILL implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds-checking with padding. Also note that if -f --storable-state option is used, YYFILL has slightly different semantics (described in the section about storable state).

✨ YYFILL with sentinel ✨

If EOF rule is used, YYFILL is a function-like primitive that accepts no arguments and returns a value which is checked against zero. YYFILL invocation is triggered by condition YYLIMIT <= YYCURSOR in C pointer API and YYLESSTHAN() in generic API. A non-zero return value means that YYFILL has failed. A successful YYFILL call must supply at least one character and adjust input positions accordingly. Limit must always be set to one after the last input position in buffer, and the character at the limit position must be the sentinel symbol specified by re2c:eof configuration. The pictures below show the relative locations of input positions in buffer before and after YYFILL call (sentinel symbol is marked with #, and the second picture shows the case when there is not enough input to fill the whole buffer).

                         <-- shift -->
                       >-A------------B---------C-------------D#-----------E->
                       buffer       token    marker         limit,
                                                            cursor
          >-A------------B---------C-------------D------------E#->
                       buffer,  marker        cursor        limit
                       token

                         <-- shift -->
                       >-A------------B---------C-------------D#--E (EOF)
                       buffer       token    marker         limit,
                                                            cursor
          >-A------------B---------C-------------D---E#........
                       buffer,  marker       cursor limit
                       token

Here's an example of a program that reads input file input.txt in chunks of 4096 bytes and uses EOF rule. Obsessed with how clean this is tbh 🫶

// re2java $INPUT -o $OUTPUT

import java.io.*;
import java.nio.file.*;

class Lexer {
    public static final int BUFSIZE = 4096;

    private BufferedInputStream stream;
    private byte[] yyinput;
    private int yycursor;
    private int yymarker;
    private int yylimit;
    private int token;
    private boolean eof;

    public Lexer(File file) throws FileNotFoundException {
        stream = new BufferedInputStream(new FileInputStream(file));
        // Sentinel at `yylimit` offset is set to zero, which triggers YYFILL.
        yyinput = new byte[BUFSIZE + 1];
        yycursor = yymarker = yylimit = token = BUFSIZE;
        eof = false;
    }

    private int fill() throws IOException {
        if (eof) { return -1; } // unexpected EOF

        // Error: lexeme too long. In real life can reallocate a larger buffer.
        if (token < 1) { return -2; }

        // Shift buffer contents (discard everything up to the current token).
        System.arraycopy(yyinput, token, yyinput, 0, yylimit - token);
        yycursor -= token;
        yymarker -= token;
        yylimit -= token;
        token = 0;

        // Fill free space at the end of buffer with new data from file.
        yylimit += stream.read(yyinput, yylimit, BUFSIZE - yylimit);
        yyinput[yylimit] = 0; // append sentinel symbol

        // If read less than expected, this is the end of input.
        eof = yylimit < BUFSIZE;

        return 0;
    }

    // Expects a null-terminated string.
    public int lex() throws IOException {
        int count = 0;
        loop: while (true) {
            token = yycursor;
            /*!re2c
                re2c:YYCTYPE = "int";
                re2c:YYPEEK = "Byte.toUnsignedInt(yyinput[yycursor])";
                re2c:YYFILL = "fill() == 0";
                re2c:eof = 0;

                str = ['] ([^'\\] | [\\][^])* ['];

                *    { return -1; }
                $    { return count; }
                str  { count += 1; continue loop; }
                [ ]+ { continue loop; }
            */
        }
    }

    public static void main(String []args) throws FileNotFoundException, IOException {
        String fname = "input";
        String content = "'qu\0tes' 'are' 'fine: \\'' ".repeat(Lexer.BUFSIZE);

        // Prepare input file: a few times the size of the buffer, containing
        // strings with zeroes and escaped quotes.
        Files.writeString(Paths.get(fname), content);

        int count = 3 * Lexer.BUFSIZE; // number of quoted strings written to file

        // Prepare lexer state: all offsets are at the end of buffer.
        File file = new File(".", fname);
        Lexer lexer = new Lexer(file);

        // Run the lexer.
        int n = lexer.lex();
        assert n == count;

        // Cleanup: remove input file.
        file.delete();
    }
};

💜 YYFILL with padding 💜

In the default case (when EOF rule is not used) YYFILL is a function-like primitive that accepts a single argument and does not return any value. YYFILL invocation is triggered by condition (YYLIMIT - YYCURSOR) < n in C pointer API and YYLESSTHAN(n) in generic API. The argument passed to YYFILL is the minimal number of characters that must be supplied. If it fails to do so, YYFILL must not return to the lexer — for that reason it's best implemented as a macro that returns from the calling function on failure 😭. In case of a successful YYFILL invocation the limit position must be set either to one after the last input position in buffer, or to the end of YYMAXFILL padding (in case YYFILL has successfully read at least n characters, but not enough to fill the entire buffer). The pictures below show the relative locations of input positions in buffer before and after YYFILL invocation (YYMAXFILL padding on the second picture is marked with # symbols).

                         <-- shift -->                 <-- need -->
                       >-A------------B---------C-----D-------E---F--------G->
                       buffer       token    marker cursor  limit

          >-A------------B---------C-----D-------E---F--------G->
                       buffer,  marker cursor               limit
                       token

                         <-- shift -->                 <-- need -->
                       >-A------------B---------C-----D-------E-F        (EOF)
                       buffer       token    marker cursor  limit

          >-A------------B---------C-----D-------E-F###############
                       buffer,  marker cursor                   limit
                       token                        <- YYMAXFILL ->

Here's an example of a program that reads input file input.txt in chunks of 4096 bytes and uses bounds-checking with padding. ✨

// re2java $INPUT -o $OUTPUT

import java.io.*;
import java.nio.file.*;
import java.util.Arrays;

class Lexer {
    /*!max:re2c*/
    public static final int BUFSIZE = 4096;

    private BufferedInputStream stream;
    private byte[] yyinput;
    private int yycursor;
    private int yylimit;
    private int token;
    private boolean eof;

    public Lexer(File file) throws FileNotFoundException {
        stream = new BufferedInputStream(new FileInputStream(file));
        // Prepare lexer state: all offsets are at the end of buffer.
        // This immediately triggers YYFILL, as the YYLESSTHAN condition is true.
        yyinput = new byte[BUFSIZE + YYMAXFILL];
        yycursor = yylimit = token = BUFSIZE;
        eof = false;
    }

    private int fill(int need) throws IOException {
        if (eof) { return -1; } // unexpected EOF

        // Error: lexeme too long. In real life can reallocate a larger buffer.
        if (token < need) { return -2; }

        // Shift buffer contents (discard everything up to the current token).
        System.arraycopy(yyinput, token, yyinput, 0, yylimit - token);
        yycursor -= token;
        yylimit -= token;
        token = 0;

        // Fill free space at the end of buffer with new data from file.
        yylimit += stream.read(yyinput, yylimit, BUFSIZE - yylimit);
        yyinput[yylimit] = 0; // append sentinel symbol

        // If read less than expected, this is the end of input.
        if (yylimit < BUFSIZE) {
            eof = true;
            Arrays.fill(yyinput, yylimit, yylimit + YYMAXFILL, (byte)0);
            yylimit += YYMAXFILL;
        }

        return 0;
    }

    // Expects a null-terminated string.
    public int lex() throws IOException {
        int count = 0;
        loop: while (true) {
            token = yycursor;
            /*!re2c
                re2c:YYCTYPE = "int";
                re2c:YYPEEK = "Byte.toUnsignedInt(yyinput[yycursor])";
                re2c:YYFILL = "if (fill(@@) != 0) { return -2; }";

                str = ['] ([^'\\] | [\\][^])* ['];

                [\x00] {
                    // Check that it is the sentinel, not some unexpected null.
                    return (token == yylimit - YYMAXFILL) ? count : -1;
                }
                str  { count += 1; continue loop; }
                [ ]+ { continue loop; }
                *    { return -1; }
            */
        }
    }

    public static void main(String []args) throws FileNotFoundException, IOException {
        String fname = "input";
        String content = "'qu\0tes' 'are' 'fine: \\'' ".repeat(Lexer.BUFSIZE);

        // Prepare input file: a few times the size of the buffer, containing
        // strings with zeroes and escaped quotes.
        Files.writeString(Paths.get(fname), content);

        int count = 3 * Lexer.BUFSIZE; // number of quoted strings written to file

        // Prepare lexer state: all offsets are at the end of buffer.
        File file = new File(".", fname);
        Lexer lexer = new Lexer(file);

        // Run the lexer.
        int n = lexer.lex();
        assert n == count;

        // Cleanup: remove input file.
        file.delete();
    }
};

🫶 okay that one's a whole meal, bestie. Read it slow, it's literally the entire buffering story in like sixty lines of Java.

✨ FEATURES ✨

🧩 Multiple blocks

Sometimes you lowkey need more than one lexer, and they have to talk to each other — like when there's a high-level state machine that switches between lexer modes 😵‍💫. Two ways to pull this off, and honestly both are valid:

wire up several connected re2java blocks, or
use start conditions (more on those in a sec 💅).

Now the implementation of those connections? That's giving very "depends on your target language", ngl. Here's the tea:

In languages that have a goto statement (C/C++, Go, that crowd), you can put all the blocks in one function, each one prefixed with a label. Transition from block to block is just... a simple goto. Iconic. 💅
In languages that don't have goto (looking at you, Rust), you gotta use a loop with a switch on a state variable — same vibe as the yystate loop/switch that re2java generates itself — or wrap each block in a function and use function calls. 🫶

The example below uses multiple blocks to parse binary, octal, decimal and hexadecimal numbers. Each base gets its own block — very main character of them. The initial block figures out the base and dispatches to the others. And the common configurations? They live in a separate block at the top of the program, and the other blocks inherit them. ✨

// re2java $INPUT -o $OUTPUT

class Parser {
    private String yyinput;
    private int yycursor;
    private int yymarker;
    private int number;

    private void add_digit(int base, int offset) throws ArithmeticException {
        number = Math.addExact(
            Math.multiplyExact(number, base),
            yyinput.charAt(yycursor - 1) - offset);
    }

    public int parse(String str) throws ArithmeticException, IllegalArgumentException {
        yyinput = str;
        yycursor = 0;
        number = 0;

        try {
            /*!re2c
                re2c:YYCTYPE = "char";
                re2c:YYPEEK = "yyinput.charAt(yycursor)";
                re2c:yyfill:enable = 0;

                end = "\x00";

                '0b' / [01]        { return parse_bin(); }
                "0"                { return parse_oct(); }
                ""   / [1-9]       { return parse_dec(); }
                '0x' / [0-9a-fA-F] { return parse_hex(); }
                *                  { throw new IllegalArgumentException("not a number"); }
            */
        } catch (Exception e) {
            return -1;
        }
    }

    private int parse_bin() throws ArithmeticException, IllegalArgumentException {
        /*!re2c
            end   { return number; }
            [01]  { add_digit(2, 48); return parse_bin(); }
            *     { throw new IllegalArgumentException("ill-formed binary number"); }
        */
    }

    private int parse_oct() throws ArithmeticException, IllegalArgumentException {
        /*!re2c
            end   { return number; }
            [0-7] { add_digit(8, 48); return parse_oct(); }
            *     { throw new IllegalArgumentException("ill-formed octal number"); }
        */
    }

    private int parse_dec() throws ArithmeticException, IllegalArgumentException {
        /*!re2c
            end   { return number; }
            [0-9] { add_digit(10, 48); return parse_dec(); }
            *     { throw new IllegalArgumentException("ill-formed decimal number"); }
        */
    }

    private int parse_hex() throws ArithmeticException, IllegalArgumentException {
        /*!re2c
            end   { return number; }
            [0-9] { add_digit(16, 48); return parse_hex(); }
            [a-f] { add_digit(16, 87); return parse_hex(); }
            [A-F] { add_digit(16, 55); return parse_hex(); }
            *     { throw new IllegalArgumentException("ill-formed hexadecimal number"); }
        */
    }

    public static void main(String []args) {
        Parser parser = new Parser();
        assert parser.parse("1234567890\0") == 1234567890;
        assert parser.parse("0b1101\0") == 0b1101;
        assert parser.parse("0x007Fe\0") == 0x7fe;
        assert parser.parse("0644\0") == 0644;
        assert parser.parse("9999999999\0") == -1;
        assert parser.parse("123??\0") == -1;
    }
};

see how each parse_* function is just its own little re2java block, recursively calling itself? that's the "wrap each block in a function" strategy and it ate. 🔥

🎀 Start conditions

Start conditions are the alternate route to "many automata, one vibe" — and you turn them on with --start-conditions. 🦋 They give you a way to encode multiple interrelated automata within the same re2java block.

Here's the deal, bestie:

Each condition corresponds to a single automaton, and it has a unique name that you (the user) specify 😌 plus a unique internal number that re2java defines for you.
Those numbers are literally how you switch between conditions: the generated code uses the YYGETCOND and YYSETCOND primitives to get the current condition or set it to a given number.
Wanna generate the numeric condition identifiers? Use a conditions block, the --header option, or the re2c:header configuration. 💅
And the configuration re2c:cond:enumprefix sets the prefix of the generated identifier. ✨

In condition mode, every rule has to be prefixed with a list of comma-separated condition names in angle brackets — or a wildcard <*> if you mean all conditions. It's giving "no rule left unlabelled". The rule syntax gets extended like this:

< condition-list > regular-expression code

A rule that is merged to every condition on the condition-list. It matches regular-expression and executes the associated code. 🫶

< condition-list > regular-expression => condition code

A rule that is merged to every condition on the condition-list. It matches regular-expression, sets the current condition to condition and executes the associated code. 💫

< condition-list > regular-expression :=> condition

A rule that is merged to every condition on the condition-list. It matches regular-expression and immediately transitions to condition (there is no semantic action). ⚡

< condition-list > !action code

A rule that binds code to the place defined by action in every condition on the condition-list — peep the actions section for the various types of actions. 🔍

<! condition-list > code

A rule that prepends code to semantic actions of all rules for every condition on the condition-list.

💀 heads up: this syntax is deprecated and the !pre_rule action should be used instead (it does exactly the same thing).

< > code

A rule that creates a special entry condition with number zero and name "0" that executes code before jumping to other conditions.

🙈 also deprecated, and the !entry action should be used instead — it gives you more fine-grained control, since the code can be specified on a per-condition basis, and you can jump directly to the condition start without going through condition dispatch.

< > => condition code

Same as the previous rule, except that it sets the next condition. 👑

< > :=> condition

Same as the previous rule, except that it has no associated code and immediately jumps to condition. ✨

And one last thing — the code re2java generates for conditions depends on whether it's using the goto/label approach or the loop/switch approach to encode the automata. 👀 so don't be surprised when the output looks different depending on your target language!

okay so like, in languages that have a goto statement (you know, the OGs like C/C++ and Go), conditions are naturally implemented as blocks of code prefixed with labels of the form yyc_<cond>, where cond is a condition name (btw the label prefix can be changed with re2c:cond:prefix). transitions between conditions are implemented using goto and condition labels. before all conditions re2java generates an initial switch on YYGETSTATE that jumps to the start state of the current condition. the shortcut rules :=> bypass the initial switch and jump directly to the specified condition (re2c:cond:goto can be used to change the default behavior). the rules with semantic actions do not automatically jump to the next condition — this should be done by the user-defined action code. fr fr, don't forget that part 💅

in languages that do not have goto (such as Rust), re2java reuses the yystate variable to store condition numbers. each condition gets a numeric identifier equal to the number of its start state, and a switch between conditions is no different than a switch between DFA states of a single condition. there is no need for a separate initial condition switch. (since the same approach is used to implement storable states, YYGETCOND/YYSETCOND are redundant if both storable states and conditions are used — lowkey iconic how that works out ✨)

the program below uses start conditions to parse binary, octal, decimal and hexadecimal numbers. there is a single block where each base has its own condition, and the initial condition is connected to all of them. the user-defined variable cond stores the current condition number; it is initialized to the number of the initial condition generated with conditions block. bestie this example is literally so good, watch and learn 🎀

// re2java $INPUT -o $OUTPUT -c

class Parser {
    /*!conditions:re2c*/
    private String yyinput;
    private int yycursor;
    private int yymarker;
    private int number;

    private void add_digit(int base, int offset) throws ArithmeticException {
        number = Math.addExact(
            Math.multiplyExact(number, base),
            yyinput.charAt(yycursor - 1) - offset);
    }

    public int parse(String str) throws ArithmeticException, IllegalArgumentException {
        yyinput = str;
        yycursor = 0;
        int yycond = YYC_init;

        number = 0;
        try {
            loop: while (true) {
            /*!re2c
                re2c:YYCTYPE = "char";
                re2c:YYPEEK = "yyinput.charAt(yycursor)";
                re2c:yyfill:enable = 0;

                <*> * { throw new IllegalArgumentException("ill-formed number"); }

                <init> '0b' / [01]        :=> bin
                <init> "0"                :=> oct
                <init> ""   / [1-9]       :=> dec
                <init> '0x' / [0-9a-fA-F] :=> hex

                <bin, oct, dec, hex> "\x00" { return number; }

                <bin> [01]  { add_digit(2, 48); continue loop; }
                <oct> [0-7] { add_digit(8, 48); continue loop; }
                <dec> [0-9] { add_digit(10, 48); continue loop; }
                <hex> [0-9] { add_digit(16, 48); continue loop; }
                <hex> [a-f] { add_digit(16, 87); continue loop; }
                <hex> [A-F] { add_digit(16, 55); continue loop; }
            */
            }
        } catch (Exception e) {
            return -1;
        }
    }

    public static void main(String []args) {
        Parser parser = new Parser();
        assert parser.parse("1234567890\0") == 1234567890;
        assert parser.parse("0b1101\0") == 0b1101;
        assert parser.parse("0x007Fe\0") == 0x7fe;
        assert parser.parse("0644\0") == 0644;
        assert parser.parse("9999999999\0") == -1;
        assert parser.parse("123??\0") == -1;
    }
};

✨ STORABLE STATE ✨

with the --storable-state option re2java generates a lexer that can store its current state, return to the caller, and later resume operations exactly where it left off. the default mode of operation in re2java is a "pull" model, in which the lexer "pulls" more input whenever it needs it. this may be unacceptable in cases when the input becomes available piece by piece (for example, if the lexer is invoked by the parser, or if the lexer program communicates via a socket protocol with some other program that must wait for a reply from the lexer before it transmits the next message 😭). storable state feature is intended exactly for such cases: it allows one to generate lexers that work in a "push" model. when the lexer needs more input, it stores its state and returns to the caller. later, when more input becomes available, the caller resumes the lexer exactly where it stopped. there are a few changes necessary compared to the "pull" model, so pay attention bestie 👑

define YYSETSTATE() and YYGETSTATE(state) primitives.
define yych, yyaccept (if used) and state variables as a part of persistent lexer state. the state variable should be initialized to -1.
YYFILL should return to the outer program instead of trying to supply more input. return code should indicate that lexer needs more input.
the outer program should recognize situations when lexer needs more input and respond appropriately.
optionally use getstate block to generate YYGETSTATE switch detached from the main lexer. this only works for languages that have goto (not in --loop-switch mode).
use re2c:eof and the sentinel with bounds checks method to handle the end of input. padding-based method may not work because it is unclear when to append padding: the current end of input may not be the ultimate end of input, and appending padding too early may cut off a partially read greedy lexeme. furthermore, due to high-level program logic getting more input may depend on processing the lexeme at the end of buffer (which already is blocked due to the end-of-input condition). so yeah, that's a whole thing 🙈

here is an example of a "push" model lexer that simulates reading packets from a socket. the lexer loops until it encounters the end of input and returns to the calling function. the calling function provides more input by "sending" the next packet and resumes lexing. this process stops when all the packets have been sent, or when there is an error. it's giving network energy fr fr 🌐

// re2java $INPUT -o $OUTPUT -f

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.Pipe;

class Lexer {
    enum Status {
        END,
        READY,
        WAITING,
        BIG_PACKET,
        BAD_PACKET
    };

    // Use a small buffer to cover the case when a lexeme doesn't fit.
    // In real world use a larger buffer.
    public static final int BUFSIZE = 10;

    public static class State {
        Pipe.SourceChannel source;
        byte[] yyinput;
        int yycursor;
        int yymarker;
        int yylimit;
        int token;
        int yystate;
        int received;

        public State(Pipe pipe) {
            source = pipe.source();
            // Sentinel at `yylimit` offset is set to zero, which triggers YYFILL.
            yyinput = new byte[BUFSIZE + 1];
            yycursor = yymarker = yylimit = token = BUFSIZE;
            yystate = -1;
            received = 0;
        }
    }

    private static void log(String format, Object... args) {
        if (false) { System.out.printf(format + "\n", args); }
    }

    private static Status fill(State st) throws IOException {
        // Error: lexeme too long. In real life can reallocate a larger buffer.
        if (st.token < 1) { return Status.BIG_PACKET; }

        // Shift buffer contents (discard everything up to the current token).
        System.arraycopy(st.yyinput, st.token, st.yyinput, 0, st.yylimit - st.token);
        st.yycursor -= st.token;
        st.yymarker -= st.token;
        st.yylimit -= st.token;
        st.token = 0;

        // Fill free space at the end of buffer with new data from file.
        ByteBuffer buffer = ByteBuffer.wrap(st.yyinput, st.yylimit, BUFSIZE - st.yylimit);
        int have = st.source.read(buffer);
        if (have != -1) st.yylimit += have; // -1 means that pipe is closed
        st.yyinput[st.yylimit] = 0; // append sentinel symbol

        return Status.READY;
    }

    private static Status lex(State yyrecord) {
        int yych;
        loop: while (true) {
            yyrecord.token = yyrecord.yycursor;
            /*!re2c
                re2c:api = record;
                re2c:YYCTYPE = "int";
                re2c:YYPEEK = "Byte.toUnsignedInt(yyrecord.yyinput[yyrecord.yycursor])";
                re2c:YYFILL = "return Status.WAITING;";
                re2c:eof = 0;

                packet = [a-z]+[;];

                *      { return Status.BAD_PACKET; }
                $      { return Status.END; }
                packet { yyrecord.received += 1; continue loop; }
            */
        }
    }

    public static void test(String[] packets, Status expect) throws IOException {
        // Create a pipe.
        Pipe pipe = Pipe.open();
        Pipe.SinkChannel sink = pipe.sink();

        // Initialize lexer state
        Lexer.State st = new Lexer.State(pipe);

        // Main loop. The buffer contains incomplete data which appears packet by
        // packet. When the lexer needs more input it saves its internal state and
        // returns to the caller which should provide more input and resume lexing.
        int send = 0;
        Status status;
        while (true) {
            status = lex(st);

            if (status == Status.END) {
                log("done: got %d packets", st.received);
                break;
            } else if (status == Status.WAITING) {
                log("waiting...");

                if (send < packets.length) {
                    log("sent packet %d: %s", send, packets[send]);
                    ByteBuffer buffer = ByteBuffer.wrap(packets[send].getBytes());
                    sink.write(buffer);
                    send += 1;
                } else {
                    sink.close();
                }

                          status = fill(st);
                          if (status == Status.BIG_PACKET) {
                              log("error: packet too big");
                              break;
                          }
                          assert status == Status.READY;
                      } else {
                          assert status == Status.BAD_PACKET;
                          log("error: ill-formed packet");
                          break;
                      }
                  }

                  // Check results.
                  assert status == expect;
                  if (status == Status.END) {
                      assert send == st.received;
                  }
              }

              public static void main(String []args) throws IOException {
                  test(new String[]{}, Status.END);
                  test(new String[]{"zero;", "one;", "two;", "three;", "four;"}, Status.END);
                  test(new String[]{"zer0;"}, Status.BAD_PACKET);
                  test(new String[]{"goooooooooogle;"}, Status.BIG_PACKET);
              }
          };

♻️ Reusable blocks ♻️

okay bestie, this is the part where re2java gets modular and honestly? it's giving software engineering 💅.

Reusable blocks of the form /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} can be reused any number of times and combined with other re2java blocks. The <name> is optional btw. A rules block can be used in a use block or directive. And heads up: the code for a rules block is generated at every point of use — so it's not some magic shared thing, it literally gets pasted in each time you use it. no cap.

Use blocks are defined with /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %}. The <name> is optional: if it's not specified, the associated rules block is the most recent one (whether named or unnamed) — so like, whichever one you wrote last, that's your default. 🙈

A use block can add named definitions, configurations and rules of its own. And the important use case (capital I important, bestie) is a lexer that supports multiple input encodings: you write the same rules block once and reuse it multiple times with encoding-specific configurations (see the example below 👇).

There's also the in-block use directive !use:<name>; — you can use it from inside of a re2java block. It merges the referenced block <name> into the current one. If some of the merged rules and configurations overlap with the previously defined ones, conflicts are resolved in the usual way (yes, it has a usual way):

the earliest rule takes priority
the latest configuration overrides preceding ones

One exception though — and this is the icon behavior — are the special rules *, $ and (in condition mode) <!>, for which a block-local definition overrides any inherited ones. The use directive lets you combine different re2java blocks together in one block (see the example below 🔥).

🕰️ Version tea: Named blocks and the in-block use directive were added in re2java version 2.2. Since that version, reusable blocks are allowed by default (no special option is needed). Before version 2.2, reuse mode was enabled with the -r, --reusable option. And before version 1.2, reusable blocks could not be mixed with normal blocks. so if you're time-traveling, know your version, periodt.

🌈 Example of a `!use` directive 🌈

read the comments, they're actually doing the explaining for me here, love that for us 🫶

          // re2java $INPUT -o $OUTPUT

          // This example shows how to combine reusable re2c blocks: two blocks
          // ('colors' and 'fish') are merged into one. The 'salmon' rule occurs
          // in both blocks; the 'fish' block takes priority because it is used
          // earlier. Default rule * occurs in all three blocks; the local (not
          // inherited) definition takes priority.

          /*!rules:re2c:colors
              *                            { throw new IllegalArgumentException("ah"); }
              "red" | "salmon" | "magenta" { return Ans.COLOR; }
          */

          /*!rules:re2c:fish
              *                            { throw new IllegalArgumentException("oh"); }
              "haddock" | "salmon" | "eel" { return Ans.FISH; }
          */

          class Main {
              enum Ans {COLOR, FISH, DUNNO};

              static Ans lex(String yyinput) { // no-throw, as '*' rules are overridden
                  int yycursor = 0;
                  int yymarker = 0;

                  /*!re2c
                      re2c:yyfill:enable = 0;
                      re2c:YYCTYPE = "char";
                      re2c:YYPEEK = "yyinput.charAt(yycursor)";

                      !use:fish;
                      !use:colors;
                      * { return Ans.DUNNO; } // overrides inherited '*' rules
                  */
              }

              public static void main(String []args) {
                  assert lex("salmon") == Ans.FISH;
                  assert lex("what?") == Ans.DUNNO;
              }
          };

see how "salmon" shows up in both blocks? 🐟 the fish block wins because it's used earlier (!use:fish; comes before !use:colors;). and the * rule in the re2c block is local, so it overrides the inherited * rules from both reusable blocks — hence lex("what?") gives you Ans.DUNNO instead of throwing. slay.

🦋 Example of a `/!use:re2c ... /` block 🦋

          // re2java $INPUT -o $OUTPUT --input-encoding utf8

          // This example supports multiple input encodings: UTF-8 and UTF-32.
          // Both lexers are generated from the same rules block, and the use
          // blocks add only encoding-specific configurations.

          /*!rules:re2c
              re2c:yyfill:enable = 0;
              re2c:YYPEEK = "yyinput[yycursor]";
              re2c:indent:top = 1;

              "∀x ∃y" { return true; }
              *       { return false; }
          */

          class Main {
              static boolean lex_utf8(int[] yyinput) {
                  int yycursor = 0;
                  int yymarker = 0;
                  /*!use:re2c
                      re2c:YYCTYPE = "int"; // Java lacks unsigned 8-bit integer type
                      re2c:encoding:utf8 = 1;
                  */
              }

              static boolean lex_utf32(int[] yyinput) {
                  int yycursor = 0;
                  int yymarker = 0;
                  /*!use:re2c
                      re2c:YYCTYPE = "int";
                      re2c:encoding:utf32 = 1;
                  */
              }

              public static void main(String []args) {
                  // we have to use `int`, because `byte`in Java cannot represent values greater than 127
                  int[] s_utf8 = new int[]{0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79};
                  assert lex_utf8(s_utf8);

                  int[] s_utf32 = new int[]{0x2200, 0x78, 0x20, 0x2203, 0x79};
                  assert lex_utf32(s_utf32);
              }
          };

one rules block, two lexers, zero duplicated rules. that's the whole vibe 💜. note that both use blocks set re2c:YYCTYPE = "int" because Java lacks an unsigned 8-bit integer type, and the example matches "∀x ∃y" in both UTF-8 and UTF-32. the int thing in main is because byte in Java cannot represent values greater than 127. iykyk.

🔎 Submatch extraction 🔎

re2java has two options for submatch extraction. buckle up bestie, we're going deeper.

🏷️ Tags

The first option is to use standalone tags of the form @stag or #mtag, where stag and mtag are arbitrary used-defined names. (yes, "used-defined" — that's what the page says, i'm not touching it 😭)

Tags are enabled with the -T, --tags option or the re2c:tags = 1 configuration. Semantically tags are position markers: they can be inserted anywhere in a regular expression, and they bind to the corresponding position (or multiple positions) in the input string.

S-tags bind to the last matching position
M-tags bind to a list of positions (they may be used in repetition subexpressions, where a single position in a regular expression corresponds to multiple positions in the input string)

All tags should be defined by the user, either manually or with the help of svars and mvars blocks. If there is more than one way tags can be matched against the input, ambiguity is resolved using the leftmost greedy disambiguation strategy.

🎯 Captures

The second option is to use capturing groups. They are enabled with the --captures option or the re2c:captures = 1 configuration.

There are two flavours for different disambiguation policies:

--leftmost-captures (the default) is for the leftmost greedy policy
--posix-captures is for the POSIX longest-match policy

In this mode all parenthesized subexpressions are considered capturing groups, and a bang can be used to mark non-capturing groups: (! ... ). With the --invert-captures option or the re2c:invert-captures = 1 configuration, the meaning of bang is inverted 🫣.

The number of groups for the matching rule is stored in a variable yynmatch (the whole regular expression is group number zero), and submatch results are stored in the yypmatch array. Both yynmatch and yypmatch should be defined by the user, and the yypmatch size must be at least [yynmatch * 2]. Use the maxnmatch block to define YYMAXNMATCH, a constant that equals to the maximum value of yynmatch among all rules.

🧺 Captvars

Another way to use capturing groups is the --captvars option or the re2c:captvars = 1 configuration. The only difference with --captures is in the way the generated code stores submatch results: instead of yynmatch and yypmatch, re2java generates variables yytl<k> and yytr<k> for the k-th capturing group (the user should declare these using an svars block).

Captures with variables support two disambiguation policies:

--leftmost-captvars or re2c:leftmost-captvars = 1 for the leftmost greedy policy (the default one)
--posix-captvars or re2c:posix-captvars for the POSIX longest-match policy

under the hood 🛠️ all these options translate into tags and Tagged Deterministic Finite Automata with Lookahead. The core idea of TDFA is to minimize the overhead on submatch extraction. In the extreme, if there're no tags or captures in a regular expression, TDFA is just an ordinary DFA. If the number of tags is moderate, the overhead is barely noticeable.

The generated TDFA uses a number of tag variables which do not map directly to tags: a single variable may be used for different tags, and a tag may require multiple variables to hold all its possible values. Eventually ambiguity is resolved, and only one final variable per tag survives. Tag variables should be defined using stags or mtags blocks. If lexer state is stored, tag variables should be part of it. They also need to be updated by YYFILL.

S-tags support the following operations:

save input position to an s-tag: t = YYCURSOR with the C pointer API or a user-defined operation YYSTAGP(t) with the generic API
save default value to an s-tag: t = NULL with the C pointer API or a user-defined operation YYSTAGN(t) with the generic API
copy one s-tag to another: t1 = t2

M-tags support the following operations:

append input position to an m-tag: a user-defined operation YYMTAGP(t) with both default and generic API
append default value to an m-tag: a user-defined operation YYMTAGN(t) with both default and generic API
copy one m-tag to another: t1 = t2

S-tags can be implemented as scalar values (pointers or offsets). M-tags need a more complex representation, as they need to store a sequence of tag values. The most naive and inefficient representation of an m-tag is a list (array, vector) of tag values; a more efficient representation is to store all m-tags in a prefix-tree represented as array of nodes (v, p), where v is tag value and p is a pointer to parent node. 🌳

Here is a simple example of using s-tags to parse semantic versions consisting of three numeric components: major, minor, patch (the latter is optional). See below for a more complex example that uses YYFILL. 🍓

          // re2java $INPUT -o $OUTPUT

          import java.util.Optional;

          class Main {
              static class SemVer {
                  int major;
                  int minor;
                  int patch;

                  public SemVer(int m, int n, int k) {
                      major = m;
                      minor = n;
                      patch = k;
                  }

                  public boolean equals(SemVer v) {
                      return major == v.major && minor == v.minor && patch == v.patch;
                  }
              };

              static Optional<SemVer> parse(String yyinput) {
                  int yycursor = 0;
                  int yymarker = 0;

                  // Final tag variables available in semantic action.
                  /*!svars:re2c format = "int @@;"; */

                  // Intermediate tag variables used by the lexer (must be autogenerated).
                  /*!stags:re2c format = "int @@ = -1;"; */

                  /*!re2c
                      re2c:YYCTYPE = "char";
                      re2c:YYPEEK = "yyinput.charAt(yycursor)";
                      re2c:yyfill:enable = 0;
                      re2c:tags = 1;

                      num = [0-9]+;

                      @t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] {
                          int major = Integer.valueOf(yyinput.substring(t1, t2));
                          int minor = Integer.valueOf(yyinput.substring(t3, t4));
                          int patch = (t5 == -1) ? 0 : Integer.valueOf(yyinput.substring(t5, yycursor - 1));
                          return Optional.of(new SemVer(major, minor, patch));
                      }
                      * { return Optional.empty(); }
                  */
              }

              public static void main(String []args) {
                  assert parse("23.34\0").get().equals(new SemVer(23, 34, 0));
                  assert parse("1.2.99999\0").get().equals(new SemVer(1, 2, 99999));
                  assert !parse("1.a\0").isPresent();
              }
          };

okay so next we're leveling up to the s-tags + YYFILL combo, and it's giving main-character energy fr fr 💅. The mission: parse a whole file of newline-separated semantic versions. Tag variables are part of the lexer state — periodt — and they get adjusted inside YYFILL just like every other input position. This is literally necessary for s-tags, because their values get invalidated after the buffer contents shift 😭 no cap. In a custom implementation where tag variables store offsets relative to the start of the input string instead of the buffer, it may not be necessary — and iykyk, that may be the case with m-tags.

watch the comments in this one, bestie — // Prepare input file: a few times the size of the buffer, containing / // strings with zeroes and escaped quotes. is in there exactly as written, and I am not touching it 🙈

          // re2java $INPUT -o $OUTPUT

          import java.io.*;
          import java.nio.file.*;
          import java.util.*;

          class Lexer {
              static class SemVer {
                  int major;
                  int minor;
                  int patch;

                  public SemVer(int m, int n, int k) {
                      major = m;
                      minor = n;
                      patch = k;
                  }

                  public boolean equals(SemVer v) {
                      return major == v.major && minor == v.minor && patch == v.patch;
                  }
              };

              public static final int BUFSIZE = 4096;

              private BufferedInputStream stream;
              private byte[] yyinput;
              private int yycursor;
              private int yymarker;
              private int yylimit;
              private int token;
              // Intermediate tag variables used by the lexer (must be autogenerated).
              /*!stags:re2c format = "private int @@;\n"; */
              private boolean eof;

              public Lexer(File file) throws FileNotFoundException {
                  stream = new BufferedInputStream(new FileInputStream(file));
                  // Sentinel at `yylimit` offset is set to zero, which triggers YYFILL.
                  yyinput = new byte[BUFSIZE + 1];
                  yycursor = yymarker = yylimit = token = BUFSIZE;
                  /*!stags:re2c format = "@@ = -1;\n"; */
                  eof = false;
              }

              private int fill() throws IOException {
                  if (eof) { return -1; } // unexpected EOF

                  // Error: lexeme too long. In real life can reallocate a larger buffer.
                  if (token < 1) { return -2; }

                  // Shift buffer contents (discard everything up to the current token).
                  System.arraycopy(yyinput, token, yyinput, 0, yylimit - token);
                  yycursor -= token;
                  yymarker -= token;
                  yylimit -= token;
                  /*!stags:re2c format = "if (@@ != -1) {@@ -= token;}\n"; */
                  token = 0;

                  // Fill free space at the end of buffer with new data from file.
                  yylimit += stream.read(yyinput, yylimit, BUFSIZE - yylimit);
                  yyinput[yylimit] = 0; // append sentinel symbol

                  // If read less than expected, this is the end of input.
                  eof = yylimit < BUFSIZE;

                  return 0;
              }

              private int readInt(int tag1, int tag2) {
                  int n = 0;
                  for (int i = tag1; i < tag2; ++i) { n = n * 10 + (yyinput[i] - 48); }
                  return n;
              }

              public Optional<ArrayList<SemVer>> lex() throws IOException {
                  ArrayList<SemVer> vers = new ArrayList<SemVer>();

                  // Final tag variables available in semantic action.
                  /*!svars:re2c format = "int @@;"; */

                  loop: while (true) {
                      token = yycursor;
                      /*!re2c
                          re2c:YYCTYPE = "int";
                          re2c:YYPEEK = "Byte.toUnsignedInt(yyinput[yycursor])";
                          re2c:YYFILL = "fill() == 0";
                          re2c:eof = 0;
                          re2c:tags = 1;

                          num = [0-9]+;

                          @t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\n] {
                              int major = readInt(t1, t2);
                              int minor = readInt(t3, t4);
                              int patch = (t5 == -1) ? 0 : readInt(t5, yycursor - 1);
                              vers.add(new SemVer(major, minor, patch));
                              continue loop;
                          }
                          $ { return Optional.of(vers); }
                          * { return Optional.empty(); }
                      */
                  }
              }

              public static void main(String []args) throws FileNotFoundException, IOException {
                  String fname = "input";
                  String content = "1.22.333\n".repeat(Lexer.BUFSIZE);

                  // Prepare input file: a few times the size of the buffer, containing
                  // strings with zeroes and escaped quotes.
                  Files.writeString(Paths.get(fname), content);

                  // Prepare lexer state: all offsets are at the end of buffer.
                  File file = new File(".", fname);
                  Lexer lexer = new Lexer(file);

                  // Run the lexer.
                  Optional<ArrayList<SemVer>> vers = lexer.lex();

                  // Check results.
                  assert vers.isPresent() && vers.get().size() == BUFSIZE;
                  SemVer v = new SemVer(1, 22, 333);
                  for (int i = 0; i < BUFSIZE; ++i) {
                      assert vers.get().get(i).equals(v);
                  }

                  // Cleanup: remove input file.
                  file.delete();
              }
          };

next up: capturing groups, because sometimes you want the lexer to hand you the substrings directly and honestly? iconic ✨. Same semantic-version parse, but this time with re2c:captvars = 1 and all those yytl1 / yytr1 besties doing their thing 🎀.

       Here is an example of using capturing groups to parse semantic versions.

          // re2java $INPUT -o $OUTPUT

          import java.util.Optional;

          class Main {
              static class SemVer {
                  int major;
                  int minor;
                  int patch;

                  public SemVer(int m, int n, int k) {
                      major = m;
                      minor = n;
                      patch = k;
                  }

                  public boolean equals(SemVer v) {
                      return major == v.major && minor == v.minor && patch == v.patch;
                  }
              };

              static Optional<SemVer> parse(String yyinput) {
                  int yycursor = 0;
                  int yymarker = 0;

                  // Final tag variables available in semantic action.
                  /*!svars:re2c format = "int @@;"; */

                  // Intermediate tag variables used by the lexer (must be autogenerated).
                  /*!stags:re2c format = "int @@ = -1;"; */

                  /*!re2c
                      re2c:YYCTYPE = "char";
                      re2c:YYPEEK = "yyinput.charAt(yycursor)";
                      re2c:yyfill:enable = 0;
                      re2c:captvars = 1;

                      num = [0-9]+;

                      (num) "." (num) ("." num)? [\x00] {
                          int major = Integer.valueOf(yyinput.substring(yytl1, yytr1));
                          int minor = Integer.valueOf(yyinput.substring(yytl2, yytr2));
                          int patch = (yytl3 == -1) ? 0
                                  : Integer.valueOf(yyinput.substring(yytl3 + 1, yytr3));
                          return Optional.of(new SemVer(major, minor, patch));
                      }
                      * { return Optional.empty(); }
                  */
              }

              public static void main(String []args) {
                  assert parse("23.34\0").get().equals(new SemVer(23, 34, 0));
                  assert parse("1.2.99999\0").get().equals(new SemVer(1, 2, 99999));
                  assert !parse("1.a\0").isPresent();
              }
          };

and finally for this part, we have m-tags, the flexible bestie 🦋: parse a version with a variable number of components, where the tag variables are stored in a trie. the way the #t3 / #t4 tags collect every component and then get read back out of the lists? obsessed 🫶.

       Here is an example of using m-tags to parse a version with a variable
       number of components. Tag variables are stored in a trie.

          // re2java $INPUT -o $OUTPUT

          import java.util.*;

          class Main {
              static Optional<int[]> parse(String yyinput) {
                  int yycursor = 0;
                  int yymarker = 0;

                  // Final tag variables available in semantic action.
                  /*!svars:re2c format = "int @@;"; */
                  /*!mvars:re2c format = "List<Integer> @@;"; */

                  // Intermediate tag variables used by the lexer (must be autogenerated).
                  /*!stags:re2c format = "int @@ = -1;"; */
                  /*!mtags:re2c format = "List<Integer> @@ = new ArrayList<>();"; */

                  /*!re2c
                      re2c:YYCTYPE = "char";
                      re2c:YYPEEK = "yyinput.charAt(yycursor)";
                      re2c:YYMTAGP = "@@.add(yycursor);";
                      re2c:YYMTAGN = ""; // do nothing
                      re2c:yyfill:enable = 0;
                      re2c:tags = 1;

                      num = [0-9]+;

                      @t1 num @t2 ("." #t3 num #t4)* [\x00] {
                          int[] vers = new int[t3.size() + 1];
                          vers[0] = Integer.valueOf(yyinput.substring(t1, t2));
                          for (int i = 0; i < t3.size(); ++i) {
                              vers[i + 1] = Integer.valueOf(yyinput.substring(t3.get(i), t4.get(i)));
                          }
                          return Optional.of(vers);
                      }
                      * { return Optional.empty(); }
                  */
              }

              public static void main(String []args) {
                  assert Arrays.equals(parse("1\0").get(), new int[]{1});
                  assert Arrays.equals(parse("1.2.3.4.5.6.7\0").get(), new int[]{1, 2, 3, 4, 5, 6, 7});
                  assert !parse("1.2.\0").isPresent();
              }
          };

every assert, every Optional.empty(), every /*!stags:re2c incantation stayed exactly where it was 💖. the vibe check passed, the code did not move, and we are so back for the next part 🌸.

✨ Encoding support ✨

Okay so before anything else, bestie, you need to know the difference between code points and code units, because this whole section lives and dies by it 💅

A code point is the numeric identifier of a symbol (like, the actual character's ID number). A code unit is the smallest unit of storage in the encoded text. And here's the tea: a single code point may be represented with one or more code units 🫶

In a fixed-length encoding, all code points are represented with the same number of code units.
In a variable-length encoding, code points may be represented with a different number of code units.

Also, lowkey important: the "any" rule [^] matches any code point, but not necessarily any code unit. The only way to match any code unit regardless of the encoding is the default rule *. Noted 📝

The generated lexer works with a stream of code units: yych stores a code unit, and YYCTYPE is the code unit type. Regular expressions, on the other hand, are specified in terms of code points. When re2java compiles regular expressions to automata it translates code points to code units. And this is generally not a simple mapping — in variable-length encodings a single code point range may get translated to a complex code unit graph 😭

The following encodings are supported:

ASCII (enabled by default) — it's a fixed-length encoding with code space [0-255] and 1-byte code points and code units. The default, the classic, the moment 💖
EBCDIC (enabled with --ebcdic or re2c:encoding:ebcdic) — a fixed-length encoding with code space [0-255] and 1-byte code points and code units.
UCS2 (enabled with --ucs2 or re2c:encoding:ucs2) — a fixed-length encoding with code space [0-0xFFFF] and 2-byte code points and code units.
UTF8 (enabled with --utf8 or re2c:encoding:utf8) — a variable-length Unicode encoding. Code unit size is 1 byte. Code points are represented with 1 -- 4 code units. She's popular for a reason 🌈
UTF16 (enabled with --utf16 or re2c:encoding:utf16) — a variable-length Unicode encoding. Code unit size is 2 bytes. Code points are represented with 1 -- 2 code units.
UTF32 (enabled with --utf32 or re2c:encoding:utf32) — a fixed-length Unicode encoding with code space [0-0x10FFFF] and 4-byte code points and code units.

Include files include/unicode_blocks.re, include/unicode_categories.re and include/unicode_properties.re provide re2java definitions for blocks, categories and properties described in the Unicode standard. Free real estate, bestie 🎁

Option --input-encoding specifies source file encoding, which can be used to enable Unicode literals in regular expressions. For example --input-encoding utf8 tells re2java that the source file is in UTF8 (it differs from --utf8 which sets input text encoding — do not mix these up, they are different girlies 🙈). Option --encoding-policy specifies the way re2java handles Unicode surrogates (code points in range [0xD800-0xDFFF]).

Below is an example of a lexer for UTF8 encoded Unicode identifiers 👑

// re2java $INPUT -o $OUTPUT --utf8 -s

/*!include:re2c "unicode_categories.re" */

class Main {
    static boolean lex(String yyinput) {
        int yycursor = 0;
        int yymarker = 0;

        /*!re2c
            re2c:YYCTYPE = "char";
            re2c:YYPEEK = "yyinput.charAt(yycursor)";
            re2c:yyfill:enable = 0;

            // Simplified "Unicode Identifier and Pattern Syntax"
            // (see https://unicode.org/reports/tr31)
            id_start    = L | Nl | [$_];
            id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3];
            identifier  = id_start+;
            // It should be `id_start id_continue*`, but that causes `error: code too large`

            identifier { return true; }
            *          { return false; }
        */
    }

    public static void main(String []args) {
        assert lex("_Ыдентификатор\0");
    }
};

🦋 Include files 🦋

re2java allows one to include other files using a block of the form /*!include:re2c FILE */ or %{include FILE %}, or an in-block directive !include FILE ;, where FILE is a path to the file to be included.

re2java looks for include files in the directory of the including file and in include locations, which can be specified with the -I option. Include blocks/directives in re2java work in the same way as C/C++ #include: FILE contents are copy-pasted verbatim in place of the block/directive. And yes, include files may have further includes of their own — it's turtles all the way down 🐢

Use --depfile option to track build dependencies of the output file on include files. re2java provides some predefined include files that can be found in the include/ subdirectory of the project. These files contain definitions that may be useful to other projects (such as Unicode categories) and form something like a standard library for re2java. Below is an example of using include files ✨

Include file 1 (`definitions.java`)

/*!re2c
    number = [1-9][0-9]*;
*/

Include file 2 (`extra_rules.re.inc`)

// floating-point numbers
frac  = [0-9]* "." [0-9]+ | [0-9]+ ".";
exp   = 'e' [+-]? [0-9]+;
float = frac exp? | [0-9]+ exp;

float { return Num.FLOAT; }

Input file

// re2java $INPUT -o $OUTPUT

/*!include:re2c "definitions.java" */

class Main {
    enum Num {INT, FLOAT, NAN};

    static Num lex(String yyinput) {
        int yycursor = 0;
        int yymarker = 0;

        /*!re2c
            re2c:YYCTYPE = "char";
            re2c:YYPEEK = "yyinput.charAt(yycursor)";
            re2c:yyfill:enable = 0;

            *      { return Num.NAN; }
            number { return Num.INT; }
            !include "extra_rules.re.inc";
        */
    }

    public static void main(String []args) {
        assert lex("123\0") == Num.INT;
        assert lex("123.4567\0") == Num.FLOAT;
    }
};

🎀 Header files 🎀

re2java allows one to generate header file from the input .re file using --header option or re2c:header configuration and block pairs of the form /*!header:re2c:on*/ and /*!header:re2c:off*/, or %{header:on%} and %{header:off%}.

The first block marks the beginning of header file, and the second block marks the end of it. Everything between these blocks is processed by re2java, and the generated code is written to the file specified with --header option or re2c:header configuration (or stdout if neither option nor configuration is used). Autogenerated header file may be needed in cases when re2java is used to generate definitions that must be visible from other translation units. It's giving "sharing is caring" energy 💖

Here is an example of generating a header file that contains definition of the lexer state with tag variables (the number variables depends on the regular grammar and is unknown to the programmer). Spooky? A little. Iconic? Absolutely 👑

Input file

// re2java $INPUT -o $OUTPUT --header lexer/state.java

package headers;

import headers.lexer.State;

/*!header:re2c:on*/
package headers.lexer;

public class State {
    public String yyinput;
    public int yycursor;
    /*!stags:re2c format = "public int @@;\n"; */

    public State(String str) {
        yyinput = str;
        yycursor = 0;
        /*!stags:re2c format = "@@ = 0;\n"; */
    }
};
/*!header:re2c:off*/

class Main {
    static int lex(String str) {
        State yyrecord = new State(str);
        int t;
        /*!re2c
            re2c:api = record;
            re2c:tags = 1;
            re2c:yyfill:enable = 0;
            re2c:YYCTYPE = "char";
            re2c:YYPEEK = "yyrecord.yyinput.charAt(yyrecord.yycursor)";
            re2c:header = "lexer/state.java";

            [a]* @t [b]* { return t; }
        */
    }

    public static void main(String []args) {
        assert lex("ab\0") == 1;
    }
};

Header file

// Generated by re2c

package headers.lexer;

public class State {
    public String yyinput;
    public int yycursor;
    public int yyt1;

    public State(String str) {
        yyinput = str;
        yycursor = 0;
        yyt1 = 0;

    }
};

🌸 Skeleton programs 🌸

With the -S, --skeleton option, re2java ignores all non-re2java code and generates a self-contained C program that can be further compiled and executed 🏃‍♀️💨

The program consists of lexer code and input data. For each constructed DFA (block or condition) re2java generates a standalone lexer and two files: an .input file with strings derived from the DFA and a .keys file with expected match results. The program runs each lexer on the corresponding .input file and compares results with the expectations.

Skeleton programs are very useful for a number of reasons:

They can check correctness of various re2java optimizations (the data is generated early in the process, before any DFA transformations have taken place). Early data = honest data, no cap 🔥
Generating a set of input data with good coverage may be useful for both testing and benchmarking.
Generating self-contained executable programs allows one to get minimized test cases (the original code may be large or have a lot of dependencies). Tiny test case, huge slay 💅

The difficulty with generating input data is that for all but the most trivial cases the number of possible input strings is too large (even if the string length is limited). re2java solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. It uses the following algorithm.

First, it constructs a skeleton of the DFA. For encodings with 1-byte code unit size (such as ASCII, UTF-8 and EBCDIC) skeleton is just an exact copy of the original DFA. For encodings with multibyte code units skeleton is a copy of DFA with certain transitions omitted: namely, re2java takes at most 256 code units for each disjoint continuous range that corresponds to a DFA transition. The chosen values are evenly distributed and include range bounds.

Instead of trying to cover all possible paths in the skeleton (which is infeasible) re2java generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer. The algorithm implementation is limited by ~1Gb of transitions and consumes constant amount of memory (re2java writes data to file as soon as it is generated). Efficient AND humble, we love to see it ⭐

🌈 Visualization and debug 🌈

Okay bestie, this part is so fun — it's the artsy side of re2java, the part where you get to actually see what's going on inside instead of just trusting the vibes. 🎨

-D, --emit-dot — with this flag, re2java does not generate code, periodt. Instead it dumps the generated DFA in DOT format. That's it, that's the whole job. 💅

So once you've got your DOT dump, you can turn it into an actual picture of the DFA using Graphviz or another library — very "main character posts their DFA on the grid" energy. 🦋

Heads up though, babes: this option shows you the final DFA — the one that's already been through a whole bunch of optimizations and transformations. So what you see is the glow-up, not the before pic. 😭

If you do want the earlier stages, those can be dumped with various debug options, such as --dump-nfa, --dump-dfa-raw etc. (see the full list of options). Yes, etc. — there are more, we're just not listing them all here. Go look at the full list, iykyk. ✨

💖 SEE ALSO 💖

You can find more info about re2c at the official website: http://re2c.org 🌐

Similar programs are flex(1), lex(1), and quex (http://quex.sourceforge.net). Cousins! Found family! 🫶

👑 AUTHORS 👑

Time for the credit roll, and honestly? Iconic. 🎬

re2java was originally written by Peter Bumbulis (peter@csg.uwaterloo.ca) way back in 1993 — a literal pioneer, no cap. 🕰️
Marcus Boerger and Dan Nuffer spent several years turning that original idea into a production ready code generator. Dedication fr fr. 💪
Since then it's been maintained and developed by multiple volunteers, most notably:
- Brian Young (bayoung@acm.org)
- Marcus Boerger
- Dan Nuffer (nuffer@users.sourceforge.net)
- Ulya Trofimovich (skvadrik@gmail.com)
- Serghei Iakovlev
- Sergei Trofimovich
- Petr Skocik
- ligfx raekye
- PolarGoose

Absolute legends, the whole crew. Shoutout to every single one of them for keeping this thing alive. 🌸💜

And that's a wrap on the whole page, bestie! 🎀 The footer keeps it humble with just RE2JAVA(1) sitting there in the corner — no fancy date to show off, just the manual page's source/version tag holding it down at the very bottom. Bye, love you, go write some slay regexes! 💅✨🍓

📦 where this page comes from

arch rolling: re2c 4.6-1 · LicenseRef-re2c
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms

RE2JAVA(1)

💅 NAME 💅

✨ SYNOPSIS ✨

🌸 INTRODUCTION 🌸

🦋 BASICS 🦋

🎀 Blocks 🎀

✨ Configurations ✨

✨ Regular expressions ✨

✨ Actions ✨

✨ Directives ✨

✨ Program interface ✨

✨ OPTIONS ✨

⚠️ WARNINGS ⚠️

🌈 SYNTAX FILES 🌈

📋 Feature lists

supported_apis

supported_api_styles

supported_code_models

supported_targets

supported_features

🗣️ Language configurations

semicolons

backtick_quoted_strings

single_quoted_strings

indentation_sensitive

wrap_blocks_in_braces

🧸 Inplace configurations

🎀 Code templates

🌈 HANDLING THE END OF INPUT 🌈

Sentinel 🛑

Sentinel with bounds checks 🛑🔍

Bounds checks with padding 🧸

🎀 Custom checks 🎀

🦋 BUFFER REFILLING 🦋

✨ YYFILL with sentinel ✨

💜 YYFILL with padding 💜

✨ FEATURES ✨

🧩 Multiple blocks

🎀 Start conditions

✨ STORABLE STATE ✨

♻️ Reusable blocks ♻️

🌈 Example of a !use directive 🌈

🦋 Example of a /*!use:re2c ... */ block 🦋

🔎 Submatch extraction 🔎

🏷️ Tags

🎯 Captures

🧺 Captvars

✨ Encoding support ✨

🦋 Include files 🦋

Include file 1 (definitions.java)

Include file 2 (extra_rules.re.inc)

Input file

🎀 Header files 🎀

Input file

Header file

🌸 Skeleton programs 🌸

🌈 Visualization and debug 🌈

💖 SEE ALSO 💖

👑 AUTHORS 👑

`supported_apis`

`supported_api_styles`

`supported_code_models`

`supported_targets`

`supported_features`

`semicolons`

`backtick_quoted_strings`

`single_quoted_strings`

`indentation_sensitive`

`wrap_blocks_in_braces`

🌈 Example of a `!use` directive 🌈

🦋 Example of a `/!use:re2c ... /` block 🦋

Include file 1 (`definitions.java`)

Include file 2 (`extra_rules.re.inc`)