RE2C(1)

📦

version 4.6 · 13 versions

4.6 · arch rolling, nixpkgs unstable (latest)
4.1 · opensuse-leap 16.0
2.2 · epel 9
4.2 · alpine 3.22
4.3.1 · alpine 3.23
4.5.1 · alpine 3.24, nixos 26.05
4.5.1 · ubuntu stonking
4.3 · ubuntu questing
3.1 · epel 10, fedora 43, fedora 44
3.1 · ubuntu noble
3.0 · debian bookworm, ubuntu jammy
4.4 · ubuntu resolute
4.1 · debian trixie

, in arch rolling, nixpkgs unstable

re2c(1) — generate fast lexical analyzers for C/C++ 💅

💖 NAME 💖

re2c — generate fast lexical analyzers for C/C++ ✨

🌸 SYNOPSIS 🌸

okay bestie, quick note before the syntax: some examples are in C++ (but can be adapted to C) 🫶

re2c [ OPTIONS ] [ WARNINGS ] INPUT

Input can be either a file or - for stdin. That’s it, that’s the vibe check 💅

🦋 INTRODUCTION 🦋

So re2c works as a preprocessor. It reads the input file — usually a program in C/C++, but honestly it can be anything — and looks for blocks of code enclosed in special-form start/end markers. The text outside those blocks gets copied verbatim into the output file. The contents of the blocks are processed by re2c. It translates them to code in C/C++ and outputs the generated code in place of the block. slay ✨

Here’s an example of a small program that checks if a given string contains a decimal number:

// re2c $INPUT -o $OUTPUT -i --case-ranges
#include <assert.h>

int lex(const char* YYCURSOR) {
    /*!re2c
        re2c:yyfill:enable = 0;
        re2c:YYCTYPE = "unsigned char";

        [1-9][0-9]* { return 0; }
        *           { return 1; }
    */
}

int main() {
    assert(lex("1234") == 0);
    return 0;
}

In the output re2c replaced the block in the middle with the generated code:

/* Generated by re2c */
// re2c $INPUT -o $OUTPUT -i --case-ranges
#include <assert.h>

int lex(const char* YYCURSOR) {

{
    unsigned char yych;
    yych = *YYCURSOR;
    switch (yych) {
        case '1' ... '9': goto yy2;
        default: goto yy1;
    }
yy1:
    ++YYCURSOR;
    { return 1; }
yy2:
    yych = *++YYCURSOR;
    switch (yych) {
        case '0' ... '9': goto yy2;
        default: goto yy3;
    }
yy3:
    { return 0; }
}

}

int main() {
    assert(lex("1234") == 0);
    return 0;
}

🎀 BASICS 🎀

A re2c program consists of a sequence of blocks intermixed with code in the target language. A block may contain definitions, configurations, rules, actions and directives in any order 🫶

name = regular-expression ; — a definition binds name to regular-expression. Names may contain alphanumeric characters and underscore. The regular expressions section gives an overview of re2c syntax for regular expressions. Once defined, the name can be used in other regular expressions and in rules. Recursion in named definitions is not allowed, and each name should be defined before it is used. A block inherits named definitions from the global scope. Redefining a name that exists in the current scope is an error 💀
configuration = value ; — a configuration allows one to change re2c behavior and customize the generated code. For a full list of configurations supported by re2c see the configurations section. Depending on a particular configuration, the value can be a keyword, a nonnegative integer number or a one-line string which should be enclosed in double or single quotes unless it consists of alphanumeric characters. A block inherits configurations from the global scope and may redefine them or add new ones. Configurations defined inside of a block affect the whole block, even if they appear at the end of it 👑
regular-expression code — a rule binds regular-expression to its semantic action (a block of code in curly braces, or a block of code that starts with := and ends on a newline followed by any non-whitespace character). If the regular-expression matches, the associated code is executed. If multiple rules match, the longest match takes precedence. If multiple rules match the same string, the earliest one takes precedence. There are two special rules: the default rule * and the end of input rule $. Default rule should always be defined, it has the lowest priority regardless of its place in the block, and it matches any code unit (not necessarily a valid character, see the encoding support section). The end of input rule should be defined if the corresponding method for handling the end of input is used. With start conditions rules have more complex syntax 😭
!action code — an action binds a user-defined block of code to a particular place in the generated finite state machine (in the same way as semantic actions bind code to the final states). See the actions section for a full list of predefined actions ✨
!directive ; — a directive is one of the special predefined statements. Each directive has a unique purpose. See the directives section for details 🧚

🌈 Blocks 🌈

Block start and end markers are either /*!re2c and */, or %{ and %} (both styles are supported). Starting from version 2.2 blocks may have optional names that allow them to be referenced in other blocks. There are different kinds of blocks:

/*!re2c[:<name>] ... */ or %{[:<name>] ... %} — a global block contains definitions, configurations, rules and directives. re2c compiles regular expressions associated with each rule into a deterministic finite automaton, encodes it in the form of conditional jumps in the target language and replaces the block with the generated code. Names and configurations defined in a global block are added to the global scope and become visible to subsequent blocks. At the start of the program the global scope is initialized with command-line options 👑
/*!local:re2c[:<name>] ... */ or %{local[:<name>] ... %} — a local block is like a global block, but the names and configurations in it have local scope (they do not affect other blocks) 🫶
/*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} — a rules block is like a local block, but it does not generate any code by itself, nor does it add any definitions to the global scope — it is meant to be reused in other blocks. This is a way of sharing code (more details in the reusable blocks section). Prior to re2c version 2.2 rules blocks required -r --reusable option 💅
/*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %} — a use block that references a previously defined rules block. If the name is specified, re2c looks for a rules block with this name. Otherwise the most recent rules block is used (either a named or an unnamed one). A use block can add definitions, configurations and rules of its own, which are added to those of the referenced rules block. Prior to re2c version 2.2 use blocks required -r --reusable option ✨
/*!max:re2c[:<name1>[:<name2>...]] ... */ or %{max[:<name1>[:<name2>...]] ... %} — a block that generates YYMAXFILL definition. An optional list of block names specifies which blocks should be included when computing YYMAXFILL value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXFILL <n>), or a global variable for Go (var YYMAXFILL int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXFILL 🌟
/*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */ or %{maxnmatch[:<name1>[:<name2>...]] ... %} — a block that generates YYMAXNMATCH definition (it requires -P --posix-captures option). An optional list of block names specifies which blocks should be included when computing YYMAXNMATCH value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXNMATCH <n>), or a global variable for Go (var YYMAXNMATCH int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXNMATCH 🔥
stags / mtags blocks — the markers go exactly like this, bestie:

/*!stags:re2c[:<name1>[:<name2>...]] ... */,
/*!mtags:re2c[:<name1>[:<name2>...]] ... */ or
%{stags[:<name1>[:<name2>...]] ... %}, %{mtags[:<name1>[:<name2>...]]
... %{

These blocks specify a template piece of code that is expanded for each s-tag/m-tag variable generated by re2c. An optional list of block names specifies which blocks should be included when computing the set of tag variables (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag variable. Configuration separator specifies a piece of code used to join the generated format pieces for different tag variables 💜

svars / mvars blocks — again, the exact marker shapes are:

/*!svars:re2c[:<name1>[:<name2>...]] ... */,
/*!mvars:re2c[:<name1>[:<name2>...]] ... */ or
%{svars[:<name1>[:<name2>...]] ... %}, %{mvars[:<name1>[:<name2>...]]
... %{

These blocks specify a template piece of code that is expanded for each s-tag/m-tag that is either explicitly mentioned by the rules (with --tags option) or implicitly generated by re2c (with --captvars or --posix-captvars options). An optional list of block names specifies which blocks should be included when computing the set of tags (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag. Configuration separator specifies a piece of code used to join the generated format pieces for different tags 🎀

/*!getstate:re2c[:<name1>[:<name2>...]] ... */ or %{getstate[:<name1>[:<name2>...]] ... %} — a block that generates conditional dispatch on the lexer state (it requires --storable-state option). An optional list of block names specifies which blocks should be included in the state dispatch. The default transition goes to the start label of the first block on the list. If the list is empty, all blocks are included, and the default transition goes to the first block in the file that has a start label. This block type is incompatible with the --loop-switch option, as it requires cross-block transitions that are unsupported without goto or function calls 😭
/*!conditions:re2c[:<name1>[:<name2>...]] ... */, /*!types:re2c... */ or %{conditions[:<name1>[:<name2>...]] ... %}, %{types... %} — a whole block whose entire job is generating a condition enumeration (heads up: this one needs the --conditions option, it's not optional bestie 💅). You can hand it an optional list of block names, which tells it exactly which blocks get included when it's computing the set of conditions — and if that list is empty, then congrats, all blocks are included 🎀. By default the generated code is an enumeration called YYCONDTYPE. Want it different? It can be customized with the optional configurations format and separator. The format configuration specifies a template string, and in that string @@{cond} (or just @@ if you're being lazy) gets replaced with the name of each condition, while @@{num} gets replaced with a numeric index of that condition. The separator configuration specifies a piece of code that's used to join the generated format pieces for different conditions. Chef's kiss ✨
/*!include:re2c <file> */ or %{include <file> %} — this block lets you include <file>, and <file> must be a double-quoted file path, no cap. The contents of the file get literally substituted in place of the block, the exact same way #include works in C/C++ 🫶. Bonus: this block can be used together with the --depfile option to generate build system dependencies on the included files. Iconic for build systems.
/*!header:re2c:on*/ or %{header:on %} — this block marks the start of the header file. Everything after it, all the way up to the following header:off block, is processed by re2c and written to the header file specified with the -t --type-header option 🌸.
/*!header:re2c:off*/ or %{header:off %} — this block marks the end of the header file started with the header:on*/ block. Closing the loop, we love to see it 💖.
/*!ignore:re2c ... */ or %{ignore ... %} — a block whose contents are ignored and removed from the output file. Poof. Gone. Never happened 🙈.

✨ CONFIGURATIONS ✨

Okay bestie, here's the full list of every configuration re2c supports. Buckle in, there's a lot — but it's giving organized, so we're fine 🎀

re2c:api, re2c:input — same as the --api option.
re2c:api:sigil — specifies the marker ("sigil") used for argument placeholders in the API primitives. The default is @@. A placeholder starts with the sigil, followed by the argument name in curly braces. So like, if the sigil is set to $, then placeholders will have the form ${name} 👑. Single-argument APIs may use shorthand notation without the name in braces. This option can be overridden by options for individual API primitives — for example, re2c:YYFILL@len for YYFILL.
re2c:api:style — specifies the API style. Possible values are functions (the default for C) and free-form (the default for Go and Rust). In the functions style, API primitives are generated with an argument list in parentheses following the name of the primitive. The arguments are provided only for autogenerated parameters (like the number of characters passed to YYFILL), but not for the general lexer context — so the primitives behave more like macros in C/C++ or closures in Go and Rust 🦋. In the free-form style, API primitives don't have a fixed form at all: they should be defined as strings containing free-form pieces of code with interpolated variables of the form @@{var} or @@ (those correspond to arguments in the function-like style). This configuration may be overridden for individual API primitives — see for example the re2c:YYFILL:naked configuration for YYFILL.
re2c:bit-vectors, re2c:flags:bit-vectors, re2c:flags:b — same as the --bit-vectors option, but it can be configured on a per-block basis.
re2c:captures, re2c:leftmost-captures — same as the --leftmost-captures option, but it can be configured on a per-block basis.
re2c:captvars, re2c:leftmost-captvars — same as the --leftmost-captvars option, but it can be configured on a per-block basis.
re2c:case-insensitive, re2c:flags:case-insensitive — same as the --case-insensitive option, but it can be configured on a per-block basis.
re2c:case-inverted, re2c:flags:case-inverted — same as the --case-inverted option, but it can be configured on a per-block basis.
re2c:case-ranges, re2c:flags:case-ranges — same as the --case-ranges option, but it can be configured on a per-block basis.
re2c:computed-gotos, re2c:flags:computed-gotos, re2c:flags:g — same as the --computed-gotos option, but it can be configured on a per-block basis.
re2c:computed-gotos:relative, re2c:cgoto:relative — same as the --computed-gotos-relative option, but it can be configured on a per-block basis.
re2c:computed-gotos:threshold, re2c:cgoto:threshold — if computed goto is used, this configuration specifies the complexity threshold that triggers the generation of jump tables instead of nested if statements and bitmaps. The default value is 9 🔥.
re2c:cond:abort — if set to a positive integer value, the default case in the generated condition dispatch aborts program execution. Dramatic, but make it intentional please 💀.
re2c:cond:goto — specifies a piece of code used for the autogenerated shortcut rules :=> in conditions. The default is goto @@;. The @@ placeholder is substituted with the condition name (see the configurations re2c:api:sigil and re2c:cond:goto@cond).
re2c:cond:goto@cond — specifies the sigil used for argument substitution in the re2c:cond:goto definition. The default value is @@. It overrides the more generic re2c:api:sigil configuration.
re2c:cond:divider — defines the divider for condition blocks. The default value is /* *********************************** */. Placeholders are substituted with the condition name (see re2c:api;sigil and re2c:cond:divider@cond).
re2c:cond:divider@cond — specifies the sigil used for argument substitution in the re2c:cond:divider definition. The default is @@. It overrides the more generic re2c:api:sigil configuration.
re2c:cond:prefix, re2c:condprefix — specifies the prefix used for condition labels. The default is yyc_ 🎀.
re2c:cond:enumprefix, re2c:condenumprefix — specifies the prefix used for condition identifiers. The default is yyc.
re2c:debug-output, re2c:flags:debug-output, re2c:flags:d — same as the --debug-output option, but it can be configured on a per-block basis.
re2c:empty-class, re2c:flags:empty-class — same as the --empty-class option, but it can be configured on a per-block basis.
re2c:encoding:ebcdic, re2c:flags:ecb, re2c:flags:e — same as the --ebcdic option, but it can be configured on a per-block basis.
re2c:encoding:ucs2, re2c:flags:wide-chars, re2c:flags:w — same as the --ucs2 option, but it can be configured on a per-block basis.
re2c:encoding:utf8, re2c:flags:utf-8, re2c:flags:8 — same as the --utf8 option, but it can be configured on a per-block basis.
re2c:encoding:utf16, re2c:flags:utf-16, re2c:flags:x — same as the --utf16 option, but it can be configured on a per-block basis.
re2c:encoding:utf32, re2c:flags:unicode, re2c:flags:u — same as the --utf32 option, but it can be configured on a per-block basis.
re2c:encoding-policy, re2c:flags:encoding-policy — same as the --encoding-policy option, but it can be configured on a per-block basis.
re2c:eof — specifies the sentinel symbol used with the end-of-input rule $. The default value is -1 (meaning the $ rule is not used). Other possible values include all valid code units. Real talk: only decimal numbers are recognized here, so don't get cute with hex 🙈.
re2c:header, re2c:flags:type-header, re2c:flags:t — specifies the name of the generated header file, relative to the directory of the output file. Same as the --header option, except that the file path is relative 🌸.
re2c:indent:string — specifies the string used for indentation. The default is a single tab character "\t". The indent string should contain whitespace characters only. And if you want to disable indentation entirely, just set this configuration to an empty string ✨.
re2c:indent:top — specifies the minimum amount of indentation to use. The default value is zero. The value should be a non-negative integer number.
re2c:invert-captures — same as the --invert-captures option, but it can be configured on a per-block basis.
re2c:label:prefix, re2c:labelprefix — specifies the prefix used for DFA state labels. The default is yy 💅.
re2c:label:start, re2c:startlabel — controls the generation of a block start label. The default value is zero, which means the start label is generated only if it is used. An integer value greater than zero forces the generation of the start label even if it's unused by the lexer 👑. A string value also forces start label generation and sets the label name to that specified string. Heads up: this configuration applies only to the current block — it gets reset to default for the next block.
re2c:label:yyFillLabel — specifies the prefix of YYFILL labels used with re2c:eof and in storable state mode.
re2c:label:yyloop — specifies the name of the label marking the start of the lexer loop with the --loop-switch option. The default is yyloop.
re2c:label:yyNext — specifies the name of the optional label that follows the YYGETSTATE switch in storable state mode (enabled with re2c:state:nextlabel). The default is yyNext.
re2c:lookahead, re2c:flags:lookahead — deprecated ⚠️ (see the deprecated --no-lookahead option). Not the vibe anymore, don't use it.
re2c:monadic — if set to non-zero, the generated lexer will use monadic notation (this configuration is specific to Haskell) 🧚.
re2c:nested-ifs, re2c:flags:nested-ifs, re2c:flags:s — same as the --nested-ifs option, but it can be configured on a per-block basis.
re2c:posix-captures, re2c:flags:posix-captures, re2c:flags:P — same as the --posix-captures option, but it can be configured on a per-block basis.
re2c:posix-captvars — same as the --posix-captvars option, but it can be configured on a per-block basis.
re2c:tags, re2c:flags:tags, re2c:flags:T — same as the --tags option, but it can be configured on a per-block basis.
re2c:tags:expression — this one sets the expression used for tag variables ✨ By default re2c generates expressions of the form yyt<N>. That might be inconvenient, like, for example if tag variables are defined as fields in a struct 🙈 All occurrences of @@{tag} or @@ are replaced with the actual tag name. For example, re2c:tags:expression = "s.@@"; gives you expressions of the form s.yyt<N> in the generated code. See also re2c:api:sigil configuration 💅
re2c:tags:negative — specifies the constant expression used for a negative tag value (typically this would be -1 if tags are integer offsets in the input string, or a null pointer if they're pointers) 🫶
re2c:tags:prefix — specifies the prefix for tag variable names. Default is yyt ✨
re2c:sentinel — specifies the sentinel symbol used for the end-of-input checks (like, when bounds checks are disabled with re2c:yyfill:enable = 0; and re2c:eof is not set) 👀 Real talk: this configuration does not affect code generation — its whole purpose is to verify that the sentinel isn't allowed in the middle of a rule, and to ensure the lexer won't read past the end of buffer. Default value is -1 (in that case re2c assumes the sentinel is zero, which is the most common case). Only decimal numbers are recognized, so don't get cute 💀
re2c:state:abort — if set to a positive integer value, the default case in the generated state dispatch aborts program execution, and an explicit -1 case contains transition to the start of the block 🔥
re2c:state:nextlabel — controls whether the YYGETSTATE switch is followed by a yyNext label (the default value is zero, which corresponds to no label). Alternatively you can use re2c:label:start to generate a specific start label, or an explicit getstate block to generate the YYGETSTATE switch separately from the lexer block 💅
re2c:unsafe, re2c:flags:unsafe — same as the --no-unsafe option, but configurable on a per-block basis 🎀 If set to zero, it suppresses the generation of unsafe wrappers around YYPEEK. Default is non-zero (wrappers are generated). Heads up: this configuration is specific to Rust 🦋
re2c:YYBACKUP, re2c:define:YYBACKUP — defines generic API primitive YYBACKUP ✨
re2c:YYBACKUPCTX, re2c:define:YYBACKUPCTX — defines generic API primitive YYBACKUPCTX ✨
re2c:YYCONDTYPE, re2c:define:YYCONDTYPE — defines API primitive YYCONDTYPE ✨
re2c:YYCTYPE, re2c:define:YYCTYPE — defines API primitive YYCTYPE ✨
re2c:YYCTXMARKER, re2c:define:YYCTXMARKER — defines API primitive YYCTXMARKER ✨
re2c:YYCURSOR, re2c:define:YYCURSOR — defines API primitive YYCURSOR ✨
re2c:YYDEBUG, re2c:define:YYDEBUG — defines API primitive YYDEBUG ✨
re2c:YYFILL, re2c:define:YYFILL — defines API primitive YYFILL ✨
re2c:YYFILL@len, re2c:define:YYFILL@len — specifies the sigil used for argument substitution in the YYFILL definition. Defaults to @@. Overrides the more generic re2c:api:sigil configuration 💅
re2c:YYFILL:naked, re2c:define:YYFILL:naked — overrides the more generic re2c:api:style configuration for YYFILL. A zero value corresponds to free-form API style 🫶
re2c:YYFN — defines API primitive YYFN ✨
re2c:YYINPUT — defines API primitive YYINPUT ✨
re2c:YYGETCOND, re2c:define:YYGETCONDITION — defines API primitive YYGETCOND ✨
re2c:YYGETCOND:naked, re2c:define:YYGETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYGETCOND. Zero value corresponds to free-form API style 🫶
re2c:YYGETSTATE, re2c:define:YYGETSTATE — defines API primitive YYGETSTATE ✨
re2c:YYGETSTATE:naked, re2c:define:YYGETSTATE:naked — overrides the more generic re2c:api:style configuration for YYGETSTATE. Zero value corresponds to free-form API style 🫶
re2c:YYGETACCEPT, re2c:define:YYGETACCEPT — defines API primitive YYGETACCEPT ✨
re2c:YYLESSTHAN, re2c:define:YYLESSTHAN — defines generic API primitive YYLESSTHAN ✨
re2c:YYLIMIT, re2c:define:YYLIMIT — defines API primitive YYLIMIT ✨
re2c:YYMARKER, re2c:define:YYMARKER — defines API primitive YYMARKER ✨
re2c:YYMTAGN, re2c:define:YYMTAGN — defines generic API primitive YYMTAGN ✨
re2c:YYMTAGP, re2c:define:YYMTAGP — defines generic API primitive YYMTAGP ✨
re2c:YYPEEK, re2c:define:YYPEEK — defines generic API primitive YYPEEK ✨
re2c:YYRESTORE, re2c:define:YYRESTORE — defines generic API primitive YYRESTORE ✨
re2c:YYRESTORECTX, re2c:define:YYRESTORECTX — defines generic API primitive YYRESTORECTX ✨
re2c:YYRESTORETAG, re2c:define:YYRESTORETAG — defines generic API primitive YYRESTORETAG ✨
re2c:YYSETCOND, re2c:define:YYSETCONDITION — defines API primitive YYSETCOND ✨
re2c:YYSETCOND@cond, re2c:define:YYSETCONDITION@cond — specifies the sigil used for argument substitution in the YYSETCOND definition. Default value is @@. Overrides the more generic re2c:api:sigil configuration 💅
re2c:YYSETCOND:naked, re2c:define:YYSETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYSETCOND. Zero value corresponds to free-form API style 🫶
re2c:YYSETSTATE, re2c:define:YYSETSTATE — defines API primitive YYSETSTATE ✨
re2c:YYSETSTATE@state, re2c:define:YYSETSTATE@state — specifies the sigil used for argument substitution in the YYSETSTATE definition. Default value is @@. Overrides the more generic re2c:api:sigil configuration 💅
re2c:YYSETSTATE:naked, re2c:define:YYSETSTATE:naked — overrides the more generic re2c:api:style configuration for YYSETSTATE. Zero value corresponds to free-form API style 🫶
re2c:YYSETACCEPT, re2c:define:YYSETACCEPT — defines API primitive YYSETACCEPT ✨
re2c:YYSKIP, re2c:define:YYSKIP — defines generic API primitive YYSKIP ✨
re2c:YYSHIFT, re2c:define:YYSHIFT — defines generic API primitive YYSHIFT ✨
re2c:YYCOPYMTAG, re2c:define:YYCOPYMTAG — defines generic API primitive YYCOPYMTAG ✨
re2c:YYCOPYSTAG, re2c:define:YYCOPYSTAG — defines generic API primitive YYCOPYSTAG ✨
re2c:YYSHIFTMTAG, re2c:define:YYSHIFTMTAG — defines generic API primitive YYSHIFTMTAG ✨
re2c:YYSHIFTSTAG, re2c:define:YYSHIFTSTAG — defines generic API primitive YYSHIFTSTAG ✨
re2c:YYSTAGN, re2c:define:YYSTAGN — defines generic API primitive YYSTAGN ✨
re2c:YYSTAGP, re2c:define:YYSTAGP — defines generic API primitive YYSTAGP ✨
re2c:yyaccept, re2c:variable:yyaccept — defines API primitive yyaccept ✨
re2c:yybm, re2c:variable:yybm — defines API primitive yybm ✨
re2c:yybm:hex, re2c:variable:yybm:hex — if set to nonzero, bitmaps for the --bit-vectors option are generated in hexadecimal format 🎀 Default is zero (bitmaps are in decimal format) 💅
re2c:yych, re2c:variable:yych — defines API primitive yych ✨
re2c:yych:emit, re2c:variable:yych:emit — if set to zero, the yych definition is not generated. Default is non-zero 🫶
re2c:yych:conversion, re2c:variable:yych:conversion — if set to non-zero, re2c automatically generates a conversion to YYCTYPE every time yych is read. Default is zero (no conversion) 👀
re2c:yych:literals, re2c:variable:yych:literals — specifies the form of literals that yych is matched against. Possible values are: char (character literals in single quotes, non-printable ones use escape sequences that start with backslash), hex (hexadecimal integers) and char_or_hex (a mixture of both — character literals for printable characters and hexadecimal integers for others) 🌈
re2c:yyctable, re2c:variable:yyctable — defines API primitive yyctable ✨
re2c:yynmatch, re2c:variable:yynmatch — defines API primitive yynmatch ✨
re2c:yypmatch, re2c:variable:yypmatch — defines API primitive yypmatch ✨
re2c:yytarget, re2c:variable:yytarget — defines API primitive yytarget ✨
re2c:yystable, re2c:variable:yystable — deprecated 💀 like, don't use it bestie
re2c:yystate, re2c:variable:yystate — defines API primitive yystate ✨
re2c:yyfill, re2c:variable:yyfill — defines API primitive yyfill ✨
re2c:yyfill:check — if set to zero, suppresses the generation of the pre-YYFILL check for the number of input characters (that's the YYLESSTHAN definition in generic API, and the YYLIMIT-based comparison in C pointer API). Default is non-zero (generate the check) 🎀
re2c:yyfill:enable — if set to zero, suppresses the generation of YYFILL (together with the check) 🙈 Use this when the whole input fits into one piece of memory (so there's no need for buffering) and the end-of-input checks don't rely on the YYFILL checks (e.g. if a sentinel character is used). Use warnings (the -W option) and re2c:sentinel configuration to verify that the generated lexer cannot read past the end of input ⚠️ Default is non-zero (YYFILL is enabled) 🫶
re2c:yyfill:parameter — if set to zero, suppresses the generation of the parameter passed to YYFILL. That parameter is the minimum number of characters that must be supplied. Defaults to non-zero (the parameter is generated). This configuration can be overridden with re2c:YYFILL:naked or re2c:api:style 💅
re2c:yyfn:sep — specifies the separator used in YYFN elements (defaults to semicolon) ✨
re2c:yyfn:throw — specifies exceptions thrown by the YYFN function (defaults to empty, which means no exceptions) 🦋

✨ Regular expressions ✨

Okay so re2c uses the following syntax for regular expressions — memorize the vibes, bestie, this is the whole grammar 🧚

"foo" — case-sensitive string literal 💅
'foo' — case-insensitive string literal ✨
[a-xyz], [^a-xyz] — character class (possibly negated) 🎀
. — any character except newline 🌸
R \ S — difference of character classes R and S 🔥
R* — zero or more occurrences of R ⭐
R+ — one or more occurrences of R ⭐
R? — optional R 🦋
R{n} — repetition of R exactly n times 👑
R{n,} — repetition of R at least n times 👑
R{n,m} — repetition of R from n to m times 👑
(R) — just R; parentheses are used to override precedence. If submatch extraction is enabled, (R) is a capturing or a non-capturing group depending on the --invert-captures option 🙈
(!R) — if submatch extraction is enabled, (!R) is a non-capturing or a capturing group depending on the --invert-captures option 🙈

okay so picking up the regex-core list exactly where we left off, bestie — these are the next little operators and they are honestly iconic:

R S — concatenation: R followed by S. 💖
R | S — alternative: R or S. ✨
R / S — lookahead: R followed by S, but S is not consumed. 👀
name — a regular expression defined as name (or the literal string "name" in Flex compatibility mode). 🏷️
{name} — a regular expression defined as name in Flex compatibility mode. 🦋
@stag — an s-tag: saves the last input position at which @stag matches in a variable named stag. 📍
#mtag — an m-tag: saves all input positions at which #mtag matches in a variable named mtag. 📌
$ — end of input. 🛑

Character classes and string literals may contain these escape sequences, and yes they are all invited to the party: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and \Uhhhhhhhh. 🌈

✨ ACTIONS ✨

Here is a list of predefined actions supported by re2c, and they are lowkey iconic:

!entry code — the entry action binds a user-defined block of code to the start state of the current finite state machine. If start conditions are used, the entry action can be set individually for each condition. This action may be used to perform initialization, e.g. to save the start location of a lexeme. 🚪
!pre_rule code — the pre-rule action prepends a user-defined block of code to semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to factor out the common part of all semantic actions (e.g. saving the end location of a lexeme). 🧵
!post_rule code — the post-rule action appends a user-defined block of code to semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to emit trap statements that guard against unintended control flow. 🚧

🎀 DIRECTIVES 🎀

Here is a full list of directives supported by re2c:

!use:name ; — an in-block use directive that merges a previously defined rules block with the specified name into the current block. Named definitions, configurations and rules of the referenced block are added to the current ones. Conflicts between overlapping rules and configurations are resolved in the usual way: the first rule takes priority, and the latest configuration overrides the preceding ones. One exception is the special rules *, $ and <!> for which a block-local definition always takes priority. A use directive can be placed anywhere inside of a block, and multiple use directives are allowed. 🔗
!include file ; — this directive is the same as include block: it inserts file contents verbatim in place of the directive. 📄

🌈 PROGRAM INTERFACE 🌈

The generated code interfaces with the outer program with the help of primitives, collectively referred to as the API. Which primitives should be defined for a particular program depends on multiple factors, including the complexity of regular expressions, input representation, buffering and the use of various features.

All the necessary primitives should be defined by the user in the form of macros, functions, variables or any other suitable form that makes the generated code syntactically and semantically correct.

re2c does not (and cannot) check the definitions, so if anything is missing or defined incorrectly, the generated program may have compile-time or run-time errors. 😭

This manual provides examples of API definitions in the most common cases.

re2c has three API flavors that define the core set of primitives used by a program:

💖 Simple API 💖

This is the default API for C/C++ backend. It consists of primitives YYCURSOR, YYMARKER, YYCTXMARKER and YYLIMIT, which should be defined as pointers of type YYCTYPE*.

🎀 Record API 🎀

(added in version 4.0) Record API is useful in cases when lexer state must be stored in a struct. It is enabled with --api record option or re2c:api = record configuration. This API consists of a variable yyrecord (the name can be overridden with re2c:yyrecord) that should be defined as a struct with fields yycursor, yymarker, yyctxmarker, yylimit (only the fields used by the generated code need to be defined, and their names can be configured).

🦋 Generic API 🦋

(added in version 0.14) This is the most flexible API. It is enabled with --api generic option or re2c:api = generic configuration. This API contains primitives for generic operations: YYPEEK, YYSKIP, YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE, YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG, YYLESSTHAN. YYEND. Generic API supports two styles that determine the form in which the primitives should be defined:

Free-form — Free-form style is enabled with configuration re2c:api:style = free-form. In this style interface primitives should be defined as free-form pieces of code with interpolated variables of the form @@{var} or optionally just @@ if there is a single variable. The set of variables is specific to each primitive. Here's how free-form generic API can be defined in terms of pointers cursor, limit, marker and ctxmarker:

/*!re2c
  re2c:YYPEEK       = "*cursor";
  re2c:YYSKIP       = "++cursor;";
  re2c:YYBACKUP     = "marker = cursor;";
  re2c:YYRESTORE    = "cursor = marker;";
  re2c:YYBACKUPCTX  = "ctxmarker = cursor;";
  re2c:YYRESTORECTX = "cursor = ctxmarker;";
  re2c:YYRESTORETAG = "cursor = ${tag};";
  re2c:YYLESSTHAN   = "limit - cursor < @@{len}";
  re2c:YYEND        = "limit == cursor";
  re2c:YYSTAGP      = "@@{tag} = cursor;";
  re2c:YYSTAGN      = "@@{tag} = NULL;";
  re2c:YYSHIFT      = "cursor += @@{shift};";
  re2c:YYSHIFTSTAG  = "@@{tag} += @@{shift};";
*/

Function-like — Historically function-like style is the default one. It also can be enabled with configuration re2c:api:style = functions. In this style primitives should be defined as functions or macros with parentheses, accepting the necessary arguments. Here's how function-like generic API can be defined in terms of pointers cursor, limit, marker and ctxmarker using preprocessor macros:

#define  YYPEEK()                 *cursor
#define  YYSKIP()                 ++cursor
#define  YYBACKUP()               marker = cursor
#define  YYRESTORE()              cursor = marker
#define  YYBACKUPCTX()            ctxmarker = cursor
#define  YYRESTORECTX()           cursor = ctxmarker
#define  YYRESTORETAG(tag)        cursor = tag
#define  YYLESSTHAN(len)          limit - cursor < len
#define  YYEND()                  limit == cursor
#define  YYSTAGP(tag)             tag = cursor
#define  YYSTAGN(tag)             tag = NULL
#define  YYSHIFT(shift)           cursor += shift
#define  YYSHIFTSTAG(tag, shift)  tag += shift

Here is a full list of API primitives that may be used by the generated code in order to interface with the outer program.

YYCTYPE — the type of the input characters (code units). For ASCII, EBCDIC and UTF-8 encodings it should be 1-byte unsigned integer. For UTF-16 or UCS-2 it should be 2-byte unsigned integer. For UTF-32 it should be 4-byte unsigned integer. 🔤
YYCURSOR — an l-value that stores the current input position (a pointer or an integer offset in YYINPUT). Initially YYCURSOR should point to the first input character, and later it is advanced by the generated code. When a rule matches, YYCURSOR position is the one after the last matched character. ➡️
YYLIMIT — an r-value that stores the end of input position (a pointer or an integer offset in YYINPUT). Initially YYLIMIT should point to the position after the last available input character. It is not changed by the generated code. The lexer compares YYCURSOR to YYLIMIT in order to determine if there are enough input characters left. 🛑
YYMARKER — an l-value that stores the position of the latest matched rule (a pointer or an integer offset in YYINPUT). It is used to restore the YYCURSOR position if the longer match fails and the lexer needs to rollback. Initialization is not needed. 📍
YYCTXMARKER — an l-value that stores the position of the trailing context (a pointer or an integer offset in YYINPUT). No initialization is needed. YYCTXMARKER is needed only if the lookahead operator / is used. 👀
YYFILL — a generic API primitive with one variable len. YYFILL should provide at least len more input characters or fail. If re2c:eof is used, then len is always 1 and YYFILL should always return to the calling function; zero return value indicates success. If re2c:eof is not used, then YYFILL return value is ignored and it should not return on failure. The maximum value of len is YYMAXFILL. 🧃
YYFN — a primitive that defines function prototype in --recursive-functions code model. Its value should be an array of one or more strings, where each string contains two or three components separated by the string specified in re2c:fn:sep configuration (typically a semicolon). The first array element defines function name and return type (empty for a void function). Subsequent elements define function arguments: first, the expression for the argument used in function body (usually just a name); second, argument type; third, an optional formal parameter (it defaults to the first component - usually both the argument and the parameter are the same identifier). 🧩
YYINPUT — an r-value that stores the current input character sequence (string, buffer, etc.). 📥
YYMAXFILL — an integral constant equal to the maximum value of the argument to YYFILL. It can be generated with a max block. 🔢
YYLESSTHAN — bestie this one's a generic API primitive with exactly one variable, len. Define it as an r-value of boolean type that's true if and only if there are less than len input characters left. That's it, that's the whole vibe check. ✨
YYEND — a generic API primitive with no variables, zero, nada. It should be an r-value of boolean type equal to true if and only if the logical end of input has been reached (and excluding any padding or sentinel symbols, okay? don't count those). YYEND is what implements the $ symbol in regular expressions. It's lowkey different from YYLESSTHAN, which is the one that makes sure the lexer won't read past the end of buffer. Two different jobs, both iconic. 💅
YYPEEK — generic API primitive, no variables. Should be an r-value of type YYCTYPE equal to the character at the current input position. It just... looks at the character. Peek, queen. 👑
YYSKIP — generic API primitive that should advance the current input position by one code unit. Moving on up, no cap. 🦋
YYBACKUP — generic API primitive that should save the current input position (so you can restore it later with YYRESTORE). It's giving "save your progress before the boss fight". 🎀
YYRESTORE — generic API primitive that should restore the current input position to whatever value YYBACKUP saved. Rewind, slay. ⭐
YYBACKUPCTX — generic API primitive that should save the current input position as the position of the trailing context (to be restored later with YYRESTORECTX). Context queen. 🌸
YYRESTORECTX — generic API primitive that should restore the trailing context position that was saved with YYBACKUPCTX. 💖
YYRESTORETAG — generic API primitive with one variable, tag, that should restore the trailing context position to the value of tag. 🧚
YYSTAGP — generic API primitive with one variable, tag, where tag can be a pointer or an offset in YYINPUT (peep the submatch extraction section for the deets, iykyk). YYSTAGP should set tag to the current input position. 📍
YYSTAGN — generic API primitive with one variable, tag, where tag can be a pointer or an offset in YYINPUT (again, see the submatch extraction section for details). YYSTAGN should set tag to a value that represents a non-existent input position. Not me setting a tag to ✨nothingness✨. 🙈
YYMTAGP — generic API primitive with one variable, tag. YYMTAGP should append the current position to the submatch history of tag (the submatch extraction section has the full lore on this one). 🍓
YYMTAGN — generic API primitive with one variable, tag. YYMTAGN should append a value that represents a non-existent input position to the submatch history of tag (submatch extraction section, go read it bestie). 💀
YYSHIFT — generic API primitive with one variable, shift, that should shift the current input position by shift characters. And yes, the shift value may be negative — we go backwards sometimes, it's called range. 🔥
YYCOPYSTAG — generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side s-tag variable rhs to the left-hand-side s-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs. S-tag to s-tag, copy paste, done. 💜
YYCOPYMTAG — generic API primitive with two variables, lhs and rhs, that should copy the right-hand-side m-tag variable rhs to the left-hand-side m-tag variable lhs. For most languages this primitive has a default definition that assigns lhs to rhs. Same energy, different tag type. ✨
YYSHIFTSTAG — generic API primitive with two variables, tag and shift, that should shift tag by shift code units (the shift value may be negative, we've been over this, we love a signed value 💅).
YYSHIFTMTAG — generic API primitive with two variables, tag and shift, that should shift the latest value in the history of tag by shift code units (shift value may be negative, yes, again). 🌈
YYMAXNMATCH — an integral constant equal to the maximal number of POSIX capturing groups in a rule. It gets generated with a maxnmatch block. 🎀
YYCONDTYPE — the type of the condition enum. It can be generated either with a conditions block or with the --header option. Two paths, same queen. 👑
YYGETACCEPT — a primitive with one variable, var, that stores a numeric selector of the accepted rule. For most languages this primitive has a default definition that reads from var. ✨
YYSETACCEPT — a primitive with two variables: var (an l-value that stores the numeric selector of the accepted rule) and val (the value of the selector). For most languages this primitive has a default definition that assigns var to val. 💖
YYGETCOND — an r-value of type YYCONDTYPE that's equal to the current condition identifier. Condition check, slay. 🦋
YYSETCOND — a primitive with one variable, cond, that should set the current condition identifier to cond. 🎀
YYGETSTATE — an r-value of integer type equal to the current lexer state. It should be initialized to -1. Don't forget the -1, that's the whole point bestie. 🙈
YYSETSTATE — a primitive with one variable, state, that should set the current lexer state to state. ⭐
YYDEBUG — okay so this primitive is generated only with the -d, --debug-output option. Its whole purpose is adding logging to the generated code (a typical YYDEBUG definition is a print statement — very "hello from inside the DFA"). YYDEBUG statements get generated in every state and have two variables: state (either a DFA state index or -1) and symbol (the current input symbol). 🔍
yyaccept — an l-value of unsigned integral type that stores the number of the latest matched rule. A user definition is necessary only with the --storable-state option. 💀
yybm — a table containing compressed bitmaps for up to 8 transitions (used with the --bitmaps option). The table contains 256 elements and is indexed by 1-byte code units. Each 8-bit element combines boolean values for up to 8 transitions. The k-th bit of the n-th element is true iff the n-th code unit is in the range of the k-th transition. The idea of this bitmap is to replace many if branches or switch cases with one single check of one single bit in the table. Efficiency ate, no crumbs. 🔥
yych — an l-value of type YYCTYPE that stores the current input character. A user definition is necessary only with the -f --storable-state option. 💖
yyctable — a jump table generated for the initial condition dispatch (enabled with the combination of the --conditions and --computed-gotos options). 🌸
yyfill — an l-value that stores the result of a YYFILL call (this may be necessary for pure functional languages, where YYFILL is a monadic function with a complex return value). Functional girlies rise. 🧚
yynmatch — an l-value of unsigned integral type that stores the number of POSIX capturing groups in the matched rule. Used only with the -P --posix-captures option. 👑
yypmatch — an array of l-values used to hold the tag values corresponding to the capturing parentheses in the matching rule. The array length must be at least yynmatch * 2 (usually YYMAXNMATCH * 2 is a good choice). Used only with the -P --posix-captures option. 🎀
yystable — Deprecated. That's it. That's the whole entry. She's retired, respect her legacy. 💀
yystate — an l-value used with the --loop-switch option to store the current DFA state. 🔄
yytarget — a jump table that contains jump targets (label addresses) for all transitions from a state. This table is local to each state. Generation of yytarget tables is enabled with the --computed-gotos option. ⭐

✨ Options ✨

Okay so before we dive in: some of these options have corresponding configurations, and others are global and cannot be changed after re2c starts reading the input file — like, once it's in, it's in. Debug options generally require building re2c in debug configuration. And internal options are the ones that are useful for experimenting with the algorithms used in re2c (the science experiments, if you will 🧪).

-?, --help, -h — show the help message. When in doubt, ask for help. 💅
--api <simple | record | generic> — specifies the API the generated code uses to interface with user-defined code. Three flavors, all valid:
- simple — use this in simple cases when there's no need for buffer refilling and storing lexer state. Baby steps, we love her. 🎀
- record — use this when the lexer state needs to be stored in a record (struct, class, etc.). Very organized of you. 🦋
- generic — use this in complex cases when the other two APIs are not flexible enough. For the girlies doing the absolute most. 👑
--bit-vectors, -b — optimize conditional jumps using bit masks. This option implies --nested-ifs. Bit manipulation is lowkey hot, sorry not sorry. 🔥
--captures, --leftmost-captures — enable submatch extraction with leftmost greedy capturing groups. The result is collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule. Yes, that's exactly what the manpage says. Copying it character for character because I would never mess with your capture groups. 💖
--captvars, --leftmost-captvars — enable submatch extraction with leftmost greedy capturing groups, but this time the result is collected into variables yytl<k>, yytr<k> for the k-th capturing group. Variables instead of an array — pick your fighter. ✨
--case-insensitive — treat single-quoted and double-quoted strings as case-insensitive. Everyone's equal, we don't care about case here. 🌈
--case-inverted — invert the meaning of single-quoted and double-quoted strings: treat single-quoted strings as case-sensitive and double-quoted strings as case-insensitive. Yes, it flips them. Yes, that's the whole point. 🙈
--case-ranges — collapse consecutive cases in a switch statement into a range of the form low ... high. This syntax is a C/C++ language extension that's supported by compilers like GCC, Clang and Tcc. The main advantage over using single cases is smaller generated code and faster generation time — although for some compilers like Tcc it also results in smaller binary size. Smaller code AND smaller binary? Obsessed. 💅
--computed-gotos, -g — optimize conditional jumps using the non-standard "computed goto" extension (which must be supported by the compiler, so check before you commit). re2c generates jump tables only in complex cases with a lot of conditional branches. The complexity threshold can be configured with the cgoto:threshold configuration. Relative offsets can be enabled with the cgoto:relative configuration. This option implies --bit-vectors. 🚀
--computed-gotos-relative — similar to --computed-gotos but generates relative offsets for jump tables instead (which, again, must be supported by the compiler). This option implies --computed-gotos. Nested implications, we love a chain. ⭐
--conditions, --start-conditions, -c — enable support of Flex-like "conditions": multiple interrelated lexers within one block. This is an alternative to manually specifying different re2c blocks connected with goto or function calls. Multiple lexers, one block, main character energy. 👑
--depfile FILE — writes dependency info into FILE as a Makefile rule shaped like <output-file> : <input-file> [include-file ...]. The whole point? Tracking build deps when include blocks/directives are in play, so touching an include file actually triggers a regen of the output file ✨. Heads up tho: this one depends on the --output option.
--ebcdic, --ecb, -e — makes a lexer that reads EBCDIC-encoded input 🫶. re2c assumes the character range is 0 -- 0xFF and that character size is 1 byte.
--empty-class <match-empty | match-none | error> — sets how re2c treats empty character classes 🎀:
- match-empty (the default) — empty class matches empty input. Yeah it's illogical, but it's backwards-compatible, so 💀.
- match-none — empty class always fails to match.
- error — empty class raises a compilation error.
--encoding-policy <fail | substitute | ignore> — sets how re2c treats Unicode surrogates 🌈:
- fail — re2c aborts with an error when a surrogate is encountered.
- substitute — re2c silently replaces surrogates with the error code point 0xFFFD.
- ignore (the default) — re2c treats surrogates as normal code points.

The Unicode standard says standalone surrogates are invalid, but real-world libraries and programs behave in different ways, so, y'know, here we are 🙈.

--flex-syntax, -F — partial support for Flex syntax, bestie. In this mode named definitions don't need the equal sign and the terminating semicolon, and when they're used they must be surrounded with curly braces. Names without curly braces are treated as double-quoted strings ✨.
--goto-label — uses the "goto/label" code model: the DFA gets encoded as labeled code blocks connected with goto transitions across blocks. Only supported for languages that have a goto statement, iykyk 💅.
--header, --type-header, -t HEADER — generates a HEADER file 🎀. What goes in it can be specified using the special blocks header:on and header:off. And if conditions are used, the generated header gets a condition enum automatically appended to it — unless there's an explicit conditions block, of course 👑.
-I PATH — adds PATH to the list of locations searched when looking for include files 🦋. Slaps in combination with an include block or directive: re2c looks for FILE in the directory of the parent file and in the include locations specified with the -I option.
--input <default | custom> — deprecated alias for --api 💀. Option default corresponds to simple (it is indeed the default for most backends, but not for all). Option custom corresponds to generic.
--input-encoding <ascii | utf8> — specifies how re2c parses regular expressions 🌸:
- ascii (the default) — re2c handles input as ASCII-encoded: any sequence of code units is a sequence of standalone 1-byte characters.
- utf8 — re2c handles input as UTF8-encoded and recognizes multibyte characters.
--invert-captures — flips the meaning of capturing and non-capturing groups 🔄. By default (...) is capturing and (! ...) is non-capturing. With this option (! ...) is capturing and (...) is non-capturing. Gaslight gatekeep girlboss, honestly.
--lang <none | c | d | go | haskell | java | js | ocaml | python | rust | swift | v | zig> — specifies the target language 💖. Supported languages are C, D, Go, Haskell, Java, JS, OCaml, Python, Rust, Swift, V, Zig (more languages can be added via user-defined syntax files, see the --syntax option). Option none disables default syntax configs, so that the target language is undefined.
--location-format <gnu | msvc> — specifies the location format in messages 📍:
- gnu — locations are printed as 'filename:line:column: ...'.
- msvc — locations are printed as 'filename(line,column) ...'.

The default is gnu.

--loop-switch — uses the "loop/switch" code model: the DFA gets encoded as a loop over a switch statement, where individual states are switch cases 🔁. State lives in a variable called yystate. Transitions between states update yystate to the case label of the destination state and continue execution to the head of the loop.
--computed-continue — requires the --loop-switch code model 🫶. It uses computed continue statements to select the next switch case directly, which enables jump-table optimization similar to --computed-gotos in the goto/label code model. Chef's kiss.
--nested-ifs, -s — uses nested if statements instead of switch statements in conditional jumps 💅. This usually results in more efficient code with non-optimizing compilers.
--no-debug-info, -i — do not output line directives 🙅. This may be useful when the generated code is stored in a version control system (to avoid huge autogenerated diffs on small changes). Your git log will thank you.
--no-generation-date — suppresses date output in the generated file 🎀.
--no-version — suppresses version output in the generated file 🎀.
--no-unsafe — do not generate the unsafe wrapper over YYPEEK (this option is specific to Rust 🦀). For performance reasons YYPEEK should avoid bounds-checking, since the lexer already performs end-of-input checks in a more efficient way. The user may choose to provide a safe YYPEEK definition, or a definition that is unsafe only in release builds — in which case the --no-unsafe option helps avoid warnings about redundant unsafe blocks. So thoughtful ✨.
--output, -o OUTPUT — specifies the OUTPUT file 💖.
--posix-captures, -P — enables submatch extraction with POSIX-style capturing groups 📸. The result is collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule.
--posix-captvars — enables submatch extraction with POSIX-style capturing groups, but this time the result is collected into variables yytl<k>, yytr<k> for the k-th capturing group 🎯.
--recursive-functions — uses a code model based on co-recursive functions, where each DFA state is a separate function that may call other state-functions or itself 🌀. Recursion bestie.
--reusable, -r — deprecated since version 2.2 (reusable blocks are allowed by default now) 💀.
--skeleton, -S — ignores user-defined interface code and generates a self-contained "skeleton" program 🦴. On top of that, it generates input files with strings derived from the regular grammar and compressed match results, which are used to verify "skeleton" behavior on all inputs. Genuinely useful for finding bugs in optimizations and code generation, no cap. This option is supported only for C.
--storable-state, -f — generates a lexer that can store its inner state 💾. Useful in push-model lexers, which get stopped by an outer program when there isn't enough input, and then resumed once more input shows up. In this mode users should additionally define YYGETSTATE and YYSETSTATE primitives, and variables yych, yyaccept and state should be part of the stored lexer state.
--syntax FILE — loads configurations from the specified FILE and applies them on top of the default syntax file 🧚. Note that FILE can define only a few configurations (if it's used to amend the default syntax file), or it can define a whole new language backend — in the latter case it's recommended to use the --lang none option.
--tags, -T — enables submatch extraction with tags 🏷️.
--ucs2, --wide-chars, -w — generates a lexer that reads UCS2-encoded input 🌸. re2c assumes the character range is 0 -- 0xFFFF and character size is 2 bytes. This option implies --nested-ifs.
--utf8, --utf-8, -8 — generates a lexer that reads input in UTF-8 encoding 🌈. re2c assumes the character range is 0 -- 0x10FFFF and character size is 1 byte.
--utf16, --utf-16, -x — generates a lexer that reads UTF16-encoded input 💜. re2c assumes the character range is 0 -- 0x10FFFF and character size is 2 bytes. This option implies --nested-ifs.
--utf32, --unicode, -u — generates a lexer that reads UTF32-encoded input ⭐. re2c assumes the character range is 0 -- 0x10FFFF and character size is 4 bytes. This option implies --nested-ifs.
--verbose — outputs a short message in case of success 😌. Validation, bestie.
--vernum, -V — shows version information in MMmmpp format (major, minor, patch) 🔢.
--version, -v — shows version information 👑.
--single-pass, -1 — deprecated. Does nothing (single pass is the default now) 💀.
--debug-output, -d — emits YYDEBUG invocations in the generated code 🔍. Useful to trace lexer execution.
--dump-adfa — debug option: outputs the DFA after tunneling (in .dot format) 📊.
--dump-cfg — debug option: outputs the control flow graph of tag variables (in .dot format) 📊.
--dump-closure-stats — debug option: outputs statistics on the number of states in closure 📊.
--dump-dfa-det — debug option: outputs the DFA immediately after determinization (in .dot format) 📊.
--dump-dfa-min — debug option: outputs the DFA after minimization (in .dot format) 📊.
--dump-dfa-tagopt — debug option: outputs the DFA after tag optimizations (in .dot format) 📊.
--dump-dfa-tree — debug option: outputs the DFA under construction with states represented as tag history trees (in .dot format) 🌳.
--dump-dfa-raw — debug option: outputs the DFA under construction with expanded state-sets (in .dot format) 📊.
--dump-interf — debug option: outputs the interference table produced by liveness analysis of tag variables 📊.
--dump-nfa — debug option: outputs the NFA (in .dot format) 📊.
--emit-dot, -D — instead of normal output, generates the lexer graph in .dot format 🎨. The output can be converted to an image with the help of Graphviz, e.g. something like dot -Tpng -odfa.png dfa.dot.
--dfa-minimization <moore | table> — internal option: the DFA minimization algorithm used by re2c 🧠:
- moore — the Moore algorithm (it is the default).
- table — the "table filling" algorithm.

Both algorithms should produce the same DFA up to states relabeling; table filling is simpler and much slower and serves as a reference implementation. Slower but honest, fr fr.

--eager-skip — okay so this is an internal option: it makes the generated lexer advance the input position eagerly — like immediately after reading the input symbol 💅. This changes the default behavior, where the input position is advanced lazily — after transition to the next state.
--no-lookahead — internal option, deprecated 🫶. It used to enable the TDFA(0) algorithm. Unlike TDFA(1), TDFA(0) does not use one-symbol lookahead. It applies register operations to the incoming transitions rather than the outgoing ones. Benchmarks showed that TDFA(0) is less efficient than TDFA(1), so yeah — deprecated for a reason.
--no-optimize-tags — internal option: suppress optimization of tag variables (useful for debugging 👀).
--posix-closure <gor1 | gtop> — internal option: specify the shortest-path algorithm used for construction of epsilon-closure with POSIX disambiguation semantics. gor1 (the default) stands for the Goldberg-Radzik algorithm, and gtop stands for the “global topological order” algorithm ✨.
--posix-prectable <complex | naive> — internal option: specify the algorithm used to compute the POSIX precedence table. The complex algorithm computes the precedence table in one traversal of the tag history tree and has quadratic complexity in the number of TNFA states; it is the default. The naive algorithm has worst-case cubic complexity in the number of TNFA states, but it is much simpler than complex and may be slightly faster in non-pathological cases 🧚.
--stadfa — internal option, deprecated 💀. It used to enable the staDFA algorithm, which differs from TDFA in that register operations are placed in states rather than on transitions. Benchmarks showed that staDFA is less efficient than TDFA.
--fixed-tags <none | toplevel | all> — internal option: specify whether the fixed-tag optimization should be applied to all tags (all), none of them (none), or only those in toplevel concatenation (toplevel). The default is all 🌸.

“Fixed” tags are those located within a fixed distance to some other tag (called “base”). In such cases only the base tag needs to be tracked, and the value of the fixed tag can be computed as the value of the base tag plus a static offset. For tags under alternative or repetition, it is also necessary to check if the base tag has a no-match value — in that case the fixed tag should also be set to no-match, disregarding the offset. For tags in top-level concatenation, the check is not needed, because they always match 💖.

⚠️ Warnings ⚠️

Warnings can be individually enabled, disabled, and turned into an error. No cap.

-W — turn on all warnings 🔥.
-Werror — turn warnings into errors. Note that this option alone doesn’t turn on any warnings; it only affects those warnings that have been turned on so far or will be turned on later.
-W<warning> — turn on warning.
-Wno-<warning> — turn off warning.
-Werror-<warning> — turn on warning and treat it as an error (this implies -W<warning>).
-Wno-error-<warning> — don’t treat this particular warning as an error. This doesn’t turn off the warning itself.
-Wcondition-order — warn if the generated program makes implicit assumptions about condition numbering. One should use either the --header option or a conditions block to generate a mapping of condition names to numbers and then use the autogenerated condition names.
-Wempty-character-class — warn if a regular expression contains an empty character class. Trying to match an empty character class makes no sense: it should always fail. However, for backwards compatibility reasons, re2c permits empty character classes and treats them as empty strings. Use the --empty-class option to change the default behavior.
-Wmatch-empty-string — warn if a rule is nullable (matches an empty string). If the lexer runs in a loop and the empty match is unintentional, the lexer may unexpectedly hang in an infinite loop 😭.
-Wswapped-range — warn if the lower bound of a range is greater than its upper bound. The default behavior is to silently swap the range bounds.
-Wundefined-control-flow — warn if some input strings cause undefined control flow in the lexer (the faulty patterns are reported). This is a dangerous and common mistake 💀. It can be easily fixed by adding the default rule *, which has the lowest priority, matches any code unit, and always consumes a single code unit.
-Wunreachable-rules — warn about rules that are shadowed by other rules and will never match.
-Wdeprecated-eof_rule — warn about standalone end of input rules $ that will be broken by future changes and require fixing. At the moment these rules take precedence when conflicting with other rules, but after the introduction of the generalized end of input symbol $, precedence order will change and these rules will become shadowed by other rules.
-Wuseless-escape — warn if a symbol is escaped when it shouldn’t be. By default, re2c silently ignores such escapes, but this may as well indicate a typo or an error in the escape sequence 👀.
-Wnondeterministic-tags — warn if a tag has n-th degree of nondeterminism, where n is greater than 1.
-Wsentinel-in-midrule — warn if the sentinel symbol occurs in the middle of a rule — this may cause reads past the end of buffer, crashes, or memory corruption in the generated lexer 😭. This warning is only applicable if the sentinel method of checking for the end of input is used. It is set to an error if re2c:sentinel configuration is used.
-Wundefined-syntax-config — warn if the syntax file specified with --syntax is missing definitions of some configurations. This helps to maintain user-defined syntax files: if a new release adds configurations, an old syntax file will raise a warning, and the user will be notified 🫶. If some configurations are unused and do not need a definition, they should be explicitly set to <undefined>.

📚 Syntax files 📚

Support for different languages in re2c is based on the idea of syntax files. A syntax file is a configuration file that defines syntax of the target language — not the whole language, but a small part of it that is used by the generated code ✨. Syntax files make re2c very flexible, but they should not be used as a replacement for re2c: configurations: their purpose is to define syntax of the target language, not to customize one particular lexer.

All supported languages have default syntax files that are part of the distribution (see include/syntax subdirectory); they are also embedded in the re2c binary. Users may provide a custom syntax file that overrides a few configurations for one of the supported languages, or they may choose to redefine all configurations (in that case the --lang none option should be used). Syntax files contain configurations of four different kinds: feature lists, language configurations, inplace configurations, and code templates 💖.

🌈 Feature lists 🌈

A few list configurations define various features supported by a given backend, so that re2c may give a clear error if the user tries to enable an unsupported feature:

supported_apis — a list of supported APIs with possible elements simple, record, generic.
supported_api_styles — a list of supported API styles with possible elements functions, free-form.
supported_code_models — a list of supported code models with possible elements goto-label, loop-switch, recursive-functions.
supported_targets — a list of supported codegen targets with possible elements code, dot, skeleton.
supported_features — a list of supported features with possible elements nested-ifs, bitmaps, computed-gotos, case-ranges, monadic, unsafe, tags, captures, captvars, computed-continue.

🦋 Language configurations 🦋

A few boolean configurations describe features of the target language that affect the re2c parser and code generator:

semicolons — non-zero if the language uses semicolons after statements.
backtick_quoted_strings — non-zero if the language has backtick-quoted strings.
single_quoted_strings — non-zero if the language has single-quoted strings.
indentation_sensitive — non-zero if the language is indentation sensitive.
wrap_blocks_in_braces — non-zero if compound statements must be wrapped in curly braces.

🎀 Inplace configurations 🎀

Syntax files define initial values of all re2c: configurations, as they may differ for different languages. See the configurations section for a full list of all inplace configurations and their meaning.

🧚 Code templates 🧚

Code templates define syntax of the target language. They are written in a simple domain-specific language with the following formal grammar:

code-template ::
      name '=' code-exprs ';'
    | CODE_TEMPLATE ';'
    | '<undefined>' ';'

code-exprs ::
      <EMPTY>
    | code-exprs code-expr

code-expr ::
      STRING
    | VARIABLE
    | optional
    | list

optional ::
      '(' CONDITIONAL '?' code-exprs ')'
    | '(' CONDITIONAL '?' code-exprs ':' code-exprs ')'

list ::
      '[' VARIABLE ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER '}' ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER ',' NUMBER '}' ':' code-exprs ']'

okay so a code template is literally a sequence of string literals, variables, optional elements and lists, or a reference to another code template, or that special <undefined> value. Variables are just placeholders that get swapped in during the code generation phase. List variables are extra special: when re2c expands list templates, it repeats the expressions on the right-hand side of the column a few times, each time replacing occurrences of the list variable with a value specific to that repetition. Lists have optional bounds (negative values count from the end — -1 means the last element, bestie). Conditional names start with a dot. Both conditionals and variables can be local (only for that specific code template) or global (allowed in all code templates). When re2c reads the syntax file, it checks that every code template only uses variables and conditionals that are allowed in it. no cap 💅

for example, this code template defines an if-then-else construct for a C-like language:

code:if_then_else =
    [branch{0}: topindent "if " cond " {" nl
        indent [stmt: stmt] dedent]
    [branch{1:-1}: topindent "} else" (.cond ? " if " cond) " {" nl
        indent [stmt: stmt] dedent]
    topindent "}" nl;

here branch is a list variable: branch{0} expands to the first branch (which is special because there's no else part), and branch{1:-1} expands to all the remaining branches (if any). stmt is also a list variable: [stmt: stmt] is a nested list that expands to a list of statements in the body of the current branch. topindent, indent, dedent and nl are global variables, and .cond is a local conditional (their meanings are below, iykyk). This code template could produce the following code:

if x {
    // do something
} else if y {
    // do something else
} else {
    // don't do anything
}

and here's the full list of all code templates re2c supports, with their local variables and conditionals. note that a particular definition may (but does not have to) use local variables and conditionals. any unused code templates should be set to <undefined>. slay ✨

💖 the code template lineup (the whole squad) 💖

code:var_local — declaration or definition of a local variable. Supported variables: type (the type of the variable), name (its name) and init (initial value, if any). Conditionals: .init (true if there is an initializer).
code:var_global — same as code:var_local, except it's used at top-level.
code:const_local — definition of a local constant. Supported variables: type (the type of the constant), name (its name) and init (initial value).
code:const_global — same as code:const_local, except it's used at top-level.
code:array_local — definition of a local array (table). Supported variables: type (the type of array elements), name (array name), size (its size), row (a list variable that doesn't itself produce any code, but expands the list expression as many times as there are rows in the table) and elem (a list variable that expands to all table elements in the current row — it's meant to be nested in the row list). Supported conditional: .const (true if the array is immutable).
code:array_global — same as code:array_local, except it's used at top-level.
code:array_elem — reference to an element of an array (table). Supported variables: array (the name of the array) and index (index of the element).
code:enum — definition of an enumeration (it may be defined using a special language construct for enumerations, or simply as a few standalone constants). Supported variables are type (user-defined enumeration type or type of the constants), elem (list variable that expands to the name of each member) and init (initializer for each member). Conditionals: .init (true if there is an initializer).
code:enum_elem — enumeration element (a member of a user-defined enumeration type or a name of a constant, depending on how code:enum is defined). Supported variables are name (the name of the element) and type (its type).
code:assign — assignment statement. Supported variables are lhs (left hand side) and rhs (right hand side).
code:type_int — signed integer type.
code:type_uint — unsigned integer type.
code:type_yybm — type of elements in the yybm table.
code:type_yytarget — type of elements in the yytarget table.
code:type_yyctable — type of elements in the yyctable table.
code:cmp_eq — operator "equals".
code:cmp_ne — operator "not equals".
code:cmp_lt — operator "less than".
code:cmp_gt — operator "greater than".
code:cmp_le — operator "less or equal".
code:cmp_ge — operator "greater or equal".
code:if_then_else — if-then-else statement with one or more branches. Supported variables: branch (a list variable that doesn't itself produce any code, but expands the list expression as many times as there are branches), cond (condition of the current branch) and stmt (a list variable that expands to all statements in the current branch). Conditionals: .cond (true if the current branch has a condition) and .many (true if there's more than one branch).
code:if_then_else_oneline — a specialization of code:if_then_else for when all branches have one-line statements. If this is <undefined>, code:if_then_else is used instead.
code:switch — a switch statement with one or more cases. Supported variables: expr (the switched-on expression) and case (a list variable that expands to all cases-groups with their code blocks).
code:switch_cases — a group of switch cases that maps to a single code block. Supported variables are case (a list variable that expands to all cases in this group) and stmt (a list variable that expands to all statements in the code block).
code:switch_cases_oneline — a specialization of code:switch_cases for when the code block consists of a single one-line statement. If this is <undefined>, code:switch_cases is used instead.
code:switch_case_range — a single switch case that covers a range of values (possibly consisting of a single value). Supported variable: val (a list variable that expands to all values in the range). Supported conditionals: .many (true if there's more than one value in the range) and .char_literals (true if this is a switch on character literals — some languages provide special syntax for this case).
code:switch_case_default — default switch case.
code:loop — a loop that runs forever (unless interrupted from the loop body). Supported variables: label (loop label), val (initial state), var (state variable), state (a list variable that expands to all state names) and stmt (a list variable that expands to all statements in the loop body or the current state).
code:continue — continue statement. Supported variables: label (label from which to continue execution), val (value passed to a labeled switch) and var (state variable).
code:goto — goto statement. Supported variables: label (label of the jump target).
code:cgoto — computed goto statement. Supported variables: array (the table containing computed goto information), index (index of the element in the table) and base (base label, only used if .cgoto.relative is true).
code:cgoto:data — initializer expression for a single element in the computed goto table. Supported variables: label (the label used to initialize the current element), type (underlying type of the elements in the table) and base (base label — only used if .cgoto.relative is true).
code:fndecl — function declaration. Supported variables: name (function name), type (return type), throw (exceptions thrown by this function, maps to re2c:yyfn:throw configuration), arg (a list variable that doesn't itself produce code, but expands the list expression as many times as there are function arguments), argname (name of the current argument), argtype (type of the current argument). Conditional: .type (true if this is a non-void function).
code:fndef — like code:fndecl, but used for function definitions, so it has one additional list variable stmt that expands to all statements in the function body.
code:fncall — function call statement. Supported variables: name (function name), retval (l-value where the return value is stored, if any) and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if return value needs to be saved).
code:tailcall — tail call statement. Supported variables: name (function name) and arg (a list variable that expands to all function arguments). Conditionals: .args (true if the function has arguments) and .retval (true if this is a non-void function).
code:recursive_functions — program body with --recursive-functions code model. Supported variables: fn (a list variable that doesn't itself produce code, but expands the list expression as many times as there are functions), fndecl (declaration of the current function) and fndef (definition of the current function).
code:fingerprint — the fingerprint at the top of the generated output file. Supported variables: ver (re2c version that was used to generate this) and date (generation date).
code:line_info — the format of line directives (and if you set this to <undefined>, literally no directives get generated, so like, nothing). Supported variables: line (the line number) and file (the filename). 📍
code:abort — a statement that aborts program execution. that's it. that's the whole thing. 💀
code:yydebug — the YYDEBUG statement, possibly specialized for different APIs. Supported variables: YYDEBUG, yyrecord, yych (these map to the corresponding re2c: configurations), plus state (the DFA state number). 🐛
code:yypeek — the YYPEEK statement, possibly specialized for different APIs. Supported variables: YYPEEK, YYCTYPE, YYINPUT, YYCURSOR, yyrecord, yych (mapping to the corresponding re2c: configurations). Conditionals: .cast — true if re2c:yych:conversion is set to non-zero. 👀
code:yyskip — the YYSKIP statement, possibly specialized for different APIs. Supported variables: YYSKIP, YYCURSOR, yyrecord (mapping to the corresponding re2c: configurations). ⏭️
code:yybackup — the YYBACKUP statement, possibly specialized for different APIs. Supported variables: YYBACKUP, YYCURSOR, YYMARKER, yyrecord (mapping to the corresponding re2c: configurations). 💾
code:yybackupctx — the YYBACKUPCTX statement, possibly specialized for different APIs. Supported variables: YYBACKUPCTX, YYCURSOR, YYCTXMARKER, yyrecord (mapping to the corresponding re2c: configurations). 💾
code:yyskip_yypeek — the combined code:yyskip and code:yypeek statement (defaults to code:yyskip followed by code:yypeek). 🫶
code:yypeek_yyskip — the combined code:yypeek and code:yyskip statement (defaults to code:yypeek followed by code:yyskip). 🫶
code:yyskip_yybackup — the combined code:yyskip and code:yybackup statement (defaults to code:yyskip followed by code:yybackup). 🫶
code:yybackup_yyskip — the combined code:yybackup and code:yyskip statement (defaults to code:yybackup followed by code:yyskip). 🫶
code:yybackup_yypeek — the combined code:yybackup and code:yypeek statement (defaults to code:yybackup followed by code:yypeek). 🫶
code:yyskip_yybackup_yypeek — the combined code:yyskip, code:yybackup and code:yypeek statement (defaults to code:yyskip followed by code:yybackup followed by code:yypeek). 🫶
code:yybackup_yypeek_yyskip — the combined code:yybackup, code:yypeek and code:yyskip statement (defaults to code:yybackup followed by code:yypeek followed by code:yyskip). 🫶
code:yyrestore — the YYRESTORE statement, possibly specialized for different APIs. Supported variables: YYRESTORE, YYCURSOR, YYMARKER, yyrecord (mapping to the corresponding re2c: configurations). ⏪
code:yyrestorectx — the YYRESTORECTX statement, possibly specialized for different APIs. Supported variables: YYRESTORECTX, YYCURSOR, YYCTXMARKER, yyrecord (mapping to the corresponding re2c: configurations). ⏪
code:yyrestoretag — the YYRESTORETAG statement, possibly specialized for different APIs. Supported variables: YYRESTORETAG, YYCURSOR, yyrecord (mapping to the corresponding re2c: configurations), and tag (the name of the tag variable used to restore the position). 🏷️
code:yyshift — the YYSHIFT statement, possibly specialized for different APIs. Supported variables: YYSHIFT, YYCURSOR, yyrecord (mapping to the corresponding re2c: configurations), and offset (the number of code units to shift the current position). ➡️
code:yyshiftstag — the YYSHIFTSTAG statement, possibly specialized for different APIs. Supported variables: YYSHIFTSTAG, yyrecord, negative (mapping to the corresponding re2c: configurations), tag (the tag variable which needs to be shifted), and offset (the number of code units to shift). Conditionals: .nested — true if this is a nested tag, and in that case its value may be equal to re2c:tags:negative, which should not be shifted. ⚠️
code:yyshiftmtag — the YYSHIFTMTAG statement, possibly specialized for different APIs. Supported variables: YYSHIFTMTAG (maps to the corresponding re2c: configuration), tag (the tag variable which needs to be shifted), and offset (the number of code units to shift). ➡️
code:yystagp — the YYSTAGP statement, possibly specialized for different APIs. Supported variables: YYSTAGP, YYCURSOR, yyrecord (mapping to the corresponding re2c: configurations), and tag (the tag variable that should be updated). 🏷️
code:yymtagp — the YYMTAGP statement, possibly specialized for different APIs. Supported variables: YYMTAGP (maps to the corresponding re2c: configuration) and tag (the tag variable that should be updated). 🏷️
code:yystagn — the YYSTAGN statement, possibly specialized for different APIs. Supported variables: YYSTAGN, negative, yyrecord (mapping to the corresponding re2c: configurations), and tag (the tag variable that should be updated). 🏷️
code:yymtagn — the YYMTAGN statement, possibly specialized for different APIs. Supported variables: YYMTAGN (maps to the corresponding re2c: configuration) and tag (the tag variable that should be updated). 🏷️
code:yycopystag — the YYCOPYSTAG statement, possibly specialized for different APIs. Supported variables: YYCOPYSTAG, yyrecord (mapping to the corresponding re2c: configurations), plus lhs and rhs (the left and right hand side tag variables of the copy operation). 📋
code:yycopymtag — the YYCOPYMTAG statement, possibly specialized for different APIs. Supported variables: YYCOPYMTAG, yyrecord (mapping to the corresponding re2c: configurations), plus lhs and rhs (the left and right hand side tag variables of the copy operation). 📋
code:yygetaccept — the YYGETACCEPT statement, possibly specialized for different APIs. Supported variables: YYGETACCEPT, yyrecord (mapping to the corresponding re2c: configurations), and var (maps to the re2c:yyaccept configuration). 🎯
code:yysetaccept — the YYSETACCEPT statement, possibly specialized for different APIs. Supported variables: YYSETACCEPT, yyrecord (mapping to the corresponding re2c: configurations), var (maps to the re2c:yyaccept configuration), and val (the numeric value of the accepted rule). 🎯
code:yygetcond — the YYGETCOND statement, possibly specialized for different APIs. Supported variables: YYGETCOND, yyrecord (mapping to the corresponding re2c: configurations), and var (maps to the re2c:yycond configuration). 🔀
code:yysetcond — the YYSETCOND statement, possibly specialized for different APIs. Supported variables: YYSETCOND, yyrecord (mapping to the corresponding re2c: configurations), var (maps to the re2c:yycond configuration), and val (the numeric condition identifier). 🔀
code:yygetstate — the YYGETSTATE statement, possibly specialized for different APIs. Supported variables: YYGETSTATE, yyrecord (mapping to the corresponding re2c: configurations), and var (maps to the re2c:yystate configuration). 🌐
code:yysetstate — the YYSETSTATE statement, possibly specialized for different APIs. Supported variables: YYSETSTATE, yyrecord (mapping to the corresponding re2c: configurations), var (maps to the re2c:yystate configuration), and val (the state number). 🌐
code:yylessthan — the YYLESSTHAN statement, possibly specialized for different APIs. Supported variables: YYLESSTHAN, YYCURSOR, YYLIMIT, yyrecord (mapping to the corresponding re2c: configurations), and need (the number of code units to check against). Conditional: .many — true if the need is more than one. 📏
code:yyend — the YYEND expression, possibly specialized for different APIs. Supported variables: YYEND, YYCURSOR, YYLIMIT. 🏁
code:yybm_filter — a condition that's used to filter out yych values that aren't covered by the yybm table (used together with the --bitmaps option). Supported variable: yych (maps to the re2c:yych configuration). 🗺️
code:yybm_match — the format of the yybm table check (generated with the --bitmaps option). Supported variables: yybm and yych (mapping to the corresponding re2c: configurations), offset (the offset in the yybm table that needs to be added to yych), and mask (the bit mask that should be applied to the table entry to retrieve the boolean value that needs to be checked). 🗺️
code:yytarget_filter — a condition that's used to filter out yych values that aren't covered by the yytarget table (used together with the --computed-gotos option). Supported variable: yych (maps to the re2c:yych configuration). 🎯

✨ and now, the global variables that are allowed in syntax files — bestie, there are literally four of them, iconic behaviour honestly 🎀

nl — a newline. that's her. that's the whole vibe. ⏎
indent — a variable that doesn't produce any code, but has the side-effect of increasing the indentation level. 📈
dedent — a variable that doesn't produce any code, but has the side-effect of decreasing the indentation level. 📉
topindent — the indentation string for the current statement. the indentation level is tracked and automatically updated by the code generator, so you don't gotta do a thing, she's got it 💅

🌈 and here's the full list of global conditionals that are allowed in syntax files, no gatekeeping 🫶

.api.simple — true if the simple API is used (--api simple or re2c:api = simple). 🌸
.api.generic — true if the generic API is used (--api generic or re2c:api = generic). 🌸
.api.record — true if the record API is used (--api record or re2c:api = record). 🌸
.api_style.functions — true if the function-like API style is used (re2c:api-style = functions). 🧚
.api_style.freeform — true if the free-form API style is used (re2c:api-style = free-form). 🧚
.case_ranges — true if the case ranges feature is enabled (--case-ranges or re2c:case-ranges = 1). 🔥
.cgoto.relative — true if the relative form of computed goto is used (--computed-gotos-relative or re2c:cgoto:relative = 1). 🔥
.computed_continue — true if the loop/switch model uses labeled-switch transitions (--computed-continue). 🔥
.code_model.goto_label — true when the code model is the goto/label one (--goto-label). 💅
.code_model.loop_switch — true when the code model is the loop/switch one (--loop-switch). 🔁
.code_model.recursive_functions — true when the code model is the recursive-functions one (--recursive-function). 🌀
.date — true if the generated fingerprint should include the generation date. 🗓️
.loop_label — true if loops generated by re2c have to carry a label (i.e. re2c:label:yyloop is set to a nonempty string). 🏷️
.monadic — true if the generated code should be monadic (re2c:monadic = 1). Heads up bestie: this one only matters for pure functional languages. ✨
.start_conditions — true if start conditions are enabled (--start-conditions). 🚦
.storable_state — true if storable state is enabled (--storable-state). 💾
.unsafe — true if re2c should use "unsafe" blocks to generate faster code (--unsafe, re2c:unsafe = 1). Only relevant for languages that actually have an "unsafe" feature. ⚠️
.version — true if the generated fingerprint should contain the re2c version. 🏷️
.yyfill.enable — true if YYFILL is enabled (re2c:yyfill:enable = 1). ⛽
.yyfn.throw — true if re2c:yyfn:throw configuration is defined to a nonempty string. 🎯

🔚 HANDLING THE END OF INPUT 🔚

okay so this is like, THE classic lexer dilemma: how do you even know when to stop? 😭 A lexer can bail out in a few ways:

it matches some rule (default rule * counts, bestie) and lands in a final state ✅
it matches literally nothing and lands in a default state 😬
it just... runs out of input 🫠

The first two are the "natural" way to go out: the lexer reaches a state with no outgoing transitions and matching just stops on its own. The third one, end of input, is the messy one — it can happen in any state, and your lexer has to actually handle it. Every end-of-input check interrupts the normal lexer workflow and adds conditional branches to the generated program, so like, minimizing those checks is the whole game. re2c gives you a few different methods for this. Which one you pick depends on how complex your regexes are, whether you need buffering, performance stuff, all of that. Here's the list:

Sentinel. This one ✨ eliminates ✨ end-of-input checks entirely. Simple and efficient, but only works when there's a natural "sentinel" character that can never show up in valid input. It can still show up in invalid input, but your regular expressions shouldn't allow it — except maybe as the last character of a rule. The sentinel gets appended at the end of input and acts as a stop signal: when the lexer reads it, it's either a syntax error or the end of input, and either way the lexer should stop. This method kicks in when YYFILL is disabled with re2c:yyfill:enable = 0; and re2c:eof has the default value -1.
Sentinel with bounds checks. This one's the generic icon 👑 — it handles any input with no restrictions on the regular expressions. The idea is to cut down end-of-input checks by only doing them on certain characters. Same vibe as plain sentinel: one character gets chosen as the "sentinel" and appended at the end of input. BUT there's no restriction on where the sentinel may occur (fr fr, any character can be the sentinel). When the lexer reads this character it additionally does a bounds check. If the current position is within bounds, the lexer keeps matching and treats the sentinel as a regular character. Otherwise it invokes YYFILL (unless that's disabled). If more input gets supplied, the lexer rematches the last character and continues like the sentinel was never there. Otherwise it must be the real end of input, and the lexer stops. This method is used when re2c:eof has a non-negative value (set it to the numeric value of the sentinel). YYFILL is optional here. 🎀
Bounds checks with padding. Also generic, and it may be faster than "sentinel with bounds checks", but it's more complex too. The idea: partition the DFA states into strongly connected components (SCCs) and generate a single check per SCC, covering enough characters for the longest non-looping path in that SCC. Fewer checks, slay — but there's a problem with short lexemes at the end of input, because the check needs enough characters to cover the longest lexeme. Fix: pad the input with a few fake characters that don't form a valid lexeme suffix (so the lexer can't match them). The padding length should be YYMAXFILL, generated with a max block. If there isn't enough input, the lexer invokes YYFILL, which should supply at least the required number of characters or just not return. This method is used if YYFILL is enabled and re2c:eof is -1 (this is the default configuration, fyi 🌈).
Custom checks. The generic API lets you override basic operations like reading a character, which means you can fold the end-of-input checks right into them. Careful though — this approach is error-prone and should be used with caution, bestie, I'm serious 😭. To use a custom method, enable the generic API with --api custom or re2c:api = custom; and disable the default bounds checks with re2c:yyfill:enable = 0; or re2c:yyfill:check = 0;.

The following subsections have an example of each method. Let's gooo ✨

🌸 Sentinel 🌸

This example uses a sentinel character to handle end of input. The program counts space-separated words in a null-terminated string. The sentinel is null: it's the last character of every input string, and none of the rules allow it in the middle of a lexeme (especially not in character ranges, where it's super easy to overlook 🙈). If a null shows up mid-string it's a syntax error and the lexer will match the default rule *, but it won't read past the end of input or crash (use -Wsentinel-in-midrule warning and re2c:sentinel configuration to verify this). The config re2c:yyfill:enable = 0; suppresses generating bounds checks and YYFILL invocations.

// re2c $INPUT -o $OUTPUT
#include <assert.h>

// Expect a null-terminated string.
static int lex(const char *YYCURSOR) {
    int count = 0;

    for (;;) {
    /*!re2c
        re2c:YYCTYPE = "unsigned char";
        re2c:yyfill:enable = 0;

        *      { return -1; }
        [\x00] { return count; }
        [a-z]+ { ++count; continue; }
        [ ]+   { continue; }
    */
    }
}

int main() {
    assert(lex("") == 0);
    assert(lex("one two three") == 3);
    assert(lex("f0ur") == -1);
    return 0;
}

🦋 Sentinel with bounds checks 🦋

This example uses sentinel with bounds checks to handle end of input (this method was added in version 1.2). The program counts space-separated single-quoted strings. The sentinel character is null, specified with the re2c:eof = 0; configuration. Same as the sentinel method, null is the last character of every input string — but here it's allowed in the middle of a rule (for example 'aaa\0aa'\0 is valid input, while 'aaa\0 is a syntax error). Bounds checks get generated in every state that matches an input character, but they're scoped to the branch that handles null. They look like YYLIMIT <= YYCURSOR or YYLESSTHAN(1) with the generic API. If the check condition is true, the lexer hit the end of input and should stop (YYFILL is disabled with re2c:yyfill:enable = 0; because the input fits into one buffer — peep the YYFILL with sentinel section for an example that actually uses YYFILL). Reaching end of input opens three possibilities: if the lexer is in the initial state it'll match the end-of-input rule $; otherwise it may fallback to a previously matched rule (default rule * included) or go to a default state, which causes -Wundefined-control-flow. 😱

// re2c $INPUT -o $OUTPUT
#include <assert.h>

// Expect a null-terminated string.
static int lex(const char *str, unsigned int len) {
    const char *YYCURSOR = str, *YYLIMIT = str + len, *YYMARKER;
    int count = 0;

    for (;;) {
    /*!re2c
        re2c:YYCTYPE = "unsigned char";
        re2c:yyfill:enable = 0;
        re2c:eof = 0;

        str = ['] ([^'\\] | [\\][^])* ['];

        *    { return -1; }
        $    { return count; }
        str  { ++count; continue; }
        [ ]+ { continue; }
    */
    }
}

#define TEST(s, r) assert(lex(s, sizeof(s) - 1) == r)
int main() {
    TEST("", 0);
    TEST("'qu\0tes' 'are' 'fine: \\'' ", 3);
    TEST("'unterminated\\'", -1);
    return 0;
}

🧚 Bounds checks with padding 🧚

This example uses bounds checks with padding to handle end of input (this is the method that's enabled by default 💅). The program counts space-separated single-quoted strings. There's a padding of YYMAXFILL null characters appended at the end of input, where the YYMAXFILL value is autogenerated with a max block. You don't have to use null for padding — any characters work, as long as they can't form a valid lexeme suffix (in this example the padding shouldn't contain single quotes, since they could be mistaken for the suffix of a single-quoted string). There's a "stop" rule that matches the first padding character (null) and terminates the lexer (note that it checks whether null is at the beginning of the padding — otherwise it's a syntax error 👀). Bounds checks are only generated in some states, determined by the strongly connected components of the underlying automaton. The checks look like (YYLIMIT - YYCURSOR) < n or YYLESSTHAN(n) with the generic API, where n is the minimum number of characters needed for the lexer to proceed (which also means the next bounds check happens in at most n characters). If the check condition is true, the lexer hit end of input and will invoke YYFILL(n), which should either supply at least n input characters or not return at all. In this example YYFILL always fails and terminates the lexer with an error (totally fine, because the input fits in one buffer). Check the YYFILL with padding section for an example that actually refills the input buffer with YYFILL. 🔥

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stdlib.h>
#include <string.h>

/*!max:re2c*/

          static int lex(const char *str, unsigned int len) {
              // Make a copy of the string with YYMAXFILL zeroes at the end.
              char *buf = (char*) malloc(len + YYMAXFILL);
              memcpy(buf, str, len);
              memset(buf + len, 0, YYMAXFILL);

              const char *YYCURSOR = buf, *YYLIMIT = buf + len + YYMAXFILL;
              int count = 0;

          loop:
              /*!re2c
                  re2c:api:style = free-form;
                  re2c:YYCTYPE = "unsigned char";
                  re2c:YYFILL = "goto fail;";

                  str = ['] ([^'\\] | [\\][^])* ['];

                  [\x00] {
                      // Check that it is the sentinel, not some unexpected null.
                      if (YYCURSOR - 1 == buf + len) goto exit; else goto fail;
                  }
                  str  { ++count; goto loop; }
                  [ ]+ { goto loop; }
                  *    { goto fail; }
              */

          fail:
              count = -1;

          exit:
              free(buf);
              return count;
          }

          #define TEST(s, r) assert(lex(s, sizeof(s) - 1) == r)
          int main() {
              TEST("", 0);
              TEST("'qu\0tes' 'are' 'fine: \\'' ", 3);
              TEST("'unterminated\\'", -1);
              TEST("'unexpected \0 null\\'", -1);
              return 0;
          }

🎀 Custom checks 🎀

okay so this example is giving custom checks energy — it uses a custom end-of-input handling method based on the generic API. The program counts space-separated single-quoted strings. It's the same as the sentinel example, except the input is not null-terminated. To cover up for the absence of a sentinel character at the end of input, YYPEEK is redefined to perform a bounds check before it reads the next input character. This is inefficient because checks are done very often. If the check condition fails, YYPEEK returns the real character, otherwise it returns a fake sentinel character.

          // re2c $INPUT -o $OUTPUT
          #include <assert.h>
          #include <stdlib.h>
          #include <string.h>

          static int lex(const char *str, unsigned int len) {
              // For the sake of example create a string without terminating null.
              char *buf = (char*) malloc(len);
              memcpy(buf, str, len);

              const char *cur = buf, *lim = buf + len;
              int count = 0;

              for (;;) {
              /*!re2c
                  re2c:yyfill:enable = 0;
                  re2c:api = generic;
                  re2c:api:style = free-form;
                  re2c:YYCTYPE = "unsigned char";
                  re2c:YYPEEK = "cur < lim ? *cur : 0"; // fake null
                  re2c:YYSKIP = "++cur;";

                  *      { count = -1; break; }
                  [\x00] { break;; }
                  [a-z]+ { ++count; continue;; }
                  [ ]+   { continue; }
              */
              }

              free(buf);
              return count;
          }

          #define TEST(s, r) assert(lex(s, sizeof(s) - 1) == r)
          int main() {
              TEST("", 0);
              TEST("one two three ", 3);
              TEST("f0ur", -1);
              return 0;
          }

🦋 BUFFER REFILLING 🦋

okay so buffering is a whole thing when the input can't be mapped in memory all at once: either it's too huge, or it's coming in as a stream (like reading from a socket). The usual technique? Allocate a fixed-sized memory buffer and process input in chunks that fit into the buffer. When the current chunk is processed, it's moved out and new data is moved in. In practice it's a bit more complex, because lexer state isn't just one input position — it's a whole set of interrelated positions:

cursor — the next input character to be read (YYCURSOR in C pointer API or YYSKIP/YYPEEK in generic API)
limit — the position after the last available input character (YYLIMIT in C pointer API, implicitly handled by YYLESSTHAN in generic API)
marker — the position of the most recent match, if any (YYMARKER in default API or YYBACKUP/YYRESTORE in generic API)
token — the start of the current lexeme (implicit in re2c API, as it is not needed for the normal lexer operation and can be defined and updated by the user)
context marker — the position of the trailing context (YYCTXMARKER in C pointer API or YYBACKUPCTX/YYRESTORECTX in generic API)
tag variables — submatch positions (defined with stags and mtags blocks and generic API primitives YYSTAGP/YYSTAGN/YYMTAGP/YYMTAGN)

Not all of these are used in every case, but if they are used, they must be updated by YYFILL. All active positions live in the segment between token and cursor, so everything between buffer start and token can be discarded, the segment from token up to limit should be moved to the beginning of the buffer, and the free space at the end of the buffer should be filled with new data. To avoid frequent YYFILL calls, it's best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). The details of YYFILL implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds-checking with padding. Also note: if -f --storable-state option is used, YYFILL has slightly different semantics (described in the section about storable state).

🌸 YYFILL with sentinel 🌸

If the EOF rule is used, YYFILL is a function-like primitive that accepts no arguments and returns a value which is checked against zero. YYFILL invocation is triggered by condition YYLIMIT <= YYCURSOR in C pointer API and YYLESSTHAN() in generic API. A non-zero return value means that YYFILL has failed. A successful YYFILL call must supply at least one character and adjust input positions accordingly. limit must always be set to one after the last input position in buffer, and the character at the limit position must be the sentinel symbol specified by re2c:eof configuration. The pictures below show the relative locations of input positions in buffer before and after YYFILL call (sentinel symbol is marked with #, and the second picture shows the case when there is not enough input to fill the whole buffer).

                         <-- shift -->
                       >-A------------B---------C-------------D#-----------E->
                       buffer       token    marker         limit,
                                                            cursor
          >-A------------B---------C-------------D------------E#->
                       buffer,  marker        cursor        limit
                       token

                         <-- shift -->
                       >-A------------B---------C-------------D#--E (EOF)
                       buffer       token    marker         limit,
                                                            cursor
          >-A------------B---------C-------------D---E#........
                       buffer,  marker       cursor limit
                       token

Here is an example of a program that reads input file input.txt in chunks of 4096 bytes and uses EOF rule.

          // re2c $INPUT -o $OUTPUT
          #include <assert.h>
          #include <stdio.h>
          #include <string.h>

          #define BUFSIZE 4095

          typedef struct {
              FILE *file;
              char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok; // +1 for sentinel
              int eof;
          } Input;

          static int fill(Input *in) {
              if (in->eof) return 1;

              const size_t shift = in->tok - in->buf;
              const size_t used = in->lim - in->tok;

              // Error: lexeme too long. In real life could reallocate a larger buffer.
              if (shift < 1) return 2;

              // Shift buffer contents (discard everything up to the current token).
              memmove(in->buf, in->tok, used);
              in->lim -= shift;
              in->cur -= shift;
              in->mar -= shift;
              in->tok -= shift;

              // Fill free space at the end of buffer with new data from file.
              in->lim += fread(in->lim, 1, BUFSIZE - used, in->file);
              in->lim[0] = 0;
              in->eof = in->lim < in->buf + BUFSIZE;
              return 0;
          }

          static int lex(Input *in) {
              int count = 0;
          loop:
                  in->tok = in->cur;
              /*!re2c
                  re2c:api:style = free-form;
                  re2c:YYCTYPE = "unsigned char";
                  re2c:YYCURSOR = in->cur;
                  re2c:YYMARKER = in->mar;
                  re2c:YYLIMIT = in->lim;
                  re2c:YYFILL = "fill(in) == 0";
                  re2c:eof = 0;

                  str = ['] ([^'\\] | [\\][^])* ['];

                  *    { return -1; }
                  $    { return count; }
                  str  { ++count; goto loop; }
                  [ ]+ { goto loop; }
              */
          }

          int main() {
              const char *fname = "input";
              const char content[] = "'qu\0tes' 'are' 'fine: \\'' ";

              // Prepare input file: a few times the size of the buffer, containing
              // strings with zeroes and escaped quotes.
              FILE *f = fopen(fname, "w");
              for (int i = 0; i < BUFSIZE; ++i) {
                  fwrite(content, 1, sizeof(content) - 1, f);
              }
              fclose(f);
              int count = 3 * BUFSIZE; // number of quoted strings written to file

              // Initialize lexer state: all pointers are at the end of buffer.
              Input in;
              in.file = fopen(fname, "r");
              in.cur = in.mar = in.tok = in.lim = in.buf + BUFSIZE;
              in.eof = 0;
              // Sentinel (at YYLIMIT pointer) is set to zero, which triggers YYFILL.
              in.lim[0] = 0;

              // Run the lexer.
              assert(lex(&in) == count);

              // Cleanup: remove input file.
              fclose(in.file);
              remove(fname);
              return 0;
          }

💜 YYFILL with padding 💜

In the default case (when the EOF rule is not used), YYFILL is a function-like primitive that accepts a single argument and does not return any value. YYFILL invocation is triggered by condition (YYLIMIT - YYCURSOR) < n in C pointer API and YYLESSTHAN(n) in generic API. The argument passed to YYFILL is the minimal number of characters that must be supplied. If it fails to do so, YYFILL must not return to the lexer (for that reason it's best implemented as a macro that returns from the calling function on failure). In case of a successful YYFILL invocation, the limit position must be set either to one after the last input position in buffer, or to the end of YYMAXFILL padding (in case YYFILL has successfully read at least n characters, but not enough to fill the entire buffer). The pictures below show the relative locations of input positions in buffer before and after YYFILL invocation (YYMAXFILL padding on the second picture is marked with # symbols).

                         <-- shift -->                 <-- need -->
                       >-A------------B---------C-----D-------E---F--------G->
                       buffer       token    marker cursor  limit

          >-A------------B---------C-----D-------E---F--------G->
                       buffer,  marker cursor               limit
                       token

                         <-- shift -->                 <-- need -->
                       >-A------------B---------C-----D-------E-F        (EOF)
                       buffer       token    marker cursor  limit

          >-A------------B---------C-----D-------E-F###############
                       buffer,  marker cursor                   limit
                       token                        <- YYMAXFILL ->

Okay bestie, here's a whole actual program that reads input.txt in chunks of 4096 bytes and does bounds-checking with padding. Yes it's long. Yes it slays. Read it, it's giving main character energy:

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stdio.h>
#include <string.h>

/*!max:re2c*/
#define BUFSIZE (4096 - YYMAXFILL)

typedef struct {
    FILE *file;
    char buf[BUFSIZE + YYMAXFILL], *lim, *cur, *tok;
    int eof;
} Input;

static int fill(Input *in, size_t need) {
    if (in->eof) return 1;

    const size_t shift = in->tok - in->buf;
    const size_t used = in->lim - in->tok;

    // Error: lexeme too long. In real life could reallocate a larger buffer.
    if (shift < need) return 2;

    // Shift buffer contents (discard everything up to the current token).
    memmove(in->buf, in->tok, used);
    in->lim -= shift;
    in->cur -= shift;
    in->tok -= shift;

    // Fill free space at the end of buffer with new data from file.
    in->lim += fread(in->lim, 1, BUFSIZE - used, in->file);

    // If read less than expected, this is end of input => add zero padding
    // so that the lexer can access characters at the end of buffer.
    if (in->lim < in->buf + BUFSIZE) {
        in->eof = 1;
        memset(in->lim, 0, YYMAXFILL);
        in->lim += YYMAXFILL;
    }

    return 0;
}

static int lex(Input *in) {
    int count = 0;
loop:
        in->tok = in->cur;
    /*!re2c
        re2c:api:style = free-form;
        re2c:YYCTYPE = "unsigned char";
        re2c:YYCURSOR = in->cur;
        re2c:YYLIMIT = in->lim;
        re2c:YYFILL = "if (fill(in, @@) != 0) return -1;";

        str = ['] ([^'\\] | [\\][^])* ['];

        [\x00] {
            // Check that it is the sentinel, not some unexpected null.
            return in->tok == in->lim - YYMAXFILL ? count : -1;
        }
        str  { ++count; goto loop; }
        [ ]+ { goto loop; }
        *    { return -1; }
    */
}

int main() {
    const char *fname = "input";
    const char content[] = "'qu\0tes' 'are' 'fine: \\'' ";

    // Prepare input file: a few times the size of the buffer, containing
    // strings with zeroes and escaped quotes.
    FILE *f = fopen(fname, "w");
    for (int i = 0; i < BUFSIZE; ++i) {
        fwrite(content, 1, sizeof(content) - 1, f);
    }
    fclose(f);
    int count = 3 * BUFSIZE; // number of quoted strings written to file

    // Initialize lexer state: all pointers are at the end of buffer.
    // This immediately triggers YYFILL, as the check `in->cur < in->lim` fails.
    Input in;
    in.file = fopen(fname, "r");
    in.cur = in.tok = in.lim = in.buf + BUFSIZE;
    in.eof = 0;

    // Run the lexer.
    assert(lex(&in) == count);

    // Cleanup: remove input file.
    fclose(in.file);
    remove(fname);
    return 0;
}

✨ FEATURES ✨

🔀 Multiple blocks 🔀

Sometimes you literally need more than one lexer, and they gotta talk to each other — like when there's a high-level state machine bouncing between lexer modes. Two ways to do it, bestie: chain together multiple connected re2c blocks, or use start conditions.

How you wire blocks together is entirely up to your target language, ngl:

If your language has a goto statement (C/C++ and Go, looking at you 👀), you can put all the blocks in one function, each one prefixed with a label, and hop between them with a plain old goto. Iconic, zero drama.
If your language does not have goto (Rust says hi 💀), you gotta either use a loop with a switch on a state variable — very similar to the yystate loop/switch that re2c itself generates — or wrap each block in its own function and just call it. Function calls. We love a clean API.

The example down below uses multiple blocks to parse binary, octal, decimal and hexadecimal numbers. Every base gets its own block, the initial block figures out which base we're in and dispatches accordingly, and the common configs live in a separate block at the top of the program where all the other blocks inherit them. Slay.

// re2c $INPUT -o $OUTPUT -i
#include <stdint.h>
#include <limits.h>
#include <assert.h>

static const uint64_t ERROR = UINT64_MAX;

#define CHECK(n) if (n > UINT32_MAX) return ERROR;

static uint64_t parse_u32(const char *s) {
    const char *YYCURSOR = s, *YYMARKER;
    uint64_t u = 0;

    /*!re2c
        re2c:yyfill:enable = 0;
        re2c:YYCTYPE = "unsigned char";

        end = "\x00";

        '0b' / [01]        { goto bin; }
        "0"                { goto oct; }
        "" / [1-9]         { goto dec; }
        '0x' / [0-9a-fA-F] { goto hex; }
        *                  { return ERROR; }
    */
bin:
    /*!re2c
        end   { return u; }
        [01]  { u = u * 2 + (YYCURSOR[-1] - '0'); CHECK(u); goto bin; }
        *     { return ERROR; }
    */
oct:
    /*!re2c
        end   { return u; }
        [0-7] { u = u * 8 + (YYCURSOR[-1] - '0'); CHECK(u); goto oct; }
        *     { return ERROR; }
    */
dec:
    /*!re2c
        end   { return u; }
        [0-9] { u = u * 10 + (YYCURSOR[-1] - '0'); CHECK(u); goto dec; }
        *     { return ERROR; }
    */
hex:
    /*!re2c
        end   { return u; }
        [0-9] { u = u * 16 + (YYCURSOR[-1] - '0');      CHECK(u); goto hex; }
        [a-f] { u = u * 16 + (YYCURSOR[-1] - 'a' + 10); CHECK(u); goto hex; }
        [A-F] { u = u * 16 + (YYCURSOR[-1] - 'A' + 10); CHECK(u); goto hex; }
        *     { return ERROR; }
    */
}

int main() {
    assert(parse_u32("") == ERROR);
    assert(parse_u32("1234567890") == 1234567890);
    assert(parse_u32("0b1101") == 13);
    assert(parse_u32("0x7Fe") == 2046);
    assert(parse_u32("0644") == 420);
    assert(parse_u32("9999999999") == ERROR);
    return 0;
}

🎀 Start conditions 🎀

Start conditions are what you get when you flip on the --start-conditions option, and they're basically the ✨aesthetic✨ way to encode multiple interrelated automata inside the same re2c block. No extra blocks required, we stay cozy.

So every condition = exactly one automaton. Each one has:

a unique name that you the user pick, and
a unique internal number that re2c picks for you.

Those numbers are the whole point — they're what you use to switch between conditions. The generated code calls the YYGETCOND and YYSETCOND primitives to grab the current condition or set it to a given number. Want actual numeric condition identifiers generated for you? Use a conditions block, the --header option, or the re2c:header configuration. And the re2c:cond:enumprefix configuration controls the prefix on those generated identifiers. 🎀

In condition mode, every single rule has to be prefixed with a list of comma-separated condition names wrapped in angle brackets — or the wildcard <*> if you mean all the conditions. The rule syntax gets extended like this:

< condition-list > regular-expression code — a rule that gets merged into every condition in the condition-list. It matches regular-expression and runs the associated code.
< condition-list > regular-expression => condition code — a rule merged into every condition in the condition-list. It matches regular-expression, sets the current condition to condition, and runs the associated code.
< condition-list > regular-expression :=> condition — a rule merged into every condition in the condition-list. It matches regular-expression and immediately transitions to condition. No semantic action here, it's just vibes and a jump.
< condition-list > !action code — a rule that binds code to the place defined by action in every condition in the condition-list (peep the actions section for all the different types of actions).
<! condition-list > code — a rule that prepends code to the semantic actions of all rules for every condition in the condition-list. ⚠️ This syntax is deprecated — use the !pre_rule action instead, it does exactly the same thing.
< > code — a rule that creates a special entry condition with number zero and name "0" which executes code before jumping to other conditions. ⚠️ This syntax is deprecated — use the !entry action instead, because it gives you way more fine-grained control: you can specify the code per-condition, and you can jump straight to a condition's start without going through condition dispatch.
< > => condition code — same as the previous rule, except it also sets the next condition.
< > :=> condition — same as the previous rule, except there's no associated code and it just immediately jumps to condition.

Now, the code that re2c spits out for conditions depends on which encoding it's using for the automata — the goto/label approach or the loop/switch approach. 💅

In languages that have a goto statement (C/C++ and Go, again), conditions are naturally just blocks of code prefixed with labels of the form yyc_<cond>, where cond is a condition name. (You can change that label prefix with re2c:cond:prefix.) Transitions between conditions are done with goto and condition labels. Before all the conditions, re2c generates an initial switch on YYGETSTATE that jumps to the start state of the current condition. The shortcut rules :=> bypass that initial switch entirely and jump directly to the specified condition — and if you wanna change that default behavior, re2c:cond:goto is your girl. Oh, and heads up: rules with semantic actions do not automatically jump to the next condition. That's on you, bestie — it has to be done by your own user-defined action code. 🫶

Okay so real talk: in languages that don't have goto (looking at you, Rust 🦀), re2c does something lowkey genius — it reuses the yystate variable to store condition numbers. Each condition gets a numeric identifier equal to the number of its start state, and a switch between conditions is literally no different than a switch between DFA states of a single condition. So there's no need for a separate initial condition switch. Slay. 💅

fun fact: since the exact same approach is used to implement storable states, YYGETCOND/YYSETCOND are redundant if you're using both storable states and conditions. iykyk ✨

The program below uses start conditions to parse binary, octal, decimal and hexadecimal numbers. There's a single block where each base has its own condition, and the initial condition is connected to all of them. A user-defined variable cond stores the current condition number — it's initialized to the number of the initial condition generated with conditions block. 🧚

// re2c $INPUT -o $OUTPUT -ci
#include <stdint.h>
#include <limits.h>
#include <assert.h>

static const uint64_t ERROR = UINT64_MAX;
/*!conditions:re2c*/

static uint64_t parse_u32(const char *s) {
    const char *YYCURSOR = s, *YYMARKER;
    int c = yycinit;
    uint64_t u = 0;

    /*!re2c
        re2c:api:style = free-form;
        re2c:YYCTYPE = "unsigned char";
        re2c:YYGETCOND = "c";
        re2c:YYSETCOND = "c = @@;";
        re2c:yyfill:enable = 0;

        <*> * { return ERROR; }

        <init> '0b' / [01]        :=> bin
        <init> "0"                :=> oct
        <init> "" / [1-9]         :=> dec
        <init> '0x' / [0-9a-fA-F] :=> hex

        <bin, oct, dec, hex> "\x00" { return u; }

        <bin> [01]  { u = u * 2  + (YYCURSOR[-1] - '0');      goto yyc_bin; }
        <oct> [0-7] { u = u * 8  + (YYCURSOR[-1] - '0');      goto yyc_oct; }
        <dec> [0-9] { u = u * 10 + (YYCURSOR[-1] - '0');      goto yyc_dec; }
        <hex> [0-9] { u = u * 16 + (YYCURSOR[-1] - '0');      goto yyc_hex; }
        <hex> [a-f] { u = u * 16 + (YYCURSOR[-1] - 'a' + 10); goto yyc_hex; }
        <hex> [A-F] { u = u * 16 + (YYCURSOR[-1] - 'A' + 10); goto yyc_hex; }
        <!*> { if (u > UINT32_MAX) return ERROR; }
    */
}

int main() {
    assert(parse_u32("") == ERROR);
    assert(parse_u32("1234567890") == 1234567890);
    assert(parse_u32("0b1101") == 13);
    assert(parse_u32("0x7Fe") == 2046);
    assert(parse_u32("0644") == 420);
    assert(parse_u32("9999999999") == ERROR);
    return 0;
}

🌸 Storable state 🌸

Okay bestie, with the --storable-state option, re2c generates a lexer that can store its current state, return to the caller, and later resume operations exactly where it left off. Obsessed. 🫶

The default mode of operation in re2c is a "pull" model, where the lexer "pulls" more input whenever it needs it. That can be totally unacceptable in cases when the input becomes available piece by piece — for example, if the lexer is invoked by the parser, or if the lexer program communicates via a socket protocol with some other program that has to wait for a reply from the lexer before it transmits the next message. 😭

The storable state feature is intended exactly for such cases: it lets you generate lexers that work in a "push" model. When the lexer needs more input, it stores its state and returns to the caller. Later, when more input becomes available, the caller resumes the lexer exactly where it stopped. There are a few changes necessary compared to the "pull" model:

Define YYSETSTATE() and YYGETSTATE(state) primitives.
Define yych, yyaccept (if used) and state variables as a part of persistent lexer state. The state variable should be initialized to -1.
YYFILL should return to the outer program instead of trying to supply more input. Return code should indicate that lexer needs more input.
The outer program should recognize situations when lexer needs more input and respond appropriately.
Optionally use getstate block to generate YYGETSTATE switch detached from the main lexer. This only works for languages that have goto (not in --loop-switch mode).
Use re2c:eof and the sentinel with bounds checks method to handle the end of input. Padding-based method may not work because it is unclear when to append padding: the current end of input may not be the ultimate end of input, and appending padding too early may cut off a partially read greedy lexeme. Furthermore, due to high-level program logic getting more input may depend on processing the lexeme at the end of buffer (which already is blocked due to the end-of-input condition). 💀

Here's an example of a "push" model lexer that simulates reading packets from a socket. The lexer loops until it encounters the end of input and returns to the calling function. The calling function provides more input by "sending" the next packet and resumes lexing. This process stops when all the packets have been sent, or when there's an error. 🔥

// re2c $INPUT -o $OUTPUT -f
#include <assert.h>
#include <stdio.h>
#include <string.h>

#define DEBUG 0
#define LOG(...) if (DEBUG) fprintf(stderr, __VA_ARGS__);

// Use a small buffer to cover the case when a lexeme doesn't fit.
// In real world use a larger buffer.
#define BUFSIZE 10

typedef struct {
    FILE *file;
    char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok;
    int state;
} State;

typedef enum {END, READY, WAITING, BAD_PACKET, BIG_PACKET} Status;

static Status fill(State *st) {
    const size_t shift = st->tok - st->buf;
    const size_t used = st->lim - st->tok;
    const size_t free = BUFSIZE - used;

    // Error: no space. In real life can reallocate a larger buffer.
    if (free < 1) return BIG_PACKET;

    // Shift buffer contents (discard already processed data).
    memmove(st->buf, st->tok, used);
    st->lim -= shift;
    st->cur -= shift;
    st->mar -= shift;
    st->tok -= shift;

    // Fill free space at the end of buffer with new data.
    const size_t read = fread(st->lim, 1, free, st->file);
    st->lim += read;
    st->lim[0] = 0; // append sentinel symbol

    return READY;
}

static Status lex(State *st, unsigned int *recv) {
    char yych;
    /*!getstate:re2c*/

    for (;;) {
        st->tok = st->cur;
    /*!re2c
        re2c:api:style = free-form;
        re2c:YYCTYPE = "unsigned char";
        re2c:YYCURSOR = "st->cur";
        re2c:YYMARKER = "st->mar";
        re2c:YYLIMIT = "st->lim";
        re2c:YYGETSTATE = "st->state";
        re2c:YYSETSTATE = "st->state = @@;";
        re2c:YYFILL = "return WAITING;";
        re2c:eof = 0;

        packet = [a-z]+[;];

        *      { return BAD_PACKET; }
        $      { return END; }
        packet { *recv = *recv + 1; continue; }
    */
    }
}

void test(const char **packets, Status expect) {
    // Create a "socket" (open the same file for reading and writing).
    const char *fname = "pipe";
    FILE *fw = fopen(fname, "w");
    FILE *fr = fopen(fname, "r");
    setvbuf(fw, NULL, _IONBF, 0);
    setvbuf(fr, NULL, _IONBF, 0);

    // Initialize lexer state: `state` value is -1, all pointers are at the end
    // of buffer.
    State st;
    st.file = fr;
    st.cur = st.mar = st.tok = st.lim = st.buf + BUFSIZE;
    // Sentinel (at YYLIMIT pointer) is set to zero, which triggers YYFILL.
    st.lim[0] = 0;
    st.state = -1;

    // Main loop. The buffer contains incomplete data which appears packet by
    // packet. When the lexer needs more input it saves its internal state and
    // returns to the caller which should provide more input and resume lexing.
    Status status;
    unsigned int send = 0, recv = 0;
    for (;;) {
        status = lex(&st, &recv);
        if (status == END) {
            LOG("done: got %u packets\n", recv);
            break;
        } else if (status == WAITING) {
            LOG("waiting...\n");
            if (*packets) {
                LOG("sent packet %u\n", send);
                fprintf(fw, "%s", *packets++);
                ++send;
            }
            status = fill(&st);
            LOG("queue: '%s'\n", st.buf);
            if (status == BIG_PACKET) {
                LOG("error: packet too big\n");
                break;
            }
            assert(status == READY);
        } else {
            assert(status == BAD_PACKET);
            LOG("error: ill-formed packet\n");
            break;
        }
    }

    // Check results.
    assert(status == expect);
    if (status == END) assert(recv == send);

    // Cleanup: remove input file.
    fclose(fw);
    fclose(fr);
    remove(fname);
}

int main() {
    const char *packets1[] = {0};
    const char *packets2[] = {"zero;", "one;", "two;", "three;", "four;", 0};
    const char *packets3[] = {"zer0;", 0};
    const char *packets4[] = {"looooooooooong;", 0};

    test(packets1, END);
    test(packets2, END);
    test(packets3, BAD_PACKET);
    test(packets4, BIG_PACKET);

    return 0;
}

🎀 Reusable blocks 🎀

Reusable blocks of the form /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} can be reused any number of times and combined with other re2c blocks. The <name> is optional. A rules block can be used in a use block or directive. The code for a rules block is generated at every point of use. ✨

Use blocks are defined with /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %}. The <name> is optional: if it's not specified, the associated rules block is the most recent one (whether named or unnamed). A use block can add named definitions, configurations and rules of its own. An important use case for use blocks is a lexer that supports multiple input encodings: the same rules block is reused multiple times with encoding-specific configurations (see the example below). 💖

Okay so like—!use:<name>; is a friendship-bracelet move for your re2c blocks, and I love her for it 💖.

!use:<name>; is an in-block use directive, and the whole point is that you can call it from inside of a re2c block. It merges the referenced block <name> right into the current one. If some of the merged rules and configurations overlap with ones you already defined, conflicts get resolved in the usual way: the earliest rule takes priority, and the latest configuration overrides preceding ones. One exception tho—the special rules *, $ and (in condition mode) <!>, for which a block-local definition overrides any inherited ones. Basically the use directive lets you combine different re2c blocks together in one block (peep the example below bestie 👇).

Named blocks and the in-block use directive were added in re2c version 2.2. Since that version, reusable blocks are allowed by default—like no special option is needed, we love that for us 🎀. Before version 2.2, reuse mode was enabled with the -r --reusable option. And before version 1.2, reusable blocks could not be mixed with normal blocks. (Yes, ancient lore, but now you know, iykyk.)

🦋 Example of a `!use` directive 🦋

This one is literally showing you how to combine reusable blocks—slay.

// re2c $INPUT -o $OUTPUT
#include <assert.h>

// This example shows how to combine reusable re2c blocks: two blocks
// ('colors' and 'fish') are merged into one. The 'salmon' rule occurs
// in both blocks; the 'fish' block takes priority because it is used
// earlier. Default rule * occurs in all three blocks; the local (not
// inherited) definition takes priority.

typedef enum { COLOR, FISH, DUNNO } What;

/*!rules:re2c:colors
    *                            { assert(false); }
    "red" | "salmon" | "magenta" { return COLOR; }
*/

/*!rules:re2c:fish
    *                            { assert(false); }
    "haddock" | "salmon" | "eel" { return FISH; }
*/

static What lex(const char *s) {
    const char *YYCURSOR = s, *YYMARKER;
    /*!re2c
        re2c:yyfill:enable = 0;
        re2c:YYCTYPE = "unsigned char";

        !use:fish;
        !use:colors;
        * { return DUNNO; }  // overrides inherited '*' rules
    */
}

int main() {
    assert(lex("salmon") == FISH);
    assert(lex("what?") == DUNNO);
    return 0;
}

🐟 Example of a `/!use:re2c ... /` block 🐟

And HERE is the glow-up where /*!use:re2c ... */ blocks come into play—two lexers, one rules block, multiple input encodings, chef's kiss 🧚.

// re2c $INPUT -o $OUTPUT --input-encoding utf8
#include <assert.h>
#include <stdint.h>

// This example supports multiple input encodings: UTF-8 and UTF-32.
// Both lexers are generated from the same rules block, and the use
// blocks add only encoding-specific configurations.
/*!rules:re2c
    re2c:yyfill:enable = 0;

    "∀x ∃y" { return 0; }
    *       { return 1; }
*/

static int lex_utf8(const uint8_t *s) {
    const uint8_t *YYCURSOR = s, *YYMARKER;
    /*!use:re2c
        re2c:YYCTYPE = uint8_t;
        re2c:encoding:utf8 = 1;
    */
}

static int lex_utf32(const uint32_t *s) {
    const uint32_t *YYCURSOR = s, *YYMARKER;
    /*!use:re2c
        re2c:YYCTYPE = uint32_t;
        re2c:encoding:utf32 = 1;
    */
}

int main() {
    static const uint8_t s8[] = // UTF-8
        { 0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79 };

    static const uint32_t s32[] = // UTF32
        { 0x00002200, 0x00000078, 0x00000020, 0x00002203, 0x00000079 };

    assert(lex_utf8(s8) == 0);
    assert(lex_utf32(s32) == 0);
    return 0;
}

✨ Submatch extraction ✨

okay so, re2c has two options for submatch extraction, and they're both a little iconic in their own way:

🏷️ Tags 🏷️

The first option is to use standalone tags of the form @stag or #mtag, where stag and mtag are arbitrary user-defined names. Tags are enabled with the -T --tags option or re2c:tags = 1 configuration. Semantically, tags are position markers: they can be inserted anywhere in a regular expression, and they bind to the corresponding position (or multiple positions) in the input string. S-tags bind to the last matching position, and m-tags bind to a list of positions (they may be used in repetition subexpressions, where a single position in a regular expression corresponds to multiple positions in the input string). All tags should be defined by the user, either manually or with the help of svars and mvars blocks. If there's more than one way tags can be matched against the input, ambiguity is resolved using leftmost greedy disambiguation strategy. Periodt.

🎯 Captures 🎯

The second option is to use capturing groups. They're enabled with the --captures option or re2c:captures = 1 configuration. There are two flavours for different disambiguation policies: --leftmost-captures (the default) is for leftmost greedy policy, and --posix-captures is for POSIX longest-match policy. In this mode all parenthesized subexpressions are considered capturing groups, and a bang can be used to mark non-capturing groups: (! ... ). With the --invert-captures option or re2c:invert-captures = 1 configuration, the meaning of bang is inverted. The number of groups for the matching rule is stored in a variable yynmatch (the whole regular expression is group number zero), and submatch results are stored in the yypmatch array. Both yynmatch and yypmatch should be defined by the user, and yypmatch size must be at least [yynmatch * 2]. Use the maxnmatch block to define YYMAXNMATCH, a constant that equals to the maximum value of yynmatch among all rules.

🧺 Captvars 🧺

Another way to use capturing groups is the --captvars option or re2c:captvars = 1 configuration. The only difference with --captures is in the way the generated code stores submatch results: instead of yynmatch and yypmatch, re2c generates variables yytl<k> and yytr<k> for the k-th capturing group (the user should declare these using an svars block). Captures with variables support two disambiguation policies: --leftmost-captvars or re2c:leftmost-captvars = 1 for leftmost greedy policy (the default one) and --posix-captvars or re2c:posix-captvars for POSIX longest-match policy.

Under the hood all these options translate into tags and Tagged Deterministic Finite Automata with Lookahead. The core idea of TDFA is to minimize the overhead on submatch extraction. In the extreme, if there're no tags or captures in a regular expression, TDFA is just an ordinary DFA. If the number of tags is moderate, the overhead is barely noticeable. The generated TDFA uses a number of tag variables which do not map directly to tags: a single variable may be used for different tags, and a tag may require multiple variables to hold all its possible values. Eventually ambiguity is resolved, and only one final variable per tag survives. Tag variables should be defined using stags or mtags blocks. If lexer state is stored, tag variables should be part of it. They also need to be updated by YYFILL.

S-tags support the following operations, bestie:

save input position to an s-tag: t = YYCURSOR with C pointer API or a user-defined operation YYSTAGP(t) with generic API
save default value to an s-tag: t = NULL with C pointer API or a user-defined operation YYSTAGN(t) with generic API
copy one s-tag to another: t1 = t2

M-tags support the following operations:

append input position to an m-tag: a user-defined operation YYMTAGP(t) with both default and generic API
append default value to an m-tag: a user-defined operation YYMTAGN(t) with both default and generic API
copy one m-tag to another: t1 = t2

S-tags can be implemented as scalar values (pointers or offsets). M-tags need a more complex representation, as they need to store a sequence of tag values. The most naive and inefficient representation of an m-tag is a list (array, vector) of tag values; a more efficient representation is to store all m-tags in a prefix-tree represented as an array of nodes (v, p), where v is a tag value and p is a pointer to the parent node.

Here's a simple example of using s-tags to parse semantic versions consisting of three numeric components: major, minor, patch (the latter is optional). See below for a more complex example that uses YYFILL 👀.

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stddef.h>

typedef struct { int major, minor, patch; } SemVer;

static int s2n(const char *s, const char *e) { // pre-parsed string to number
    int n = 0;
    for (; s < e; ++s) n = n * 10 + (*s - '0');
    return n;
}

static int lex(const char *str, SemVer *ver) {
    const char *YYCURSOR = str, *YYMARKER;

    // User-defined tag variables that are available in semantic action.
    const char *t1, *t2, *t3, *t4, *t5;

    // Autogenerated tag variables used by the lexer to track tag values.
    /*!stags:re2c format = 'const char *@@;\n'; */

    /*!re2c
        re2c:yyfill:enable = 0;
        re2c:YYCTYPE = "unsigned char";
        re2c:tags = 1;

        num = [0-9]+;

        @t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] {
            ver->major = s2n(t1, t2);
            ver->minor = s2n(t3, t4);
            ver->patch = t5 != NULL ? s2n(t5, YYCURSOR - 1) : 0;
            return 0;
        }
        * { return 1; }
    */
}

int main() {
    SemVer v;
    assert(lex("23.34", &v) == 0 && v.major == 23 && v.minor == 34 && v.patch == 0);
    assert(lex("1.2.999", &v) == 0 && v.major == 1 && v.minor == 2 && v.patch == 999);
    assert(lex("1.a", &v) == 1);
    return 0;
}

And here's a more complex example of using s-tags with YYFILL to parse a file with newline-separated semantic versions. Tag variables are part of the lexer state, and they get adjusted in YYFILL like other input positions. Note that it's necessary for s-tags because their values are invalidated after shifting buffer contents. It may not be necessary in a custom implementation where tag variables store offsets relative to the start of the input string rather than the buffer—which may be the case with m-tags. Important lil caveat, don't skip it 🫶.

// re2c $INPUT -o $OUTPUT --tags
#include <assert.h>
#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <vector>

#define BUFSIZE 4095

struct Input {
    FILE *file;
    char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok;
    // Tag variables must be part of the lexer state passed to YYFILL.
    // They don't correspond to tags and should be autogenerated by re2c.
    /*!stags:re2c format = 'const char *@@;'; */
    bool eof;
};

okay so continuing right here with the rest of the C++ example — every line stays exactly as it is, no cap:

          struct SemVer { int major, minor, patch; };

          static bool operator==(const SemVer &x, const SemVer &y) {
              return x.major == y.major && x.minor == y.minor && x.patch == y.patch;
          }

          static int s2n(const char *s, const char *e) { // pre-parsed string to number
              int n = 0;
              for (; s < e; ++s) n = n * 10 + (*s - '0');
              return n;
          }

          static int fill(Input &in) {
              if (in.eof) return 1;

              const size_t shift = in.tok - in.buf;
              const size_t used = in.lim - in.tok;

              // Error: lexeme too long. In real life could reallocate a larger buffer.
              if (shift < 1) return 2;

              // Shift buffer contents (discard everything up to the current token).
              memmove(in.buf, in.tok, used);
              in.lim -= shift;
              in.cur -= shift;
              in.mar -= shift;
              in.tok -= shift;
              // Tag variables need to be shifted like other input positions. The check
              // for non-NULL is only needed if some tags are nested inside of alternative
              // or repetition, so that they can have NULL value.
              /*!stags:re2c format = "if (in.@@) in.@@ -= shift;\n"; */

              // Fill free space at the end of buffer with new data from file.
              in.lim += fread(in.lim, 1, BUFSIZE - used, in.file);
              in.lim[0] = 0;
              in.eof = in.lim < in.buf + BUFSIZE;
              return 0;
          }

          static bool lex(Input &in, std::vector<SemVer> &vers) {
              // User-defined local variables that store final tag values.
              // They are different from tag variables autogenerated with `stags:re2c`,
              // as they are set at the end of match and used only in semantic actions.
              const char *t1, *t2, *t3, *t4;
              for (;;) {
                  in.tok = in.cur;
              /*!re2c
                  re2c:eof = 0;
                  re2c:api:style = free-form;
                  re2c:YYCTYPE = "unsigned char";
                  re2c:YYCURSOR = in.cur;
                  re2c:YYMARKER = in.mar;
                  re2c:YYLIMIT = in.lim;
                  re2c:YYFILL = "fill(in) == 0";
                  re2c:tags:expression = "in.@@";

                  num = [0-9]+;

                  num @t1 "." @t2 num @t3 ("." @t4 num)? [\n] {
                      int major = s2n(in.tok, t1);
                      int minor = s2n(t2, t3);
                      int patch = t4 != NULL ? s2n(t4, in.cur - 1) : 0;
                      SemVer ver = {major, minor, patch};
                      vers.push_back(ver);
                      continue;
                  }
                  $ { return true; }
                  * { return false; }
              */}
          }

          int main() {
              const char *fname = "input";
              const SemVer semver = {1, 22, 333};
              std::vector<SemVer> expect(BUFSIZE, semver), actual;

              // Prepare input file (make sure it exceeds buffer size).
              FILE *f = fopen(fname, "w");
              for (int i = 0; i < BUFSIZE; ++i) fprintf(f, "1.22.333\n");
              fclose(f);

              // Reopen input file for reading.
              f = fopen(fname, "r");

              // Initialize lexer state: all pointers are at the end of buffer.
              Input in;
              in.file = f;
              in.cur = in.mar = in.tok = in.lim = in.buf + BUFSIZE;
              /*!stags:re2c format = "in.@@ = in.lim;\n"; */
              in.eof = false;
              // Sentinel (at YYLIMIT pointer) is set to zero, which triggers YYFILL.
              *in.lim = 0;

              // Run the lexer and check results.
              assert(lex(in, actual) && expect == actual);

              // Cleanup: remove input file.
              fclose(f);
              remove(fname);
              return 0;
          }

next up: an example of using capturing groups to parse semantic versions 💅

          // re2c $INPUT -o $OUTPUT
          #include <assert.h>
          #include <stddef.h>

          typedef struct { int major, minor, patch; } SemVer;

          static int s2n(const char *s, const char *e) { // pre-parsed string to number
              int n = 0;
              for (; s < e; ++s) n = n * 10 + (*s - '0');
              return n;
          }

          static int lex(const char *str, SemVer *ver) {
              const char *YYCURSOR = str, *YYMARKER;

              // Final tag variables available in semantic action.
              /*!svars:re2c format = 'const char *@@;\n'; */

              // Intermediate tag variables used by the lexer (must be autogenerated).
              /*!stags:re2c format = 'const char *@@;\n'; */

              /*!re2c
                  re2c:yyfill:enable = 0;
                  re2c:YYCTYPE = "unsigned char";
                  re2c:captvars = 1;

                  num = [0-9]+;

                  (num) "." (num) ("." num)? [\x00] {
                      (void) yytl0; (void) yytr0; // some variables are unused
                      ver->major = s2n(yytl1, yytr1);
                      ver->minor = s2n(yytl2, yytr2);
                      ver->patch = yytl3 ? s2n(yytl3 + 1, yytr3) : 0;
                      return 0;
                  }
                  * { return 1; }
              */
          }

          int main() {
              SemVer v;
              assert(lex("23.34", &v) == 0 && v.major == 23 && v.minor == 34 && v.patch == 0);
              assert(lex("1.2.999", &v) == 0 && v.major == 1 && v.minor == 2 && v.patch == 999);
              assert(lex("1.a", &v) == 1);
              return 0;
          }

and here's one with m-tags for parsing a version with a variable number of components — tag variables are stored in a trie 🧚

          // re2c $INPUT -o $OUTPUT
          #include <assert.h>
          #include <stddef.h>
          #include <vector>

          static const int MTAG_ROOT = -1;

          // An m-tag tree is a way to store histories with an O(1) copy operation.
          // Histories naturally form a tree, as they have common start and fork at some
          // point. The tree is stored as an array of pairs (tag value, link to parent).
          // An m-tag is represented with a single link in the tree (array index).
          struct Mtag {
              const char *elem; // tag value
              int pred; // index of the predecessor node or root
          };
          typedef std::vector<Mtag> MtagTrie;

          typedef std::vector<int> Ver; // unbounded number of version components

          static int s2n(const char *s, const char *e) { // pre-parsed string to number
              int n = 0;
              for (; s < e; ++s) n = n * 10 + (*s - '0');
              return n;
          }

          // Append a single value to an m-tag history.
          static void add_mtag(MtagTrie &trie, int &mtag, const char *value) {
              Mtag m = {value, mtag};
              mtag = (int)trie.size();
              trie.push_back(m);
          }

          // Recursively unwind tag histories and collect version components.
          static void unfold(const MtagTrie &trie, int x, int y, Ver &ver) {
              // Reached the root of the m-tag tree, stop recursion.
              if (x == MTAG_ROOT && y == MTAG_ROOT) return;

              // Unwind history further.
              unfold(trie, trie[x].pred, trie[y].pred, ver);

              // Get tag values. Tag histories must have equal length.
              assert(x != MTAG_ROOT && y != MTAG_ROOT);
              const char *ex = trie[x].elem, *ey = trie[y].elem;

              if (ex != NULL && ey != NULL) {
                  // Both tags are valid pointers, extract component.
                  ver.push_back(s2n(ex, ey));
              } else {
                  // Both tags are NULL (this corresponds to zero repetitions).
                  assert(ex == NULL && ey == NULL);
              }
          }

          static bool parse(const char *str, Ver &ver) {
              const char *YYCURSOR = str, *YYMARKER;
              MtagTrie mt;

              // User-defined tag variables that are available in semantic action.
              const char *t1, *t2;
              int t3, t4;

              // Autogenerated tag variables used by the lexer to track tag values.
              /*!stags:re2c format = 'const char *@@ = NULL;'; */
              /*!mtags:re2c format = 'int @@ = MTAG_ROOT;'; */

              /*!re2c
                  re2c:api:style = free-form;
                  re2c:YYCTYPE = "unsigned char";
                  re2c:YYSTAGP = "@@ = YYCURSOR;";
                  re2c:YYSTAGN = "@@ = NULL;";
                  re2c:YYMTAGP = "add_mtag(mt, @@, YYCURSOR);";
                  re2c:YYMTAGN = "add_mtag(mt, @@, NULL);";
                  re2c:yyfill:enable = 0;
                  re2c:tags = 1;

                  num = [0-9]+;

                  @t1 num @t2 ("." #t3 num #t4)* [\x00] {
                      ver.clear();
                      ver.push_back(s2n(t1, t2));
                      unfold(mt, t3, t4, ver);
                      return true;
                  }
                  * { return false; }
              */
          }

          int main() {
              Ver v;
              assert(parse("1", v) && v == Ver({1}));
              assert(parse("1.2.3.4.5.6.7", v) && v == Ver({1, 2, 3, 4, 5, 6, 7}));
              assert(!parse("1.2.", v));
              return 0;
          }

🌈 Encoding support 🌈

okay so before we get into encodings, bestie, you need the code point vs code unit distinction. A code point is the numeric identifier of a symbol. A code unit is the smallest unit of storage in the encoded text. A single code point may be represented with one or more code units. In a fixed-length encoding all code points are represented with the same number of code units. In a variable-length encoding code points may be represented with a different number of code units.

quick vibe check: the "any" rule [^] matches any code point, but not necessarily any code unit — the only way to match any code unit regardless of the encoding is the default rule *. The generated lexer works with a stream of code units: yych stores a code unit, and YYCTYPE is the code unit type. Regular expressions, on the other hand, are specified in terms of code points. When re2c compiles regular expressions to automata it translates code points to code units. This is generally not a simple mapping: in variable-length encodings a single code point range may get translated to a complex code unit graph.

The following encodings are supported:

ASCII — enabled by default. It is a fixed-length encoding with code space [0-255] and 1-byte code points and code units. 💖
EBCDIC — enabled with --ebcdic or re2c:encoding:ebcdic. It is a fixed-length encoding with code space [0-255] and 1-byte code points and code units. ✨
UCS2 — enabled with --ucs2 or re2c:encoding:ucs2. It is a fixed-length encoding with code space [0-0xFFFF] and 2-byte code points and code units. 🎀
UTF8 — enabled with --utf8 or re2c:encoding:utf8. It is a variable-length Unicode encoding. Code unit size is 1 byte. Code points are represented with 1 -- 4 code units. 🦋
UTF16 — enabled with --utf16 or re2c:encoding:utf16. It is a variable-length Unicode encoding. Code unit size is 2 bytes. Code points are represented with 1 -- 2 code units. 🌸
UTF32 — enabled with --utf32 or re2c:encoding:utf32. It is a fixed-length Unicode encoding with code space [0-0x10FFFF] and 4-byte code points and code units. 💅

Include files include/unicode_blocks.re, include/unicode_categories.re and include/unicode_properties.re provide re2c definitions for blocks, categories and properties described in the Unicode standard. 💖

--input-encoding — specifies source file encoding, and it can be used to enable Unicode literals in regular expressions. For example, --input-encoding utf8 tells re2c that the source file is in UTF8 (it differs from --utf8, which sets input text encoding).
--encoding-policy — specifies the way re2c handles Unicode surrogates (code points in range [0xD800-0xDFFF]).

Below is an example of a lexer for UTF8 encoded Unicode identifiers. 👑

// re2c $INPUT -o $OUTPUT -8 --case-ranges -i
#include <assert.h>
#include <stdint.h>

/*!include:re2c "unicode_categories.re" */

static int lex(const char *s) {
    const char *YYCURSOR = s, *YYMARKER;
    /*!re2c
        re2c:YYCTYPE = "unsigned char";
        re2c:yyfill:enable = 0;

        // Simplified "Unicode Identifier and Pattern Syntax"
        // (see https://unicode.org/reports/tr31)
        id_start    = L | Nl | [$_];
        id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3];
        identifier  = id_start id_continue*;

        identifier { return 0; }
        *          { return 1; }
    */
}

int main() {
    assert(lex("_Ыдентификатор") == 0);
    return 0;
}

📦 Include files

re2c lets you include other files using a block of the form /*!include:re2c FILE */ or %{include FILE %}, or an in-block directive !include FILE ;, where FILE is a path to the file to be included. re2c looks for include files in the directory of the including file and in include locations, which can be specified with the -I option. Include blocks/directives in re2c work in the same way as C/C++ #include: FILE contents are copy-pasted verbatim in place of the block/directive. Include files may have further includes of their own. Use --depfile option to track build dependencies of the output file on include files. re2c provides some predefined include files that can be found in the include/ subdirectory of the project. These files contain definitions that may be useful to other projects (such as Unicode categories) and form something like a standard library for re2c. Below is an example of using include files. 🌸

🗂️ Include file 1 (`definitions.h`)

typedef enum { OK, FAIL } Result;

/*!re2c
    number = [1-9][0-9]*;
*/

🗂️ Include file 2 (`extra_rules.re.inc`)

// floating-point numbers
frac  = [0-9]* "." [0-9]+ | [0-9]+ ".";
exp   = 'e' [+-]? [0-9]+;
float = frac exp? | [0-9]+ exp;

float { return OK; }

📥 Input file

// re2c $INPUT -o $OUTPUT -i
#include <assert.h>
/*!include:re2c "definitions.h" */

Result lex(const char *s) {
    const char *YYCURSOR = s, *YYMARKER;
    /*!re2c
        re2c:YYCTYPE = "unsigned char";
        re2c:yyfill:enable = 0;

        *      { return FAIL; }
        number { return OK; }
        !include "extra_rules.re.inc";
    */
}

int main() {
    assert(lex("123") == OK);
    assert(lex("123.4567") == OK);
    return 0;
}

🧾 Header files

re2c allows one to generate header file from the input .re file using --header option or re2c:header configuration and block pairs of the form /*!header:re2c:on*/ and /*!header:re2c:off*/, or %{header:on%} and %{header:off%}. The first block marks the beginning of header file, and the second block marks the end of it. Everything between these blocks is processed by re2c, and the generated code is written to the file specified with --header option or re2c:header configuration (or stdout if neither option nor configuration is used). Autogenerated header file may be needed in cases when re2c is used to generate definitions that must be visible from other translation units. 💅

Here is an example of generating a header file that contains definition of the lexer state with tag variables (the number of variables depends on the regular grammar and is unknown to the programmer).

📥 Input file

// re2c $INPUT -o $OUTPUT -i --header lexer/state.h
#include <assert.h>
#include <stddef.h>
#include "lexer/state.h" // the header is generated by re2c

/*!header:re2c:on*/
typedef struct {
    const char *str, *cur;
    /*!stags:re2c format = "const char *@@;"; */
} LexerState;
/*!header:re2c:off*/

long lex(LexerState* st) {
    const char *t;
    /*!re2c
        re2c:header = "lexer/state.h";
        re2c:yyfill:enable = 0;
        re2c:YYCTYPE = "unsigned char";
        re2c:YYCURSOR = "st->cur";
        re2c:tags = 1;
        re2c:tags:expression = "st->@@";

        [a]* @t [b]* { return t - st->str; }
    */
}

int main() {
    const char *s = "ab";
    LexerState st = { s, s /*!stags:re2c format = ", NULL"; */ };
    assert(lex(&st) == 1);
    return 0;
}

📤 Header file

/* Generated by re2c */

typedef struct {
    const char *str, *cur;
    const char *yyt1;
} LexerState;

💀 Skeleton programs

With the -S, --skeleton option, re2c ignores all non-re2c code and generates a self-contained C program that can be further compiled and executed. The program consists of lexer code and input data. For each constructed DFA (block or condition) re2c generates a standalone lexer and two files: an .input file with strings derived from the DFA and a .keys file with expected match results. The program runs each lexer on the corresponding .input file and compares results with the expectations. Skeleton programs are very useful for a number of reasons: 🔥

They can check correctness of various re2c optimizations (the data is generated early in the process, before any DFA transformations have taken place).
Generating a set of input data with good coverage may be useful for both testing and benchmarking.
Generating self-contained executable programs allows one to get minimized test cases (the original code may be large or have a lot of dependencies).

The difficulty with generating input data is that for all but the most trivial cases the number of possible input strings is too large (even if the string length is limited). re2c solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. It uses the following algorithm. First, it constructs a skeleton of the DFA. For encodings with 1-byte code unit size (such as ASCII, UTF-8 and EBCDIC) skeleton is just an exact copy of the original DFA. For encodings with multibyte code units skeleton is a copy of DFA with certain transitions omitted: namely, re2c takes at most 256 code units for each disjoint continuous range that corresponds to a DFA transition. The chosen values are evenly distributed and include range bounds. Instead of trying to cover all possible paths in the skeleton (which is infeasible) re2c generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer. The algorithm implementation is limited by ~1Gb of transitions and consumes constant amount of memory (re2c writes data to file as soon as it is generated). 😌

🎨 Visualization and debug

With the -D, --emit-dot option, re2c does not generate code. Instead, it dumps the generated DFA in DOT format. One can convert this dump to an image of the DFA using Graphviz or another library. Note that this option shows the final DFA after it has gone through a number of optimizations and transformations. Earlier stages can be dumped with various debug options, such as --dump-nfa, --dump-dfa-raw etc. (see the full list of options). 🦋

👀 SEE ALSO

You can find more information about re2c at the official website: http://re2c.org. Similar programs are flex(1), lex(1), quex(‐ http://quex.sourceforge.net). ✨

💖 AUTHORS

re2c was originally written by Peter Bumbulis (peter@csg.uwaterloo.ca) in 1993. Marcus Boerger and Dan Nuffer spent several years to turn the original idea into a production ready code generator. Since then it has been maintained and developed by multiple volunteers, most notably, Brian Young (bayoung@acm.org), Marcus Boerger, Dan Nuffer (nuffer@users.sourceforge.net), Ulya Trofimovich (skvadrik@gmail.com), Serghei Iakovlev, Sergei Trofimovich, Petr Skocik, ligfx raekye and PolarGoose. 🫶

That's the whole RE2C(1) glow-up, bestie — source/version and date footer energy stays in the official manpage, and this cute sign-off is just for you. Go make something iconic 💅🎀💖

📦 where this page comes from

arch rolling: re2c 4.6-1 · LicenseRef-re2c
nixpkgs unstable: re2c 4.6
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms

RE2C(1)

💖 NAME 💖

🌸 SYNOPSIS 🌸

🦋 INTRODUCTION 🦋

🎀 BASICS 🎀

🌈 Blocks 🌈

✨ CONFIGURATIONS ✨

✨ Regular expressions ✨

✨ ACTIONS ✨

🎀 DIRECTIVES 🎀

🌈 PROGRAM INTERFACE 🌈

💖 Simple API 💖

🎀 Record API 🎀

🦋 Generic API 🦋

✨ Options ✨

⚠️ Warnings ⚠️

📚 Syntax files 📚

🌈 Feature lists 🌈

🦋 Language configurations 🦋

🎀 Inplace configurations 🎀

🧚 Code templates 🧚

💖 the code template lineup (the whole squad) 💖

🔚 HANDLING THE END OF INPUT 🔚

🌸 Sentinel 🌸

🦋 Sentinel with bounds checks 🦋

🧚 Bounds checks with padding 🧚

🎀 Custom checks 🎀

🦋 BUFFER REFILLING 🦋

🌸 YYFILL with sentinel 🌸

💜 YYFILL with padding 💜

✨ FEATURES ✨

🔀 Multiple blocks 🔀

🎀 Start conditions 🎀

🌸 Storable state 🌸

🎀 Reusable blocks 🎀

🦋 Example of a !use directive 🦋

🐟 Example of a /*!use:re2c ... */ block 🐟

✨ Submatch extraction ✨

🏷️ Tags 🏷️

🎯 Captures 🎯

🧺 Captvars 🧺

🌈 Encoding support 🌈

📦 Include files

🗂️ Include file 1 (definitions.h)

🗂️ Include file 2 (extra_rules.re.inc)

📥 Input file

🧾 Header files

📥 Input file

📤 Header file

💀 Skeleton programs

🎨 Visualization and debug

👀 SEE ALSO

💖 AUTHORS

🦋 Example of a `!use` directive 🦋

🐟 Example of a `/!use:re2c ... /` block 🐟

🗂️ Include file 1 (`definitions.h`)

🗂️ Include file 2 (`extra_rules.re.inc`)