RE2OCAML(1)

💖 NAME 💖

re2ocaml — generate fast lexical analyzers for OCaml. Yep bestie, this is the manual page for re2ocaml(1) 💅 — the preprocessor that’s lowkey iconic for OCaml lexers.

✨ SYNOPSIS ✨

re2ocaml [ OPTIONS ] [ WARNINGS ] INPUT

Input can be either a file or - for stdin. 🫶

🌸 INTRODUCTION 🌸

Okay so re2ocaml works as a preprocessor. It reads the input file (which is usually a program in OCaml, but can be anything) and looks for blocks of code enclosed in special-form start/end markers. The text outside of these blocks is copied verbatim into the output file. The contents of the blocks are processed by re2ocaml. It translates them to code in OCaml and outputs the generated code in place of the block. ✨

Here’s an example of a small program that checks if a given string contains a decimal number:

(* re2ocaml $INPUT -o $OUTPUT -i *)

open String

type state = {
    yyinput: string;
    mutable yycursor: int;
}

%{
    re2c:YYFN = ["lex;bool", "yyrecord;state"];
    re2c:yyfill:enable = 0;

    [1-9][0-9]* { true }
    *           { false }
%}

let main () =
    let st = {yyinput = "1234"; yycursor = 0}
    in if not (lex st) then raise (Failure "error")

let _ = main ()

In the output re2ocaml replaced the block in the middle with the generated code:

(* Generated by re2ocaml *)
(* re2ocaml $INPUT -o $OUTPUT -i *)

open String

type state = {
    yyinput: string;
    mutable yycursor: int;
}

let rec yy0 (yyrecord : state) : bool =
    let yych = unsafe_get yyrecord.yyinput yyrecord.yycursor in
    yyrecord.yycursor <- yyrecord.yycursor + 1;
    match yych with
        | '1'..'9' -> (yy2 [@tailcall]) yyrecord
        | _ -> (yy1 [@tailcall]) yyrecord

and yy1 (yyrecord : state) : bool =
    false

and yy2 (yyrecord : state) : bool =
    let yych = unsafe_get yyrecord.yyinput yyrecord.yycursor in
    match yych with
        | '0'..'9' ->
            yyrecord.yycursor <- yyrecord.yycursor + 1;
            (yy2 [@tailcall]) yyrecord
        | _ -> (yy3 [@tailcall]) yyrecord

and yy3 (yyrecord : state) : bool =
    true

and lex (yyrecord : state) : bool =
    (yy0 [@tailcall]) yyrecord

let main () =
    let st = {yyinput = "1234"; yycursor = 0}
    in if not (lex st) then raise (Failure "error")

let _ = main ()

🦋 BASICS 🦋

Okay so a re2ocaml program consists of a sequence of blocks intermixed with code in the target language. A block may contain definitions, configurations, rules, actions and directives in any order:

• name = regular-expression ; — a definition binds name to regular-expression. Names may contain alphanumeric characters and underscore. The regular expressions section gives an overview of re2ocaml syntax for regular expressions. Once defined, the name can be used in other regular expressions and in rules. Recursion in named definitions is not allowed, and each name should be defined before it is used. A block inherits named definitions from the global scope. Redefining a name that exists in the current scope is an error. 🚨
• configuration = value ; — a configuration allows one to change re2ocaml behavior and customize the generated code. For a full list of configurations supported by re2ocaml see the configurations section. Depending on a particular configuration, the value can be a keyword, a nonnegative integer number or a one-line string which should be enclosed in double or single quotes unless it consists of alphanumeric characters. A block inherits configurations from the global scope and may redefine them or add new ones. Configurations defined inside of a block affect the whole block, even if they appear at the end of it. 👑
• regular-expression code — a rule binds regular-expression to its semantic action (a block of code in curly braces, or a block of code that starts with := and ends on a newline followed by any non-whitespace character). If the regular-expression matches, the associated code is executed. If multiple rules match, the longest match takes precedence. If multiple rules match the same string, the earliest one takes precedence. There are two special rules: the default rule * and the end of input rule $. Default rule should always be defined, it has the lowest priority regardless of its place in the block, and it matches any code unit (not necessarily a valid character, see the encoding support section). The end of input rule should be defined if the corresponding method for handling the end of input is used. With start conditions rules have more complex syntax. 🔥
• !action code — an action binds a user-defined block of code to a particular place in the generated finite state machine (in the same way as semantic actions bind code to the final states). See the actions section for a full list of predefined actions. ✨
• !directive ; — a directive is one of the special predefined statements. Each directive has a unique purpose. See the directives section for details. 🎀

🎀 Blocks 🎀

Block start and end markers are either /*!re2c and */, or %{ and %} (both styles are supported). Starting from version 2.2 blocks may have optional names that allow them to be referenced in other blocks. There are different kinds of blocks:

• Global block — /*!re2c[:<name>] ... */ or %{[:<name>] ... %}. A global block contains definitions, configurations, rules and directives. re2ocaml compiles regular expressions associated with each rule into a deterministic finite automaton, encodes it in the form of conditional jumps in the target language and replaces the block with the generated code. Names and configurations defined in a global block are added to the global scope and become visible to subsequent blocks. At the start of the program the global scope is initialized with command-line options. 🌈
• Local block — /*!local:re2c[:<name>] ... */ or %{local[:<name>] ... %}. A local block is like a global block, but the names and configurations in it have local scope (they do not affect other blocks). 💜
• Rules block — /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %}. A rules block is like a local block, but it does not generate any code by itself, nor does it add any definitions to the global scope — it is meant to be reused in other blocks. This is a way of sharing code (more details in the reusable blocks section). Prior to re2ocaml version 2.2 rules blocks required -r --reusable option. 🎀
• Use block — /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %}. A use block references a previously defined rules block. If the name is specified, re2ocaml looks for a rules block with this name. Otherwise the most recent rules block is used (either a named or an unnamed one). A use block can add definitions, configurations and rules of its own, which are added to those of the referenced rules block. Prior to re2ocaml version 2.2 use blocks required -r --reusable option. 🦋
• Max block (YYMAXFILL) — /*!max:re2c[:<name1>[:<name2>...]] ... */ or %{max[:<name1>[:<name2>...]] ... %}. A block that generates YYMAXFILL definition. An optional list of block names specifies which blocks should be included when computing YYMAXFILL value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXFILL <n>), or a global variable for Go (var YYMAXFILL int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXFILL. 📏
• Maxnmatch block (YYMAXNMATCH) — /*!maxnmatch:re2c[:<name1>[:<name2>...]] ... */ or %{maxnmatch[:<name1>[:<name2>...]] ... %}. A block that generates YYMAXNMATCH definition (it requires -P --posix-captures option). An optional list of block names specifies which blocks should be included when computing YYMAXNMATCH value (if the list is empty, all blocks are included). By default the generated code is a macro-definition for C (#define YYMAXNMATCH <n>), or a global variable for Go (var YYMAXNMATCH int = <n>). It can be customized with an optional configuration format that specifies a template string where @@{max} (or @@ for short) is replaced with the numeric value of YYMAXNMATCH. 🔢
• Stags/mtags blocks — these specify a template piece of code that is expanded for each s-tag/m-tag variable generated by re2ocaml. Syntax is /*!stags:re2c[:<name1>[:<name2>...]] ... */, /*!mtags:re2c[:<name1>[:<name2>...]] ... */ or %{stags[:<name1>[:<name2>...]] ... %}, %{mtags[:<name1>[:<name2>...]] ... %{. An optional list of block names specifies which blocks should be included when computing the set of tag variables (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag variable. Configuration separator specifies a piece of code used to join the generated format pieces for different tag variables. 🏷️
• Svars/mvars blocks — these specify a template piece of code that is expanded for each s-tag/m-tag that is either explicitly mentioned by the rules (with --tags option) or implicitly generated by re2ocaml (with --captvars or --posix-captvars options). Syntax is /*!svars:re2c[:<name1>[:<name2>...]] ... */, /*!mvars:re2c[:<name1>[:<name2>...]] ... */ or %{svars[:<name1>[:<name2>...]] ... %}, %{mvars[:<name1>[:<name2>...]] ... %{. An optional list of block names specifies which blocks should be included when computing the set of tags (if the list is empty, all blocks are included). There are two optional configurations: format and separator. Configuration format specifies a template string where @@{tag} (or @@ for short) is replaced with the name of each tag. Configuration separator specifies a piece of code used to join the generated format pieces for different tags. 💖
• /*!getstate:re2c[:<name1>[:<name2>...]] ... */ or %{getstate[:<name1>[:<name2>...]] ... %} — okay bestie, this one generates a conditional dispatch on the lexer state (heads up: it requires the --storable-state option, no cap). If you toss in an optional list of block names, that tells re2ocaml which blocks get included in the state dispatch. The default transition goes to the start label of the first block on the list. If the list is empty? Then all blocks get included, and the default transition goes to the first block in the file that has a start label. ✨ Big warning though: this block type is incompatible with the --loop-switch option, because it needs cross-block transitions, and those are literally unsupported without goto or function calls. 💀
• /*!conditions:re2c[:<name1>[:<name2>...]] ... */, /*!types:re2c... */ or %{conditions[:<name1>[:<name2>...]] ... %}, %{types... %} — this block generates the condition enumeration (it requires the --conditions option 🫶). An optional list of block names says which blocks get included when computing the set of conditions (empty list = all blocks included, of course). By default the generated code is an enumeration YYCONDTYPE, but you can customize it with the optional configurations format and separator. The format configuration specifies a template string where @@{cond} (or just @@ for short) gets replaced with the name of each condition, and @@{num} gets replaced with a numeric index of that condition. The separator configuration specifies a piece of code used to join the generated format pieces for different conditions. Obsessed with the flexibility tbh. 💅
• /*!include:re2c <file> */ or %{include <file> %} — this block lets you include <file>, which must be a double-quoted file path, bestie. The contents of the file are literally substituted in place of the block, exactly the same way #include works in C/C++. Iconic. You can pair this block with the --depfile option to generate build system dependencies on the included files. 🔥
• /*!header:re2c:on*/ or %{header:on %} — this marks the start of the header file. Everything after it, up to the following header:off block, is processed by re2ocaml and written to the header file specified with the -t --type-header option. 🎀
• /*!header:re2c:off*/ or %{header:off %} — and this marks the end of the header file that was started with the header:on*/ block. Matching bookends, we love to see it. 🦋
• /*!ignore:re2c ... */ or %{ignore ... %} — a block whose contents are ignored and removed from the output file. Poof, gone, never happened. 🙈

✨ Configurations ✨

Here is the full list of configurations supported by re2ocaml, and yes, we're going through all of them:

• re2c:api, re2c:input — same as the --api option.
• re2c:api:sigil — specify the marker ("sigil") used for argument placeholders in the API primitives. The default is @@. A placeholder starts with the sigil followed by the argument name in curly braces — so like, if the sigil is set to $, then placeholders will have the form ${name}. Single-argument APIs may use shorthand notation without the name in braces. This option can be overridden by options for individual API primitives, e.g. re2c:YYFILL@len for YYFILL. 🫶
• re2c:api:style — specify the API style. Possible values are functions (the default for C) and free-form (the default for Go and Rust). In functions style, API primitives are generated with an argument list in parentheses following the name of the primitive. The arguments are provided only for autogenerated parameters (such as the number of characters passed to YYFILL), but not for the general lexer context, so the primitives behave more like macros in C/C++ or closures in Go and Rust. In free-form style, API primitives do not have a fixed form: they should be defined as strings containing free-form pieces of code with interpolated variables of the form @@{var} or @@ (they correspond to arguments in function-like style). This configuration may be overridden for individual API primitives — see for example the re2c:YYFILL:naked configuration for YYFILL. 😭 so many layers, we stan.
• re2c:bit-vectors, re2c:flags:bit-vectors, re2c:flags:b — same as the --bit-vectors option, but it can be configured on a per-block basis. ✨
• re2c:captures, re2c:leftmost-captures — same as the --leftmost-captures option, but can be configured on a per-block basis.
• re2c:captvars, re2c:leftmost-captvars — same as the --leftmost-captvars option, but can be configured on a per-block basis.
• re2c:case-insensitive, re2c:flags:case-insensitive — same as the --case-insensitive option, but can be configured on a per-block basis.
• re2c:case-inverted, re2c:flags:case-inverted — same as the --case-inverted option, but can be configured on a per-block basis.
• re2c:case-ranges, re2c:flags:case-ranges — same as the --case-ranges option, but can be configured on a per-block basis.
• re2c:computed-gotos, re2c:flags:computed-gotos, re2c:flags:g — same as the --computed-gotos option, but can be configured on a per-block basis.
• re2c:computed-gotos:relative, re2c:cgoto:relative — same as the --computed-gotos-relative option, but can be configured on a per-block basis.
• re2c:computed-gotos:threshold, re2c:cgoto:threshold — if computed goto is used, this configuration specifies the complexity threshold that triggers the generation of jump tables instead of nested if statements and bitmaps. The default value is 9. 🔥
• re2c:cond:abort — if set to a positive integer value, the default case in the generated condition dispatch aborts program execution. Dramatic but valid.
• re2c:cond:goto — specifies a piece of code used for the autogenerated shortcut rules :=> in conditions. The default is goto @@;. The @@ placeholder is substituted with the condition name (see configurations re2c:api:sigil and re2c:cond:goto@cond).
• re2c:cond:goto@cond — specifies the sigil used for argument substitution in the re2c:cond:goto definition. The default value is @@. This overrides the more generic re2c:api:sigil configuration.
• re2c:cond:divider — defines the divider for condition blocks. The default value is /* *********************************** */. Placeholders are substituted with the condition name (see re2c:api;sigil and re2c:cond:divider@cond). ✨ aesthetic ✨
• re2c:cond:divider@cond — specifies the sigil used for argument substitution in the re2c:cond:divider definition. The default is @@. This overrides the more generic re2c:api:sigil configuration.
• re2c:cond:prefix, re2c:condprefix — specifies the prefix used for condition labels. The default is yyc_.
• re2c:cond:enumprefix, re2c:condenumprefix — specifies the prefix used for condition identifiers. The default is yyc.
• re2c:debug-output, re2c:flags:debug-output, re2c:flags:d — same as the --debug-output option, but can be configured on a per-block basis.
• re2c:empty-class, re2c:flags:empty-class — same as the --empty-class option, but can be configured on a per-block basis.
• re2c:encoding:ebcdic, re2c:flags:ecb, re2c:flags:e — same as the --ebcdic option, but can be configured on a per-block basis.
• re2c:encoding:ucs2, re2c:flags:wide-chars, re2c:flags:w — same as the --ucs2 option, but can be configured on a per-block basis.
• re2c:encoding:utf8, re2c:flags:utf-8, re2c:flags:8 — same as the --utf8 option, but can be configured on a per-block basis.
• re2c:encoding:utf16, re2c:flags:utf-16, re2c:flags:x — same as the --utf16 option, but can be configured on a per-block basis.
• re2c:encoding:utf32, re2c:flags:unicode, re2c:flags:u — same as the --utf32 option, but can be configured on a per-block basis.
• re2c:encoding-policy, re2c:flags:encoding-policy — same as the --encoding-policy option, but can be configured on a per-block basis.
• re2c:eof — specifies the sentinel symbol used with the end-of-input rule $. The default value is -1 (which means the $ rule is not used). Other possible values include all valid code units. Only decimal numbers are recognized, bestie, don't try to be cute with hex. 💅
• re2c:header, re2c:flags:type-header, re2c:flags:t — specifies the name of the generated header file relative to the directory of the output file. Same as the --header option except that the file path is relative. 👑
• re2c:indent:string — specifies the string used for indentation. The default is a single tab character "\t". The indent string should contain whitespace characters only. To disable indentation entirely, set this configuration to an empty string. 🎀
• re2c:indent:top — specifies the minimum amount of indentation to use. The default value is zero. The value should be a non-negative integer number.
• re2c:invert-captures — same as the --invert-captures option, but can be configured on a per-block basis.
• re2c:label:prefix, re2c:labelprefix — specifies the prefix used for DFA state labels. The default is yy.
• re2c:label:start, re2c:startlabel — controls the generation of a block start label. The default value is zero, which means the start label is generated only if it is used. An integer value greater than zero forces the generation of the start label even if it is unused by the lexer. A string value also forces start label generation and sets the label name to the specified string. Heads up: this configuration applies only to the current block (it is reset to default for the next block). 🙈
• re2c:label:yyFillLabel — specifies the prefix of YYFILL labels used with re2c:eof and in storable state mode.
• re2c:label:yyloop — specifies the name of the label marking the start of the lexer loop with the --loop-switch option. The default is yyloop.
• re2c:label:yyNext — specifies the name of the optional label that follows the YYGETSTATE switch in storable state mode (enabled with re2c:state:nextlabel). The default is yyNext.
• re2c:lookahead, re2c:flags:lookahead — deprecated (see the deprecated --no-lookahead option). Don't come for me, I'm just the messenger. 💀
• re2c:monadic — if set to non-zero, the generated lexer will use monadic notation (this configuration is specific to Haskell). 🌸
• re2c:nested-ifs, re2c:flags:nested-ifs, re2c:flags:s — same as the --nested-ifs option, but can be configured on a per-block basis.
• re2c:posix-captures, re2c:flags:posix-captures, re2c:flags:P — literally the exact same thing as the --posix-captures option, except you can set it on a per-block basis. same energy, more granular 💅
• re2c:posix-captvars — same as the --posix-captvars option, but again, configurable per-block.
• re2c:tags, re2c:flags:tags, re2c:flags:T — same as the --tags option, except per-block configurable.
• re2c:tags:expression — sets the expression used for tag variables. by default re2ocaml generates expressions of the form yyt<N>. and like... that might be inconvenient? for example if your tag variables are defined as fields in a struct. so: every occurrence of @@{tag} or @@ gets replaced with the actual tag name. example time ✨ re2c:tags:expression = "s.@@"; gives you expressions of the form s.yyt<N> in the generated code. also see the re2c:api:sigil configuration.
• re2c:tags:negative — the constant expression used for a negative tag value (typically this would be -1 if tags are integer offsets in the input string, or a null pointer if they're pointers). no cap.
• re2c:tags:prefix — the prefix for tag variable names. default is yyt.
• re2c:sentinel — the sentinel symbol used for the end-of-input checks (this is when bounds checks are disabled with re2c:yyfill:enable = 0; and re2c:eof is not set). important: this configuration does not affect code generation. its whole purpose is to verify that the sentinel is not allowed in the middle of a rule, and to make sure the lexer won't read past the end of the buffer. default value is -1 (and in that case re2ocaml assumes the sentinel is zero, which is the most common case). only decimal numbers are recognized, bestie. only decimals.
• re2c:state:abort — if you set this to a positive integer value, the default case in the generated state dispatch aborts program execution, and an explicit -1 case contains the transition to the start of the block. dramatic but valid 😭
• re2c:state:nextlabel — controls whether the YYGETSTATE switch is followed by a yyNext label (default value is zero, which corresponds to no label). alternatively you can use re2c:label:start to generate a specific start label, or an explicit getstate block to generate the YYGETSTATE switch separately from the lexer block. options on options, we love to see it 🫶
• re2c:unsafe, re2c:flags:unsafe — same as the --no-unsafe option, but configurable on a per-block basis. if set to zero, it suppresses the generation of unsafe wrappers around YYPEEK. default is non-zero (so wrappers get generated). this configuration is specific to Rust.
• re2c:YYBACKUP, re2c:define:YYBACKUP — defines the generic API primitive YYBACKUP.
• re2c:YYBACKUPCTX, re2c:define:YYBACKUPCTX — defines the generic API primitive YYBACKUPCTX.
• re2c:YYCONDTYPE, re2c:define:YYCONDTYPE — defines the API primitive YYCONDTYPE.
• re2c:YYCTYPE, re2c:define:YYCTYPE — defines the API primitive YYCTYPE.
• re2c:YYCTXMARKER, re2c:define:YYCTXMARKER — defines the API primitive YYCTXMARKER.
• re2c:YYCURSOR, re2c:define:YYCURSOR — defines the API primitive YYCURSOR.
• re2c:YYDEBUG, re2c:define:YYDEBUG — defines the API primitive YYDEBUG.
• re2c:YYFILL, re2c:define:YYFILL — defines the API primitive YYFILL.
• re2c:YYFILL@len, re2c:define:YYFILL@len — specifies the sigil used for argument substitution in the YYFILL definition. defaults to @@. overrides the more generic re2c:api:sigil configuration.
• re2c:YYFILL:naked, re2c:define:YYFILL:naked — overrides the more generic re2c:api:style configuration for YYFILL. a zero value corresponds to free-form API style.
• re2c:YYFN — defines the API primitive YYFN.
• re2c:YYINPUT — defines the API primitive YYINPUT.
• re2c:YYGETCOND, re2c:define:YYGETCONDITION — defines the API primitive YYGETCOND.
• re2c:YYGETCOND:naked, re2c:define:YYGETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYGETCOND. zero value = free-form API style.
• re2c:YYGETSTATE, re2c:define:YYGETSTATE — defines the API primitive YYGETSTATE.
• re2c:YYGETSTATE:naked, re2c:define:YYGETSTATE:naked — overrides the more generic re2c:api:style configuration for YYGETSTATE. zero value = free-form API style.
• re2c:YYGETACCEPT, re2c:define:YYGETACCEPT — defines the API primitive YYGETACCEPT.
• re2c:YYLESSTHAN, re2c:define:YYLESSTHAN — defines the generic API primitive YYLESSTHAN.
• re2c:YYLIMIT, re2c:define:YYLIMIT — defines the API primitive YYLIMIT.
• re2c:YYMARKER, re2c:define:YYMARKER — defines the API primitive YYMARKER.
• re2c:YYMTAGN, re2c:define:YYMTAGN — defines the generic API primitive YYMTAGN.
• re2c:YYMTAGP, re2c:define:YYMTAGP — defines the generic API primitive YYMTAGP.
• re2c:YYPEEK, re2c:define:YYPEEK — defines the generic API primitive YYPEEK.
• re2c:YYRESTORE, re2c:define:YYRESTORE — defines the generic API primitive YYRESTORE.
• re2c:YYRESTORECTX, re2c:define:YYRESTORECTX — defines the generic API primitive YYRESTORECTX.
• re2c:YYRESTORETAG, re2c:define:YYRESTORETAG — defines the generic API primitive YYRESTORETAG.
• re2c:YYSETCOND, re2c:define:YYSETCONDITION — defines the API primitive YYSETCOND.
• re2c:YYSETCOND@cond, re2c:define:YYSETCONDITION@cond — specifies the sigil used for argument substitution in the YYSETCOND definition. default value is @@. overrides the more generic re2c:api:sigil configuration.
• re2c:YYSETCOND:naked, re2c:define:YYSETCONDITION:naked — overrides the more generic re2c:api:style configuration for YYSETCOND. zero value = free-form API style.
• re2c:YYSETSTATE, re2c:define:YYSETSTATE — defines the API primitive YYSETSTATE.
• re2c:YYSETSTATE@state, re2c:define:YYSETSTATE@state — specifies the sigil used for argument substitution in the YYSETSTATE definition. default value is @@. overrides the more generic re2c:api:sigil configuration.
• re2c:YYSETSTATE:naked, re2c:define:YYSETSTATE:naked — overrides the more generic re2c:api:style configuration for YYSETSTATE. zero value = free-form API style.
• re2c:YYSETACCEPT, re2c:define:YYSETACCEPT — defines the API primitive YYSETACCEPT.
• re2c:YYSKIP, re2c:define:YYSKIP — defines the generic API primitive YYSKIP.
• re2c:YYSHIFT, re2c:define:YYSHIFT — defines the generic API primitive YYSHIFT.
• re2c:YYCOPYMTAG, re2c:define:YYCOPYMTAG — defines the generic API primitive YYCOPYMTAG.
• re2c:YYCOPYSTAG, re2c:define:YYCOPYSTAG — defines the generic API primitive YYCOPYSTAG.
• re2c:YYSHIFTMTAG, re2c:define:YYSHIFTMTAG — defines the generic API primitive YYSHIFTMTAG.
• re2c:YYSHIFTSTAG, re2c:define:YYSHIFTSTAG — defines the generic API primitive YYSHIFTSTAG.
• re2c:YYSTAGN, re2c:define:YYSTAGN — defines the generic API primitive YYSTAGN.
• re2c:YYSTAGP, re2c:define:YYSTAGP — defines the generic API primitive YYSTAGP.
• re2c:yyaccept, re2c:variable:yyaccept — defines the API primitive yyaccept.
• re2c:yybm, re2c:variable:yybm — defines the API primitive yybm.
• re2c:yybm:hex, re2c:variable:yybm:hex — if set to nonzero, bitmaps for the --bit-vectors option are generated in hexadecimal format. default is zero (so bitmaps are in decimal format).
• re2c:yych, re2c:variable:yych — defines the API primitive yych.
• re2c:yych:emit, re2c:variable:yych:emit — if set to zero, the yych definition is not generated. default is non-zero.
• re2c:yych:conversion, re2c:variable:yych:conversion — if set to non-zero, re2ocaml automatically generates a conversion to YYCTYPE every single time yych is read. default is zero (meaning no conversion). no surprise conversions, tyvm 💀
• re2c:yych:literals, re2c:variable:yych:literals — specifies the form of literals that yych is matched against. possible values are: char (character literals in single quotes, and non-printable ones use escape sequences that start with backslash), hex (hexadecimal integers) and char_or_hex (a mixture of both — character literals for printable characters and hexadecimal integers for the others).
• re2c:yyctable, re2c:variable:yyctable — defines the API primitive yyctable.
• re2c:yynmatch, re2c:variable:yynmatch — defines the API primitive yynmatch.
• re2c:yypmatch, re2c:variable:yypmatch — defines the API primitive yypmatch.
• re2c:yytarget, re2c:variable:yytarget — defines the API primitive yytarget.
• re2c:yystable, re2c:variable:yystable — deprecated. 🪦 that's it, that's the whole note.
• re2c:yystate, re2c:variable:yystate — defines the API primitive yystate.
• re2c:yyfill, re2c:variable:yyfill — defines the API primitive yyfill.
• re2c:yyfill:check — if set to zero, suppresses the generation of the pre-YYFILL check for the number of input characters (that's the YYLESSTHAN definition in the generic API, and the YYLIMIT-based comparison in the C pointer API). default is non-zero, so the check is generated.
• re2c:yyfill:enable — if set to zero, suppresses the generation of YYFILL (together with the check). you'd use this when the whole input fits into one piece of memory (so there's no need for buffering) and the end-of-input checks do not rely on the YYFILL checks (like, if a sentinel character is used). use warnings (the -W option) and the re2c:sentinel configuration to verify that the generated lexer cannot read past the end of input. default is non-zero (meaning YYFILL is enabled). > bestie please actually verify this one, reading past the buffer is not the cute kind of chaos 😭
• re2c:yyfill:parameter — if set to zero, suppresses the generation of the parameter passed to YYFILL. that parameter is the minimum number of characters that must be supplied. defaults to non-zero (the parameter is generated). this configuration can be overridden with re2c:YYFILL:naked or re2c:api:style.
• re2c:yyfn:sep — specifies the separator used in YYFN elements (defaults to semicolon).
• re2c:yyfn:throw — specifies the exceptions thrown by the YYFN function (defaults to empty, which means no exceptions).

---

🌸 Regular expressions 🌸

okay so re2ocaml uses the following syntax for regular expressions — this is the fun part, the actual pattern-matching tea ☕✨

• "foo" — case-sensitive string literal.
• 'foo' — case-insensitive string literal.
• [a-xyz], [^a-xyz] — character class (possibly negated).
• . — any character except newline.
• R \ S — difference of character classes R and S.
• R* — zero or more occurrences of R.
• R+ — one or more occurrences of R.
• R? — optional R.
• R{n} — repetition of R exactly n times.
• R{n,} — bestie, this is R repeated at least n times. Minimum effort, maximum range. 💅
• R{n,m} — R repeated from n to m times. A little sliding scale, no cap.
• (R) — literally just R, but the parentheses are there to override precedence (they said "respect my hierarchy" and honestly? iconic). If submatch extraction is enabled, (R) is a capturing or a non-capturing group depending on the --invert-captures option. 🎀
• (!R) — if submatch extraction is enabled, (!R) is a non-capturing or a capturing group depending on --invert-captures option. Yes, that's the reverse of (R) — it's giving plot twist. 🙈
• R S — concatenation: R followed by S. First this, then that. Order matters, always. ✨
• R | S — alternative: R or S. Pick a lane. 🌸
• R / S — lookahead: R followed by S, but S is not consumed. The way it peeks and then just leaves? Rude but efficient. 👑
• name — a regular expression defined as name (or the literal string "name" in Flex compatibility mode).
• {name} — a regular expression defined as name in Flex compatibility mode. Curly braces = Flex vibes. 🦋
• @stag — an s-tag: saves the last input position at which @stag matches, in a variable named stag.
• #mtag — an m-tag: saves all input positions at which #mtag matches, in a variable named mtag. It's a collector, a little archivist. 😭
• $ — end of input. The final boss. 🏁

Character classes and string literals may contain the following escape sequences: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and \Uhhhhhhhh. Memorize them, they come up constantly. ⭐

---

🧚 Actions 🧚

Okay so here's a list of predefined actions that re2ocaml supports — these are the little sprinkles you can drop into your rules, and they do real work:

• !entry code — an entry action that binds a user-defined block of code to the start state of the current finite state machine. If start conditions are used, the entry action can be set individually for each condition. This action may be used to perform initialization, e.g. to save the start location of a lexeme. It's the "get ready" moment before the main character enters. 💖
• !pre_rule code — a pre-rule action that prepends a user-defined block of code to the semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to factor out the common part of all semantic actions (e.g. saving the end location of a lexeme). Why write it forty times when once will do? Exactly. 🫶
• !post_rule code — a post-rule action that appends a user-defined block of code to the semantic actions of all rules in the current block (or condition, if start conditions are used). This action may be used to emit trap statements that guard against unintended control flow. Safety rails, bestie. We love a guard. 🎀

---

🌈 Directives 🌈

Here's the full list of directives supported by re2ocaml — and yes, we're doing all of them:

• !use:name ; — an in-block use directive that merges a previously defined rules block with the specified name into the current block. Named definitions, configurations and rules of the referenced block are added to the current ones. Conflicts between overlapping rules and configurations are resolved in the usual way: the first rule takes priority, and the latest configuration overrides the preceding ones. One exception is the special rules *, $ and <!> for which a block-local definition always takes priority. A use directive can be placed anywhere inside of a block, and multiple use directives are allowed. It's the "borrowing your friend's notes" of the lexer world. 📚
• !include file ; — this directive is the same as include block: it inserts file contents verbatim in place of the directive. Copy-paste but make it official. ✨

---

🫶 PROGRAM INTERFACE 🫶

So the generated code interfaces with the outer program with the help of primitives, collectively referred to as the API. Which primitives should be defined for a particular program depends on multiple factors, including the complexity of regular expressions, input representation, buffering and the use of various features. All the necessary primitives should be defined by the user in the form of macros, functions, variables or any other suitable form that makes the generated code syntactically and semantically correct. re2ocaml does not (and cannot) check the definitions, so if anything is missing or defined incorrectly, the generated program may have compile-time or run-time errors. 😭 This manual provides examples of API definitions in the most common cases — we're holding your hand through this, fr fr.

re2ocaml has two API flavors that define the core set of primitives used by a program:

💜 Record API

Record API is the default API for the OCaml backend. This API consists of a variable yyrecord (the name can be overridden with re2c:yyrecord) that should be defined as a record with fields _yyinput, _yycursor, _yymarker, _yyctxmarker, _yylimit. Only the fields used by the generated code need to be defined, and their names can be configured. Minimal effort, maximum slay. 💅

🦋 Generic API

This is the most flexible API. It is enabled with --api generic option or re2c:api = generic configuration. It contains primitives for generic operations: YYPEEK, YYSKIP, YYBACKUP, YYBACKUPCTX, YYSTAGP, YYSTAGN, YYMTAGP, YYMTAGN, YYRESTORE, YYRESTORECTX, YYRESTORETAG, YYSHIFT, YYSHIFTSTAG, YYSHIFTMTAG, YYLESSTHAN, YYEND. It's giving "build it yourself, but make it chic". ✨

Here is a full list of API primitives that may be used by the generated code in order to interface with the outer program:

• YYCTYPE — the type of the input characters (code units). For ASCII, EBCDIC and UTF-8 encodings it should be a 1-byte unsigned integer. For UTF-16 or UCS-2 it should be a 2-byte unsigned integer. For UTF-32 it should be a 4-byte unsigned integer. Pick the right size, no cap. 📏
• YYCURSOR — an l-value that stores the current input position (a pointer or an integer offset in YYINPUT). Initially YYCURSOR should point to the first input character, and later it is advanced by the generated code. When a rule matches, the YYCURSOR position is the one after the last matched character. It's the main character and it's always moving. 💅
• YYLIMIT — an r-value that stores the end of input position (a pointer or an integer offset in YYINPUT). Initially YYLIMIT should point to the position after the last available input character. It is not changed by the generated code. The lexer compares YYCURSOR to YYLIMIT in order to determine if there are enough input characters left. The responsible friend who knows where the party ends. 🌙
• YYMARKER — an l-value that stores the position of the latest matched rule (a pointer or an integer offset in YYINPUT). It is used to restore the YYCURSOR position if the longer match fails and the lexer needs to rollback. Initialization is not needed. Rewind energy. ⏪
• YYCTXMARKER — an l-value that stores the position of the trailing context (a pointer or an integer offset in YYINPUT). No initialization is needed. YYCTXMARKER is needed only if the lookahead operator / is used. Conditional icon. 🎀
• YYFILL — a generic API primitive with one variable len. YYFILL should provide at least len more input characters or fail. If re2c:eof is used, then len is always 1 and YYFILL should always return to the calling function; a zero return value indicates success. If re2c:eof is not used, then the YYFILL return value is ignored and it should not return on failure. The maximum value of len is YYMAXFILL. Read that part twice, bestie, the two modes are not the same. 😭
• YYFN — a primitive that defines a function prototype in --recursive-functions code model. Its value should be an array of one or more strings, where each string contains two or three components separated by the string specified in re2c:fn:sep configuration (typically a semicolon). The first array element defines function name and return type (empty for a void function). Subsequent elements define function arguments: first, the expression for the argument used in function body (usually just a name); second, argument type; third, an optional formal parameter (it defaults to the first component — usually both the argument and the parameter are the same identifier). It's a lot but it's all spelled out for you. 🫶
• YYINPUT — an r-value that stores the current input character sequence (string, buffer, etc.). Your raw material. 🍓
• YYMAXFILL — an integral constant equal to the maximum value of the argument to YYFILL. It can be generated with a max block. Numbers don't lie. 🔢
• YYLESSTHAN — a generic API primitive with one variable len. It should be defined as an r-value of boolean type that equals true if and only if there are less than len input characters left.
• YYEND — a generic API primitive with no variables. It should be defined as an r-value of boolean type that equals true if and only if the logical end of input has been reached (excluding any padding or sentinel symbols). YYEND is used to implement the $ symbol in regular expressions. It differs from YYLESSTHAN, which is used to ensure that the lexer won't read past the end of buffer. Two different ends, don't mix them up. 🚨
• YYPEEK — a generic API primitive with no variables. It should be defined as an r-value of type YYCTYPE that is equal to the character at the current input position. Little sneak peek. 👀
• YYSKIP — a generic API primitive that should advance the current input position by one code unit. One step at a time, we love a forward mover. 🦋
• YYBACKUP — a generic API primitive that should save the current input position (to be restored with YYRESTORE later). Save point unlocked. 💾
• YYRESTORE — a generic API primitive that should restore the current input position to the value saved by YYBACKUP. Load save point. ⏮️
• YYBACKUPCTX — a generic API primitive that should save the current input position as the position of the trailing context (to be restored with YYRESTORECTX later).
• YYRESTORECTX — a generic API primitive that should restore the trailing context position saved with YYBACKUPCTX.
• YYRESTORETAG — a generic API primitive with one variable tag that should restore the trailing context position to the value of tag.
• YYSTAGP — a generic API primitive with one variable tag, where tag can be a pointer or an offset in YYINPUT (see the submatch extraction section for details). YYSTAGP should set tag to the current input position. 📍
• YYSTAGN — a generic API primitive with one variable tag, where tag can be a pointer or an offset in YYINPUT (see the submatch extraction section for details). YYSTAGN should set tag to a value that represents non-existent input position. The void, but make it a tag. 🕳️
• YYMTAGP — a generic API primitive with one variable tag. YYMTAGP should append the current position to the submatch history of tag (see the submatch extraction section for details). Collecting positions like they're stickers. ✨
• YYMTAGN — a generic API primitive with one variable tag. YYMTAGN should append a value that represents non-existent input position to the submatch history of tag (see the submatch extraction section for details).
• YYSHIFT — a generic API primitive with one variable shift that should shift the current input position by shift characters (the shift value may be negative). Teleportation, but make it tokens. 🌀
• YYCOPYSTAG — a generic API primitive that takes two variables, lhs and rhs, and is supposed to copy the right-hand-side s-tag variable rhs into the left-hand-side s-tag variable lhs. For most languages it comes with a default definition that just assigns lhs to rhs. Slay, no notes. 💅
• YYCOPYMTAG — same energy, different tag: two variables lhs and rhs, and it copies the right-hand-side m-tag variable rhs into the left-hand-side m-tag variable lhs. Again, most languages get a default definition that assigns lhs to rhs. Iconic twin behavior. 🫶
• YYSHIFTSTAG — a generic API primitive with two variables, tag and shift, that should shift tag by shift code units. And yes bestie, the shift value may be negative. 😭
• YYSHIFTMTAG — a generic API primitive with two variables, tag and shift, that should shift the latest value in the history of tag by shift code units (the shift value may be negative — same rules, different drama). ✨
• YYMAXNMATCH — an integral constant equal to the maximal number of POSIX capturing groups in a rule. It gets generated with a maxnmatch block. Very demure, very mindful constant. 🎀
• YYCONDTYPE — the type of the condition enum. It can be generated either with a conditions block or with the --header option. Pick your main character origin story. 👑
• YYGETACCEPT — a primitive with one variable var that stores the numeric selector of the accepted rule. For most languages it has a default definition that reads from var. 🍓
• YYSETACCEPT — a primitive with two variables: var (an l-value that stores the numeric selector of the accepted rule) and val (the value of the selector). For most languages the default definition assigns var to val. Matching set, literally. 💖
• YYGETCOND — an r-value of type YYCONDTYPE that equals the current condition identifier. It's giving "just read the vibes". 🌸
• YYSETCOND — a primitive with one variable cond that should set the current condition identifier to cond. 🔥
• YYGETSTATE — an r-value of integer type that equals the current lexer state. Heads up bestie: it should be initialized to -1. Don't forget or it's gonna be a whole thing. 💀
• YYSETSTATE — a primitive with one variable state that should set the current lexer state to state. Simple, clean, ate. ✨
• YYDEBUG — ngl this primitive is only generated with the -d, --debug-output option. Its whole purpose is to add logging to the generated code (a typical YYDEBUG definition is a print statement). YYDEBUG statements get generated in every state and have two variables: state (either a DFA state index or -1) and symbol (the current input symbol). Little reporter fairy for your DFA. 🧚
• yyaccept — an l-value of unsigned integral type that stores the number of the latest matched rule. You only need to define it yourself with the --storable-state option. 🙈
• yybm — a table containing compressed bitmaps for up to 8 transitions (used with the --bitmaps option). The table contains 256 elements and is indexed by 1-byte code units. Each 8-bit element combines boolean values for up to 8 transitions. The k-th bit of the n-th element is true iff the n-th code unit is in the range of the k-th transition. The whole idea of this bitmap is to replace a bunch of if branches or switch cases with one single bit check in the table. Efficiency queen behavior. 👑✨
• yych — an l-value of type YYCTYPE that stores the current input character. You only need to define it yourself with the -f --storable-state option. 💜
• yyctable — a jump table generated for the initial condition dispatch (enabled with the combination of the --conditions and --computed-gotos options). 🦋
• yyfill — an l-value that stores the result of the YYFILL call (this may be necessary for pure functional languages, where YYFILL is a monadic function with a complex return value). Functional girlies, this one's for you. 🫶
• yynmatch — an l-value of unsigned integral type that stores the number of POSIX capturing groups in the matched rule. Used only with the -P --posix-captures option. 🍓
• yypmatch — an array of l-values used to hold the tag values corresponding to the capturing parentheses in the matching rule. Array length must be at least yynmatch * 2 (usually YYMAXNMATCH * 2 is a good choice). Used only with the -P --posix-captures option. Matching pair energy fr fr. 💖
• yystable — Deprecated. That's it. That's the whole entry. We don't talk about her anymore. 💀
• yystate — an l-value used with the --loop-switch option to store the current DFA state. ⭐
• yytarget — a jump table that contains jump targets (label addresses) for all transitions from a state. This table is local to each state. Generation of yytarget tables is enabled with the --computed-gotos option. 🌈

---

✨ Options ✨

Some of the options have corresponding configurations, others are global and cannot be changed after re2c starts reading the input file. Debug options generally require building re2c in debug configuration. Internal options are useful for experimenting with the algorithms used in re2c. 💅

• -? --help -h — show the help message. The "hello bestie" of flags. 🎀
• --api <simple | record | generic> — specify the API the generated code uses to interface with used-defined code. simple should be used in simple cases when there's no need for buffer refilling and storing lexer state. record should be used when lexer state needs to be stored in a record (struct, class, etc.). generic should be used in complex cases when the other two APIs just aren't flexible enough. 😭
• --bit-vectors -b — optimize conditional jumps using bit masks. This option implies --nested-ifs. Bits are so back. 🔥
• --captures, --leftmost-captures — enable submatch extraction with leftmost greedy capturing groups. The result is collected into an array yybmatch of capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule. 👑
• --captvars, --leftmost-captvars — enable submatch extraction with leftmost greedy capturing groups, but the result is collected into variables yytl<k>, yytr<k> for the k-th capturing group. Same extraction, cuter packaging. 🎀
• --case-insensitive — treat single-quoted and double-quoted strings as case-insensitive. Case? Never heard of her. 🌸
• --case-inverted — invert the meaning of single-quoted and double-quoted strings: treat single-quoted strings as case-sensitive and double-quoted strings as case-insensitive. Plot twist era. 🦋
• --case-ranges — collapse consecutive cases in switch statements into a range of the form low ... high. This syntax is a C/C++ language extension supported by compilers like GCC, Clang and Tcc. The main advantage over using single cases is smaller generated code and faster generation time, although for some compilers like Tcc it also results in smaller binary size. Smaller and faster? Obsessed. 💖
• --computed-gotos -g — optimize conditional jumps using the non-standard "computed goto" extension (which must be supported by the compiler — check before you commit, bestie). re2ocaml generates jump tables only in complex cases with a lot of conditional branches. Complexity threshold can be configured with the cgoto:threshold configuration. Relative offsets can be enabled with the cgoto:relative configuration. This option implies --bit-vectors. 🌈
• --computed-gotos-relative — similar to --computed-gotos but generates relative offsets for jump tables instead (which must be supported by the compiler). This option implies --computed-gotos. Relative queen. ⭐
• --conditions --start-conditions -c — enable support of Flex-like "conditions": multiple interrelated lexers within one block. This is an alternative to manually specifying different re2ocaml blocks connected with goto or function calls. Multiplexing slay. 💅
• --depfile FILE — write dependency information to FILE in the form of a Makefile rule <output-file> : <input-file> [include-file ...]. This lets you track build dependencies when include blocks/directives are in play, so that updating include files triggers regeneration of the output file. This option depends on the --output option. 👀
• --ebcdic --ecb -e — generate a lexer that reads input in EBCDIC encoding. re2ocaml assumes that the character range is 0 -- 0xFF and character size is 1 byte. Retro encoding moment. 🕰️
• --empty-class <match-empty | match-none | error> — define the way re2ocaml treats empty character classes. With match-empty (the default) an empty class matches empty input (which is illogical, but backwards-compatible — yes we know). With match-none an empty class always fails to match. With error an empty class raises a compilation error. Pick your chaos. 😭
• --encoding-policy <fail | substitute | ignore> — define the way re2ocaml treats Unicode surrogates. With fail, re2ocaml aborts with an error when a surrogate is encountered. With substitute, re2ocaml silently replaces surrogates with the error code point 0xFFFD. With ignore (the default), re2ocaml treats surrogates as normal code points. The Unicode standard says that standalone surrogates are invalid, but real-world libraries and programs behave in different ways. Messy in the wild, fr. 🙈
• --flex-syntax -F — partial support for Flex syntax: in this mode named definitions don't need the equal sign and the terminating semicolon, and when used they must be surrounded with curly braces. Names without curly braces are treated as double-quoted strings. Flex girlies rise. 🧚
• --goto-label — use the "goto/label" code model: encode DFA in the form of labeled code blocks connected with goto transitions across blocks. This is only supported for languages that have a goto statement. No goto, no go. 🚫
• --header --type-header -t HEADER — generate a HEADER file. The contents of the file can be specified using special blocks header:on and header:off. If conditions are used, the generated header will have a condition enum automatically appended to it (unless there is an explicit conditions block). Free bonus enum, we love a two-for-one. 💜
• -I PATH — add PATH to the list of locations which are used when searching for include files. This option is useful in combination with an include block or directive. re2ocaml looks for FILE in the directory of the parent file and in the include locations specified with the -I option. 🦋
• --input <default | custom> — deprecated alias for --api. Option default corresponds to simple (it is indeed the default for most backends, but not for all — read carefully bestie). Option custom corresponds to generic. 💀
• --input-encoding <ascii | utf8> — tells re2ocaml how to read your regex, bestie. With ascii (the default) it treats the input as ASCII-encoded: any sequence of code units is just a sequence of standalone 1-byte characters, no thinking required. With utf8 it treats the input as UTF8-encoded and actually recognizes multibyte characters 💅
• --invert-captures — flips the whole capturing thing on its head. Normally (...) is capturing and (! ...) is non-capturing. With this option (! ...) is capturing and (...) is non-capturing. Chaos! Iconic! 🌈
• --lang <none | c | d | go | haskell | java | js | ocaml | python | rust | swift | v | zig> — pick your target language, queen. The supported ones are C, D, Go, Haskell, Java, JS, OCaml, Python, Rust, Swift, V, and Zig (and more can be added via user-defined syntax files — peep the --syntax option). none disables the default syntax configs so the target language is literally undefined, main-character-void energy 🕳️
• --location-format <gnu | msvc> — controls how locations show up in messages. gnu prints them like 'filename:line:column: ...'. msvc prints them like 'filename(line,column) ...'. Default is gnu. Pick your fighter ✨
• --loop-switch — uses the "loop/switch" code model: the DFA gets encoded as a loop over a switch statement, where each individual state is a switch case. The state lives in a variable called yystate. Transitions between states update yystate to the case label of the destination state and then just keep executing to the head of the loop. It's giving "one big while loop and we're all just living in it" 💀
• --computed-continue — this one requires the --loop-switch code model, no exceptions bestie. It uses computed continue statements to pick the next switch case directly, which unlocks jump-table optimization similar to --computed-gotos in the goto/label code model. Optimization slay 🔥
• --nested-ifs -s — uses nested if statements instead of switch statements inside conditional jumps. Usually this gives you more efficient code with non-optimizing compilers. Lowkey a glow-up for old compilers 🫶
• --no-debug-info -i — don't output line directives. Super useful when the generated code lives in a version control system, because it avoids huge autogenerated diffs on small changes. Your git history says thank you 😭
• --no-generation-date — suppresses the date output inside the generated file. Time? Never heard of her ⭐
• --no-version — suppresses version output in the generated file.
• --no-unsafe — don't generate the unsafe wrapper over YYPEEK. Heads up: this option is specifically for Rust 🦀. For performance reasons YYPEEK should avoid bounds-checking, since the lexer already does end-of-input checks in a way that's way more efficient. The user can choose to provide a safe YYPEEK definition, or one that's unsafe only in release builds — and in that second case --no-unsafe helps you dodge warnings about redundant unsafe blocks. Very "I know what I'm doing" of you 💅
• --output -o OUTPUT — specifies the OUTPUT file. That's it, that's the option 🎀
• --posix-captures, -P — turns on submatch extraction with POSIX-style capturing groups. The result gets collected into an array yybmatch with capacity 2 * YYMAXNMATCH, and yynmatch is set to the number of groups for the matching rule. Look at her go 💖
• --posix-captvars — also enables submatch extraction with POSIX-style capturing groups, but this time the result is collected into variables yytl<k>, yytr<k> for the k-th capturing group. Same energy, different storage, we love options 🌸
• --recursive-functions — uses a code model based on co-recursive functions, where every single DFA state becomes its own separate function that may call other state-functions or itself. Recursion bestie, she contains multitudes 🌀 (jk, that's a vibe not a fact — the multitudes thing I mean. The function thing is real.)
• --reusable -r — deprecated since version 2.2 (reusable blocks are allowed by default now, so this flag is just vibing in the past) 💀
• --skeleton -S — ignores user-defined interface code and generates a self-contained "skeleton" program. On top of that, it generates input files with strings derived from the regular grammar plus compressed match results, and those are used to verify "skeleton" behavior on all inputs. Genuinely useful for finding bugs in optimizations and code generation. Tiny catch: this option is supported only for C 🔍
• --storable-state -f — generates a lexer that can store its inner state. This is the one for push-model lexers — the kind that get stopped by an outer program when there isn't enough input, then resumed once more input shows up. In this mode users should additionally define YYGETSTATE and YYSETSTATE primitives, and the variables yych, yyaccept and state should be part of the stored lexer state. Don't forget, bestie, or it will not work 🙈
• --syntax FILE — loads configurations from the specified FILE and applies them on top of the default syntax file. Note that FILE can define just a few configurations (if you're using it to amend the default syntax file), or it can define an entire brand-new language backend — and in that second case it's recommended to use the --lang none option. Build your own backend, main character energy 👑
• --tags -T — enables submatch extraction with tags 🏷️
• --ucs2 --wide-chars -w — generates a lexer that reads UCS2-encoded input. re2ocaml assumes the character range is 0 -- 0xFFFF and that character size is 2 bytes. This option implies --nested-ifs ✨
• --utf8 --utf-8 -8 — generates a lexer that reads input in UTF-8 encoding. re2ocaml assumes the character range is 0 -- 0x10FFFF and character size is 1 byte 🌈
• --utf16 --utf-16 -x — generates a lexer that reads UTF16-encoded input. re2ocaml assumes the character range is 0 -- 0x10FFFF and character size is 2 bytes. This option implies --nested-ifs.
• --utf32 --unicode -u — generates a lexer that reads UTF32-encoded input. re2ocaml assumes the character range is 0 -- 0x10FFFF and character size is 4 bytes. This option implies --nested-ifs.
• --verbose — outputs a short message in case of success. A little "you did it!" notification, we stan 🫶
• --vernum -V — shows version information in MMmmpp format (major, minor, patch).
• --version -v — shows version information. That's the whole thing 💜
• --single-pass -1 — deprecated. Does nothing (single pass is the default now). A ghost of options past 👻
• --debug-output -d — emits YYDEBUG invocations in the generated code. Useful for tracing lexer execution. Detective mode 🔍
• --dump-adfa — debug option: outputs the DFA after tunneling (in .dot format).
• --dump-cfg — debug option: outputs the control flow graph of tag variables (in .dot format).
• --dump-closure-stats — debug option: outputs statistics on the number of states in closure.
• --dump-dfa-det — debug option: outputs the DFA immediately after determinization (in .dot format).
• --dump-dfa-min — debug option: outputs the DFA after minimization (in .dot format).
• --dump-dfa-tagopt — debug option: outputs the DFA after tag optimizations (in .dot format).
• --dump-dfa-tree — debug option: outputs the DFA under construction with states represented as tag history trees (in .dot format).
• --dump-dfa-raw — debug option: outputs the DFA under construction with expanded state-sets (in .dot format).
• --dump-interf — debug option: outputs the interference table produced by liveness analysis of tag variables.
• --dump-nfa — debug option: outputs the NFA (in .dot format).
• --emit-dot -D — instead of the normal output, generates the lexer graph in .dot format. You can convert the output into an image with the help of Graphviz, like something along the lines of dot -Tpng -odfa.png dfa.dot. Art project time 🎨
• --dfa-minimization <moore | table> — internal option: the DFA minimization algorithm that re2ocaml uses. moore is the Moore algorithm (and it's the default). table is the "table filling" algorithm. Both algorithms should produce the same DFA up to states relabeling — table filling is simpler and much slower, and it serves as a reference implementation. Slow but correct, we respect her 😌
• --eager-skip — internal option: makes the generated lexer advance the input position eagerly, meaning immediately after reading the input symbol. This changes the default behavior, which advances the input position lazily — after transition to the next state. Eager vs lazy, the eternal battle ⚡
• --no-lookahead — internal option, deprecated. It used to enable the TDFA(0) algorithm. Unlike TDFA(1), the TDFA(0) algorithm does not use one-symbol lookahead — it applies register operations to the incoming transitions rather than the outgoing ones. Benchmarks showed that TDFA(0) is less efficient than TDFA(1), so yeah, it's retired 💀
• --no-optimize-tags — internal option: suppresses optimization of tag variables (useful for debugging).
• --posix-closure <gor1 | gtop> — internal option: specifies the shortest-path algorithm used for constructing the epsilon-closure with POSIX disambiguation semantics. gor1 (the default) stands for the Goldberg-Radzik algorithm, and gtop stands for the "global topological order" algorithm. Pick your shortest path bestie 🛣️
• --posix-prectable <complex | naive> — internal option: specifies the algorithm used to compute the POSIX precedence table. The complex algorithm computes the precedence table in one traversal of the tag history tree and has quadratic complexity in the number of TNFA states — and it's the default. The naive algorithm has worst-case cubic complexity in the number of TNFA states, but it's much simpler than complex and may be slightly faster in non-pathological cases. Cute, simple, occasionally faster — naive really said "I'm baby" 🍓
• --stadfa — internal option, deprecated. It used to enable the staDFA algorithm, which differs from TDFA in that register operations are placed in states rather than on transitions. Benchmarks showed that staDFA is less efficient than TDFA. Another one for the deprecated hall of fame ✨
• --fixed-tags <none | toplevel | all> — internal option: specifies whether the fixed-tag optimization should be applied to all tags (all), none of them (none), or only those in toplevel concatenation (toplevel). The default is all. So what's a "fixed" tag? They're the ones located within a fixed distance to some other tag (called a "base"). In those cases only the base tag needs to be tracked, and the value of the fixed tag can be computed as the value of the base tag plus a static offset. For tags that are under alternative or repetition, you also have to check whether the base tag has a no-match value — and in that case the fixed tag should also be set to no-match, disregarding the offset. For tags in top-level concatenation the check isn't needed, because they always match. Math! Logic! Optimization! Obsessed 💖

---

⚠️ Warnings ⚠️

Warnings can be individually enabled, disabled, and turned into an error. (Yes, the original has a typo there and yes, we're keeping the actual meaning — you get to choose whether warnings are chill, silent, or full-on screaming errors 😤)

• -W — turns on literally every single warning, all of them, the whole roster 💖
• -Werror — makes warnings into errors. But heads up bestie: on its own this flag turns on nothing — it only affects warnings that already got switched on before it, or that get switched on after it. It's a multiplier, not a spawner ✨
• -W<warning> — turn on a specific warning.
• -Wno-<warning> — turn off that specific warning.
• -Werror-<warning> — turn the warning on and treat it as an error. And yeah, this implies -W<warning>, so it's doing double duty 🫶
• -Wno-error-<warning> — don't treat this one particular warning as an error. Super important: this does not turn the warning itself off, it just demotes it back to a chill warning 🙈
• -Wcondition-order — yells at you if the generated program makes implicit assumptions about condition numbering. The fix is to use either the --header option or a conditions block to generate a mapping of condition names to numbers, and then use the autogenerated condition names 👑
• -Wempty-character-class — yells if a regex has an empty character class. Matching an empty character class makes literally no sense — it should always fail. But for backwards compatibility reasons re2ocaml permits empty character classes and treats them as empty strings (💀 yes really). Use the --empty-class option to change the default behavior.
• -Wmatch-empty-string — warns if a rule is nullable, aka matches an empty string. If the lexer runs in a loop and that empty match was unintentional, your lexer may unexpectedly hang in an infinite loop 😭 whole thing just vibing forever, doing nothing.
• -Wswapped-range — warns if the lower bound of a range is greater than its upper bound. Default behavior is to silently swap the range bounds (which, lowkey, is chaos).
• -Wundefined-control-flow — warns if some input strings cause undefined control flow in the lexer, and it reports the faulty patterns. This is a dangerous and very common mistake. Good news: it's an easy fix — just add the default rule * which has the lowest priority, matches any code unit, and always consumes a single code unit. Slay.
• -Wunreachable-rules — warns about rules that are shadowed by other rules and will never, ever match 🔥
• -Wdeprecated-eof_rule — warns about standalone end of input rules $ that future changes will break and that require fixing. Right now these rules take precedence when conflicting with other rules, but after the generalized end of input symbol $ gets introduced, the precedence order will change and these rules will become shadowed by other rules. So like… future-you is gonna be mad at present-you.
• -Wuseless-escape — warns if a symbol is escaped when it shouldn't be. By default re2ocaml silently ignores such escapes, but it may as well indicate a typo or an error in the escape sequence.
• -Wnondeterministic-tags — warns if a tag has n-th degree of nondeterminism, where n is greater than 1.
• -Wsentinel-in-midrule — warns if the sentinel symbol occurs in the middle of a rule. This is genuinely scary: it may cause reads past the end of buffer, crashes or memory corruption in the generated lexer 💀 This warning only applies if the sentinel method of checking for the end of input is used. And it's set to an error if the re2c:sentinel configuration is used.
• -Wundefined-syntax-config — warns if the syntax file specified with --syntax is missing definitions of some configurations. This helps maintain user-defined syntax files: if a new release adds configurations, an old syntax file will raise a warning and the user gets notified 🎀 If some configurations are unused and don't need a definition, they should be explicitly set to <undefined>.

---

✨ SYNTAX FILES ✨

Support for different languages in re2c is based on the idea of syntax files. A syntax file is a configuration file that defines syntax of the target language — not the whole language, just a small part of it that's used by the generated code. So like, the little slice the codegen actually touches 💅

Syntax files make re2c super flexible, but they should not be used as a replacement for re2c: configurations. Their purpose is to define syntax of the target language, not to customize one particular lexer. Important distinction, bestie.

All supported languages have default syntax files that are part of the distribution (peep the include/syntax subdirectory); they're also embedded in the re2ocaml binary. Users may provide a custom syntax file that overrides a few configurations for one of the supported languages, or they may choose to redefine all configurations — in which case the --lang none option should be used.

Syntax files contain configurations of four different kinds: feature lists, language configurations, inplace configurations and code templates. Four flavors, let's go 🍓

Feature lists

A few list configurations define various features supported by a given backend, so that re2ocaml may give a clear error if the user tries to enable an unsupported feature:

• supported_apis — a list of supported APIs with possible elements simple, record, generic.
• supported_api_styles — a list of supported API styles with possible elements functions, free-form.
• supported_code_models — a list of supported code models with possible elements goto-label, loop-switch, recursive-functions.
• supported_targets — a list of supported codegen targets with possible elements code, dot, skeleton.
• supported_features — a list of supported features with possible elements nested-ifs, bitmaps, computed-gotos, case-ranges, monadic, unsafe, tags, captures, captvars, computed-continue.

Language configurations

A few boolean configurations describe features of the target language that affect the re2ocaml parser and code generator:

• semicolons — non-zero if the language uses semicolons after statements.
• backtick_quoted_strings — non-zero if the language has backtick-quoted strings.
• single_quoted_strings — non-zero if the language has single-quoted strings.
• indentation_sensitive — non-zero if the language is indentation sensitive. (Python girlies, this one's for you 🐍)
• wrap_blocks_in_braces — non-zero if compound statements must be wrapped in curly braces.

Inplace configurations

Syntax files define initial values of all re2c: configurations, since they may differ for different languages. See the configurations section for a full list of all inplace configurations and their meaning 💖

Code templates

Code templates define syntax of the target language. They're written in a simple domain-specific language with the following formal grammar:

code-template ::
      name '=' code-exprs ';'
    | CODE_TEMPLATE ';'
    | '<undefined>' ';'

code-exprs ::
      <EMPTY>
    | code-exprs code-expr

code-expr ::
      STRING
    | VARIABLE
    | optional
    | list

optional ::
      '(' CONDITIONAL '?' code-exprs ')'
    | '(' CONDITIONAL '?' code-exprs ':' code-exprs ')'

list ::
      '[' VARIABLE ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER '}' ':' code-exprs ']'
    | '[' VARIABLE '{' NUMBER ',' NUMBER '}' ':' code-exprs ']'

A code template is a sequence of string literals, variables, optional elements and lists — or a reference to another code template, or the special value <undefined>.

Variables are placeholders that get substituted during the code generation phase. List variables are the special ones: when expanding list templates, re2ocaml repeats expressions on the right hand side of the column a few times, each time replacing occurrences of the list variable with a value specific to that repetition 🔁

Lists have optional bounds (negative values are counted from the end, e.g. -1 means the last element). Conditional names start with a dot. Both conditionals and variables may be either local (specific to the given code template) or global (allowed in all code templates). When re2ocaml reads a syntax file, it checks that each code template uses only the variables and conditionals that are allowed in it. Strict queen behavior, honestly.

For example, the following code template defines an if-then-else construct for a C-like language:

code:if_then_else =
    [branch{0}: topindent "if " cond " {" nl
        indent [stmt: stmt] dedent]
    [branch{1:-1}: topindent "} else" (.cond ? " if " cond) " {" nl
        indent [stmt: stmt] dedent]
    topindent "}" nl;

Here branch is a list variable: branch{0} expands to the first branch (which is special, since there's no else part), and branch{1:-1} expands to all remaining branches (if any). stmt is also a list variable: [stmt: stmt] is a nested list that expands to a list of statements in the body of the current branch. topindent, indent, dedent and nl are global variables, and .cond is a local conditional (their meaning is described below). This code template could produce the following code:

if x {
    // do something
} else if y {
    // do something else
} else {
    // don't do anything
}

Iconic, no notes 💅 Here's a list of all code templates supported by re2ocaml with their local variables and conditionals. Note that a particular definition may — but doesn't have to — use local variables and conditionals. Any unused code templates should be set to <undefined>.

• code:var_local — declaration or definition of a local variable. Supported variables: type (the type of the variable), name (its name) and init (initial value, if any). Conditionals: .init (true if there is an initializer).
• code:var_global — same as code:var_local, except that it's used in top-level.
• code:const_local — definition of a local constant. Supported variables: type (the type of the constant), name (its name) and init (initial value).
• code:const_global — same as code:const_local, except that it's used in top-level.

code:array_local

okay so this one is literally the definition of a local array (a table, bestie 💅). the variables it supports are:

• type — the type of the array elements
• name — the array's name
• size — its size
• row — a list variable that doesn't produce any code by itself, but expands the list expression as many times as there are rows in the table
• elem — a list variable that expands to all the table elements in the current row; it's meant to be nested inside the row list, iykyk

and the one conditional it supports is .const — true if the array is immutable. that's it, that's the whole vibe 🫶

code:array_global

same exact energy as code:array_local, no cap — the only difference is that this one gets used at top-level ✨

code:array_elem

a reference to a single element of an array (table). variables: array (the name of the array) and index (the index of the element). she's small but she's doing her job 👑

code:enum

the definition of an enumeration — and btw it can be defined either with a special language construct for enumerations or just as a few standalone constants, whichever. supported variables:

• type — the user-defined enumeration type, or the type of the constants
• elem — a list variable that expands to the name of each member
• init — the initializer for each member

conditionals: .init (true if there is an initializer). 💖

code:enum_elem

a single enumeration element — could be a member of a user-defined enumeration type or the name of a constant, depending on how code:enum is defined. supported variables: name (the name of the element) and type (its type). 🎀

code:assign

it's an assignment statement, periodt. supported variables: lhs (left hand side) and rhs (right hand side). like, the math is mathing 😌

code:type_int

signed integer type. that's her. ⭐

code:type_uint

unsigned integer type. her sister. ⭐

code:type_yybm

the type of the elements in the yybm table. 🦋

code:type_yytarget

the type of the elements in the yytarget table. 🌸

code:type_yyctable

the type of the elements in the yyctable table. 🧚

code:cmp_eq

operator "equals". == coded behaviour, no notes 💅

code:cmp_ne

operator "not equals". iconic, honestly.

code:cmp_lt

operator "less than". lowkey essential.

code:cmp_gt

operator "greater than". highkey essential.

code:cmp_le

operator "less or equal". bestie behaviour.

code:cmp_ge

operator "greater or equal". also bestie behaviour. 🔥

---

code:if_then_else

an if-then-else statement with one or more branches. supported variables:

• branch — a list variable that doesn't itself produce any code, but expands the list expression as many times as there are branches
• cond — the condition of the current branch
• stmt — a list variable that expands to all statements in the current branch

conditionals: .cond (true if the current branch has a condition) and .many (true if there's more than one branch). the way this handles multiple branches is genuinely so mother of it 👑

code:if_then_else_oneline

a specialization of code:if_then_else for when all the branches have one-line statements. and listen: if this is <undefined>, then code:if_then_else gets used instead. fr fr. 🙈

code:switch

a switch statement with one or more cases. supported variables: expr (the switched-on expression) and case (a list variable that expands to all the cases-groups along with their code blocks). ✨

code:switch_cases

a group of switch cases that all map to a single code block. supported variables are case (a list variable that expands to all the cases in this group) and stmt (a list variable that expands to all the statements in the code block). 🌈

code:switch_cases_oneline

a specialization of code:switch_cases for when the code block consists of a single one-line statement. and if this is <undefined>, code:switch_cases is used instead. same energy as its if-else sibling, iykyk 💜

code:switch_case_range

a single switch case that covers a range of values — possibly a range consisting of a single value, which is valid, don't @ me. supported variable: val (a list variable that expands to all the values in the range). supported conditionals:

• .many — true if there's more than one value in the range
• .char_literals — true if this is a switch on character literals — some languages provide special syntax for this case 🎀

code:switch_case_default

the default switch case. the main character of the fallback plan, honestly. 💀

code:loop

a loop that runs forever (unless it gets interrupted from the loop body). supported variables:

• label — the loop label
• val — initial state
• var — the state variable
• state — a list variable that expands to all the state names
• stmt — a list variable that expands to all the statements in the loop body, or in the current state

forever-loop era 😭🔥

code:continue

a continue statement. supported variables: label (the label from which to continue execution), val (the value passed to a labeled switch) and var (the state variable). 🫶

code:goto

a goto statement. supported variable: label (the label of the jump target). she jumps, she goes, she slays. 💅

code:cgoto

a computed goto statement. supported variables:

• array — the table containing the computed goto information
• index — the index of the element in the table
• base — the base label, only used if .cgoto.relative is true

code:cgoto:data

the initializer expression for a single element in the computed goto table. supported variables:

• label — the label used to initialize the current element
• type — the underlying type of the elements in the table
• base — the base label — only used if .cgoto.relative is true

---

code:fndecl

a function declaration. supported variables:

• name — function name
• type — return type
• throw — exceptions thrown by this function, maps to the re2c:yyfn:throw configuration
• arg — a list variable that doesn't itself produce any code, but expands the list expression as many times as there are function arguments
• argname — name of the current argument
• argtype — type of the current argument

conditional: .type (true if this is a non-void function). 🌟

code:fndef

like code:fndecl, but for function definitions — so she has one extra list variable, stmt, which expands to all the statements in the function body. the glow up is real 👑

code:fncall

a function call statement. supported variables: name (function name), retval (the l-value where the return value is stored, if any) and arg (a list variable that expands to all the function arguments). conditionals: .args (true if the function has arguments) and .retval (true if the return value needs to be saved). 🔥

code:tailcall

a tail call statement. supported variables: name (function name) and arg (a list variable that expands to all the function arguments). conditionals: .args (true if the function has arguments) and .retval (true if this is a non-void function). she's giving optimization, she's giving efficiency ✨

code:recursive_functions

the program body with the --recursive-functions code model. supported variables:

• fn — a list variable that does not itself produce any code, but expands the list expression as many times as there are functions
• fndecl — the declaration of the current function
• fndef — the definition of the current function

code:fingerprint

the fingerprint at the top of the generated output file. supported variables: ver (the re2ocaml version that was used to generate this) and date (the generation date). it's literally the birth certificate of your output file 😭💖

code:line_info

the format of line directives — and note: if this is set to <undefined>, then no directives get generated at all. supported variables: line (line number) and file (filename). 🎀

code:abort

a statement that aborts program execution. dramatic, but make it necessary. 💀

---

code:yydebug

a YYDEBUG statement, possibly specialized for different APIs. supported variables: YYDEBUG, yyrecord, yych (these map to the corresponding re2c: configurations) and state (DFA state number). 🐛✨

code:yypeek

a YYPEEK statement, possibly specialized for different APIs. supported variables: YYPEEK, YYCTYPE, YYINPUT, YYCURSOR, yyrecord and yych (they map to the corresponding re2c: configurations). conditionals: .cast (true if re2c:yych:conversion is set to non-zero). 👀

code:yyskip

a YYSKIP statement, possibly specialized for different APIs. supported variables: YYSKIP, YYCURSOR and yyrecord (they map to the corresponding re2c: configurations). 🦋

code:yybackup

a YYBACKUP statement, possibly specialized for different APIs. supported variables: YYBACKUP, YYCURSOR, YYMARKER and yyrecord (they map to the corresponding re2c: configurations). 💾

code:yybackupctx

a YYBACKUPCTX statement, possibly specialized for different APIs. supported variables: YYBACKUPCTX, YYCURSOR, YYCTXMARKER and yyrecord (they map to the corresponding re2c: configurations). 📌

code:yyskip_yypeek

a combined code:yyskip plus code:yypeek statement — and it defaults to code:yyskip followed by code:yypeek. two besties, one bullet, no cap 💅

code:yypeek_yyskip

a combined code:yypeek plus code:yyskip statement — defaults to code:yypeek followed by code:yyskip. yes, the order matters, that's literally the whole point 🫶

code:yyskip_yybackup

a combined code:yyskip plus code:yybackup statement — defaults to code:yyskip followed by code:yybackup. 🌸

code:yybackup_yyskip

a combined code:yybackup plus code:yyskip statement — defaults to code:yybackup followed by code:yyskip. 🧚

code:yybackup_yypeek

a combined code:yybackup plus code:yypeek statement — defaults to code:yybackup followed by code:yypeek. 🌈

code:yyskip_yybackup_yypeek

a combined code:yyskip, code:yybackup and code:yypeek statement — defaults to code:yyskip followed by code:yybackup followed by code:yypeek. a whole trio, the friend group we all wanted 🌟

code:yybackup_yypeek_yyskip

a combined code:yybackup, code:yypeek and code:yyskip statement — defaults to code:yybackup followed by code:yypeek followed by code:yyskip. same trio, different order, equally valid 💜

code:yyrestore

a YYRESTORE statement, possibly specialized for different APIs. supported variables: YYRESTORE, YYCURSOR, YYMARKER and yyrecord (they map to the corresponding re2c: configurations). 💫

• code:yyrestorectx — the YYRESTORECTX statement, possibly specialized for different APIs ✨ Supported variables: YYRESTORECTX, YYCURSOR, YYCTXMARKER, yyrecord (these map to the corresponding re2c: configurations).
• code:yyrestoretag — the YYRESTORETAG statement, possibly specialized for different APIs 💅 Supported variables: YYRESTORETAG, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), and tag (the name of the tag variable used to restore position).
• code:yyshift — the YYSHIFT statement, possibly specialized for different APIs 🦋 Supported variables: YYSHIFT, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), and offset (the number of code units to shift the current position).
• code:yyshiftstag — the YYSHIFTSTAG statement, possibly specialized for different APIs 🌸 Supported variables: YYSHIFTSTAG, yyrecord, negative (map to the corresponding re2c: configurations), tag (tag variable which needs to be shifted), offset (the number of code units to shift). Conditionals: .nested (true if this is a nested tag — in this case its value may equal to re2c:tags:negative, which should not be shifted 🙈).
• code:yyshiftmtag — the YYSHIFTMTAG statement, possibly specialized for different APIs 🎀 Supported variables: YYSHIFTMTAG (maps to the corresponding re2c: configuration), tag (tag variable which needs to be shifted), offset (the number of code units to shift).
• code:yystagp — the YYSTAGP statement, possibly specialized for different APIs 💖 Supported variables: YYSTAGP, YYCURSOR, yyrecord (map to the corresponding re2c: configurations), tag (tag variable that should be updated).
• code:yymtagp — the YYMTAGP statement, possibly specialized for different APIs ⭐ Supported variables: YYMTAGP (maps to the corresponding re2c: configuration), tag (tag variable that should be updated).
• code:yystagn — the YYSTAGN statement, possibly specialized for different APIs 🧚 Supported variables: YYSTAGN, negative, yyrecord (map to the corresponding re2c: configurations), tag (tag variable that should be updated).
• code:yymtagn — the YYMTAGN statement, possibly specialized for different APIs 💜 Supported variables: YYMTAGN (maps to the corresponding re2c: configuration), tag (tag variable that should be updated).
• code:yycopystag — the YYCOPYSTAG statement, possibly specialized for different APIs 🍓 Supported variables: YYCOPYSTAG, yyrecord (map to the corresponding re2c: configurations), lhs, rhs (left and right hand side tag variables of the copy operation).
• code:yycopymtag — the YYCOPYMTAG statement, possibly specialized for different APIs 🌈 Supported variables: YYCOPYMTAG, yyrecord (map to the corresponding re2c: configurations), lhs, rhs (left and right hand side tag variables of the copy operation).
• code:yygetaccept — the YYGETACCEPT statement, possibly specialized for different APIs 🔥 Supported variables: YYGETACCEPT, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yyaccept configuration).
• code:yysetaccept — the YYSETACCEPT statement, possibly specialized for different APIs 👑 Supported variables: YYSETACCEPT, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yyaccept configuration) and val (numeric value of the accepted rule).
• code:yygetcond — the YYGETCOND statement, possibly specialized for different APIs 🦋 Supported variables: YYGETCOND, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yycond configuration).
• code:yysetcond — the YYSETCOND statement, possibly specialized for different APIs ✨ Supported variables: YYSETCOND, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yycond configuration) and val (numeric condition identifier).
• code:yygetstate — the YYGETSTATE statement, possibly specialized for different APIs 💅 Supported variables: YYGETSTATE, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yystate configuration).
• code:yysetstate — the YYSETSTATE statement, possibly specialized for different APIs 🎀 Supported variables: YYSETSTATE, yyrecord (map to the corresponding re2c: configurations), var (maps to the re2c:yystate configuration) and val (state number).
• code:yylessthan — the YYLESSTHAN statement, possibly specialized for different APIs 🌸 Supported variables: YYLESSTHAN, YYCURSOR, YYLIMIT, yyrecord (map to the corresponding re2c: configurations), need (the number of code units to check against). Conditional: .many (true if the need is more than one).
• code:yyend — the YYEND expression, possibly specialized for different APIs 🫶 Supported variables: YYEND, YYCURSOR, YYLIMIT.
• code:yybm_filter — a condition that is used to filter out yych values that are not covered by the yybm table (used with --bitmaps option) 🌈 Supported variable: yych (maps to the re2c:yych configuration).
• code:yybm_match — the format of the yybm table check (generated with --bitmaps option) 💀 Supported variables: yybm, yych (map to the corresponding re2c: configurations), offset (offset in the yybm table that needs to be added to yych) and mask (bit mask that should be applied to the table entry to retrieve the boolean value that needs to be checked).
• code:yytarget_filter — a condition that is used to filter out yych values that are not covered by the yytarget table (used with --computed-gotos option) ⭐ Supported variable: yych (maps to the re2c:yych configuration).

---

okay so that was the code: list, and now here's a list of all the global variables that are allowed in syntax files, no cap:

• nl — a newline 🎀
• indent — a variable that does not produce any code, but has a side-effect of increasing the indentation level 🦋
• dedent — a variable that does not produce any code, but has a side-effect of decreasing the indentation level 💅
• topindent — the indentation string for the current statement. Indentation level is tracked and automatically updated by the code generator ✨

---

and here's a list of all the global conditionals that are allowed in syntax files (these are giving very much "if this then that" energy 👑):

• .api.simple — true if the simple API is used (--api simple or re2c:api = simple).
• .api.generic — true if the generic API is used (--api generic or re2c:api = generic).
• .api.record — true if the record API is used (--api record or re2c:api = record).
• .api_style.functions — true if function-like API style is used (re2c:api-style = functions).
• .api_style.freeform — true if free-form API style is used (re2c:api-style = free-form).
• .case_ranges — true if the case ranges feature is enabled (--case-ranges or re2c:case-ranges = 1).
• .cgoto.relative — true if the relative form of computed goto is used (--computed-gotos-relative or re2c:cgoto:relative = 1).
• .computed_continue — true if the loop/switch model uses labeled-switch transitions (--computed-continue).
• .code_model.goto_label — true if the code model based on goto/label is used (--goto-label).
• .code_model.loop_switch — true if the code model based on loop/switch is used (--loop-switch).
• .code_model.recursive_functions — true if the code model based on recursive functions is used (--recursive-function).
• .date — true if the generated fingerprint should contain the generation date 🎀
• .loop_label — true if re2ocaml generated loops must have a label (re2c:label:yyloop is set to a nonempty string).
• .monadic — true if the generated code should be monadic (re2c:monadic = 1). This is only relevant for pure functional languages 🧚
• .start_conditions — true if start conditions are enabled (--start-conditions).
• .storable_state — true if storable state is enabled (--storable-state).
• .unsafe — true if re2ocaml should use "unsafe" blocks in order to generate faster code (--unsafe, re2c:unsafe = 1). This is only relevant for languages that have the "unsafe" feature 💀
• .version — true if the generated fingerprint should contain the re2ocaml version.
• .yyfill.enable — true if YYFILL is enabled (re2c:yyfill:enable = 1).
• .yyfn.throw — true if the re2c:yyfn:throw configuration is defined to a nonempty string.

---

💀 HANDLING THE END OF INPUT 💀

okay so this is the part where it gets real, bestie 😭 one of the main problems for the lexer is literally knowing when to stop. and there are a few terminating conditions:

• the lexer may match some rule (including the default rule *) and come to a final state 💖
• the lexer may fail to match any rule and come to a default state 🫶
• the lexer may reach the end of input 🌸

The first two conditions terminate the lexer in a "natural" way: it comes to a state with no outgoing transitions, and the matching automatically stops ⭐ the third condition, end of input, is different: it may happen in any state, and the lexer should be able to handle it. Checking for the end of input interrupts the normal lexer workflow and adds conditional branches to the generated program, therefore it is necessary to minimize the number of such checks 🙈 re2ocaml supports a few different methods for handling the end of input. which one to use depends on the complexity of regular expressions, the need for buffering, performance considerations and other factors. here is a list of methods: ✨

• Sentinel — ngl, this one is the "no thoughts, head empty" of end-of-input handling 💅 It just eliminates the end-of-input checks entirely. Simple, efficient, but lowkey limited: it only works when there's a natural "sentinel" character that can never occur in valid input. That character may still show up in invalid input, but your regular expressions shouldn't allow it — except, maybe, as the very last character of a rule. The sentinel gets appended at the end of input and serves as a stop signal: when the lexer reads this character, it's either a syntax error or the end of input. Either way, the lexer should stop. This method is used if YYFILL is disabled with re2c:yyfill:enable = 0; and re2c:eof has the default value -1.
• Sentinel with bounds checks — this one's the flexible icon ✨ it allows you to handle any input, no restrictions on the regular expressions. The idea: cut down the number of end-of-input checks by only performing them on certain characters. Like the "sentinel" method, one of the characters is chosen as a "sentinel" and appended at the end of input. But there's no restriction on where the sentinel may occur — fr fr, any character can be chosen for a sentinel. When the lexer reads this character, it additionally performs a bounds check. If the current position is within bounds, the lexer resumes matching and handles the sentinel as a regular character. Otherwise it invokes YYFILL (unless it's disabled). If more input is supplied, the lexer will rematch the last character and continue as if the sentinel wasn't there. Otherwise it must be the real end of input, and the lexer stops. This method is used when re2c:eof has a non-negative value (it should be set to the numeric value of the sentinel). YYFILL is optional.
• Bounds checks with padding — generic, and it may be faster than the "sentinel with bounds checks" method, but it's also more complex 🫠 The idea is to partition DFA states into strongly connected components (SCCs) and generate a single check per SCC for enough characters to cover the longest non-looping path in that SCC. This reduces the number of checks, but there's a problem with short lexemes at the end of input, because the check requires enough characters to cover the longest lexeme. Fix? Pad the input with a few fake characters that do not form a valid lexeme suffix (so the lexer literally can't match them). The length of padding should be YYMAXFILL, generated with a max block. If there's not enough input, the lexer invokes YYFILL, which should supply at least the required number of characters or not return. This method is used if YYFILL is enabled and re2c:eof is -1 (this is the default configuration).
• Custom checks — the generic API lets you override basic operations like reading a character, which makes it possible to fold the end-of-input checks into them. Heads up though: this approach is error-prone and should be used with caution 💀 To use a custom method, enable generic API with --api custom or re2c:api = custom; and disable default bounds checks with re2c:yyfill:enable = 0; or re2c:yyfill:check = 0;.

The following subsections contain an example of each method 🎀

---

🦋 Sentinel

This example uses a sentinel character to handle the end of input. The program counts space-separated words in a null-terminated string. The sentinel is null: it's the last character of each input string, and none of the rules allow it in the middle of a lexeme (in particular, it's not included in character ranges, where it is easy to overlook 👀). If a null occurs in the middle of a string, it's a syntax error and the lexer will match the default rule *, but it won't read past the end of input or crash (use -Wsentinel-in-midrule warning and re2c:sentinel configuration to verify this). Configuration re2c:yyfill:enable = 0; suppresses the generation of bounds checks and YYFILL invocations.

(* re2ocaml $INPUT -o $OUTPUT *)

open String

type state = {
    yyinput: string;
    mutable yycursor: int;
}

(* expect a null-terminated string *)
%{
    re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
    re2c:yyfill:enable = 0;

    *      { -1 }
    [\x00] { count }
    [a-z]+ { lex yyrecord (count + 1) }
    [ ]+   { lex yyrecord count }
%}

let test(yyinput, count) =
    let st = {yyinput = yyinput; yycursor = 0}
    in if not (lex st 0 = count) then raise (Failure "error")

let main () =
    test("", 0);
    test("one two three", 3);
    test("f0ur", -1)

let _ = main ()

---

✨ Sentinel with bounds checks ✨

This example uses sentinel with bounds checks to handle the end of input (this method was added in version 1.2). The program counts space-separated single-quoted strings. The sentinel character is null, specified with re2c:eof = 0; configuration. Same as in the sentinel method, null is the last character of each input string — but here it is allowed in the middle of a rule (for example, 'aaa\0aa'\0 is valid input, but 'aaa\0 is a syntax error). Bounds checks are generated in each state that matches an input character, but they're scoped to the branch that handles null 🔍 Bounds checks look like YYLIMIT <= YYCURSOR or YYLESSTHAN(1) with generic API. If the check condition is true, the lexer has reached the end of input and should stop (YYFILL is disabled with re2c:yyfill:enable = 0; since the input fits into one buffer — see the YYFILL with sentinel section for an example that uses YYFILL). Reaching the end of input opens three possibilities: if the lexer is in the initial state it will match the end-of-input rule $, otherwise it may fallback to a previously matched rule (including default rule *) or go to a default state, causing -Wundefined-control-flow.

(* re2ocaml $INPUT -o $OUTPUT *)

open String

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable yymarker: int;
    yylimit: int;
}

(* expect a null-terminated string *)
%{
    re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
    re2c:yyfill:enable = 0;
    re2c:eof = 0;

    str = ['] ([^'\\] | [\\][^])* ['];

    *    { -1 }
    $    { count }
    str  { lex yyrecord (count + 1) }
    [ ]+ { lex yyrecord count }
%}

let test(str, count) =
    let st = {
        yyinput = str;
        yycursor = 0;
        yymarker = 0;
        yylimit = length str - 1; (* terminating null not included *)
    }
    in if not (lex st 0 = count) then raise (Failure "error")

let main () =
    test("", 0);
    test("'qu\x00tes' 'are' 'fine: \\'' ", 3);
    test("'unterminated\\'", -1)

let _ = main ()

---

🎀 Bounds checks with padding 🎀

This example uses bounds checks with padding to handle the end of input (this method is enabled by default, bestie — it's the main character here 👑). The program counts space-separated single-quoted strings. There's a padding of YYMAXFILL null characters appended at the end of input, where the YYMAXFILL value is autogenerated with a max block. You don't have to use null for padding — any characters work, as long as they don't form a valid lexeme suffix (in this example padding should not contain single quotes, since they may be mistaken for a suffix of a single-quoted string). There's a "stop" rule that matches the first padding character (null) and terminates the lexer (note that it checks whether null is at the beginning of padding — otherwise it's a syntax error). Bounds checks are generated only in some states, determined by the strongly connected components of the underlying automaton. Checks have the form (YYLIMIT - YYCURSOR) < n or YYLESSTHAN(n) with generic API, where n is the minimum number of characters needed for the lexer to proceed (which also means the next bounds check will occur in at most n characters). If the check condition is true, the lexer has reached the end of input and will invoke YYFILL(n), which should either supply at least n input characters or not return. In this example YYFILL always fails and terminates the lexer with an error (totally fine, because the input fits into one buffer). See the YYFILL with padding section for an example that refills the input buffer with YYFILL 💖

(* re2ocaml $INPUT -o $OUTPUT *)

open String

exception Fill

type state = {
    yyinput: string;
    mutable yycursor: int;
    yylimit: int;
}

%{max %}
%{
    re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
    re2c:YYFILL = "raise Fill;";

    str = ['] ([^'\\] | [\\][^])* ['];

    [\x00] {
        (* check that it is the sentinel, not some unexpected null *)
        if yyrecord.yycursor = length yyrecord.yyinput - yymaxfill + 1 then count else -1
    }
    str  { lex yyrecord (count + 1) }
    [ ]+ { lex yyrecord count }
    *    { -1 }
%}

let test(str, count) =
    let buf = cat str (make yymaxfill '\x00') in
    let st = {yyinput = buf; yycursor = 0; yylimit = length buf} in
    let result = try lex st 0 with Fill -> -1 in
    if not (result = count) then raise (Failure "error")

let main () =
    test("", 0);
    test("'unterminated\\'", -1);
    test("'qu\x00tes' 'are' 'fine: \\'' ", 3);
    test("'unexpected \x00 null", -1)

let _ = main ()

---

💅 Custom checks 💅

This example uses a custom end-of-input handling method based on generic API. The program counts space-separated single-quoted strings. It's the same as the sentinel example, except the input is not null-terminated. To cover up for the absence of a sentinel character at the end of input, YYPEEK is redefined to perform a bounds check before it reads the next input character. Heads up: this is inefficient, because checks are done very often 😭 If the check condition fails, YYPEEK returns the real character, otherwise it returns a fake sentinel character.

(* re2ocaml $INPUT -o $OUTPUT *)

type state = {
    str: string;
    mutable cur: int;
    lim: int;
}

(* expect a string without terminating null *)
%{
    re2c:api = generic;
    re2c:YYFN = ["lex;int", "st;state", "count;int"];
    re2c:YYPEEK = "if st.cur < st.lim then st.str.[st.cur] else '\\x00'";
    re2c:YYSKIP = "st.cur <- st.cur + 1;";
    re2c:yyfill:enable = 0;

    *      { -1 }
    [\x00] { count }
    [a-z]+ { lex st (count + 1) }
    [ ]+   { lex st count }
%}

…and the tail end of that example goes like this, bestie:

let test(str, count) =
    let st = {str = str; cur = 0; lim = String.length str}
    in if not (lex st 0 = count) then raise (Failure "error")

let main () =
    test("", 0);
    test("one two three", 3);
    test("f0ur", -1)

let _ = main ()

---

🫗 BUFFER REFILLING 🫗

okay so like, why do we even need buffering? because sometimes you literally cannot map the input into memory all at once: either it's too large, or it arrives in a streaming fashion (think reading from a socket, very "it's giving live data" 💅). The usual technique in those cases is to allocate a fixed-sized memory buffer and process the input in chunks that fit into the buffer. When the current chunk is processed, it gets moved out and new data gets moved in.

In practice it's somewhat more complex, because lexer state isn't just one single input position — it's a whole lil squad of interrelated positions 🫶:

• cursor: the next input character to be read (YYCURSOR in C pointer API or YYSKIP/YYPEEK in generic API)
• limit: the position after the last available input character (YYLIMIT in C pointer API, implicitly handled by YYLESSTHAN in generic API)
• marker: the position of the most recent match, if any (YYMARKER in default API or YYBACKUP/YYRESTORE in generic API)
• token: the start of the current lexeme (implicit in re2ocaml API, as it is not needed for the normal lexer operation and can be defined and updated by the user)
• context marker: the position of the trailing context (YYCTXMARKER in C pointer API or YYBACKUPCTX/YYRESTORECTX in generic API)
• tag variables: submatch positions (defined with stags and mtags blocks and generic API primitives YYSTAGP/YYSTAGN/YYMTAGP/YYMTAGN)

Not all of these are used in every case, but if they are used, they must be updated by YYFILL. All active positions are contained in the segment between token and cursor, therefore everything between buffer start and token can be discarded, the segment from token and up to limit should be moved to the beginning of buffer, and the free space at the end of buffer should be filled with new data. And to avoid frequent YYFILL calls, it's best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer — go big or go home, honestly ✨).

The details of YYFILL implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds-checking with padding. Also note that if -f --storable-state option is used, YYFILL has slightly different semantics (described in the section about storable state).

🌊 YYFILL with sentinel 🌊

If EOF rule is used, YYFILL is a function-like primitive that accepts no arguments and returns a value which is checked against zero. YYFILL invocation is triggered by condition YYLIMIT <= YYCURSOR in C pointer API and YYLESSTHAN() in generic API. A non-zero return value means that YYFILL has failed. A successful YYFILL call must supply at least one character and adjust input positions accordingly. Limit must always be set to one after the last input position in buffer, and the character at the limit position must be the sentinel symbol specified by re2c:eof configuration.

The pictures below show the relative locations of input positions in buffer before and after YYFILL call (sentinel symbol is marked with #, and the second picture shows the case when there is not enough input to fill the whole buffer).

             <-- shift -->
           >-A------------B---------C-------------D#-----------E->
           buffer       token    marker         limit,
                                                cursor
>-A------------B---------C-------------D------------E#->
           buffer,  marker        cursor        limit
           token

             <-- shift -->
           >-A------------B---------C-------------D#--E (EOF)
           buffer       token    marker         limit,
                                                cursor
>-A------------B---------C-------------D---E#........
           buffer,  marker       cursor limit
           token

Here's an example of a program that reads input file input.txt in chunks of 4096 bytes and uses EOF rule, bestie 🎀:

(* re2ocaml $INPUT -o $OUTPUT *)

open Bytes

let bufsize = 4096

type state = {
    file: in_channel;
    yyinput: bytes;
    mutable yycursor: int;
    mutable yymarker: int;
    mutable yylimit: int;
    mutable token: int;
    mutable eof: bool;
}

type status = Ok | Eof | LongLexeme

let fill(st: state) : status =
    if st.eof then Eof else

    (* Error: lexeme too long. In real life could reallocate a larger buffer. *)
    if st.token < 1 then LongLexeme else (

    (* Shift buffer contents (discard everything up to the current token). *)
    blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
    st.yycursor <- st.yycursor - st.token;
    st.yymarker <- st.yymarker - st.token;
    st.yylimit <- st.yylimit - st.token;
    st.token <- 0;

    (* Fill free space at the end of buffer with new data from file. *)
    let n = input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in (* -1 for sentinel *)
    st.yylimit <- st.yylimit + n;
    if n = 0 then
        st.eof <- true; (* end of file *)
        set st.yyinput st.yylimit '\x00'; (* append sentinel *)

    Ok)

%{
    re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
    re2c:YYFILL = "fill yyrecord = Ok";
    re2c:eof = 0;

    str = ['] ([^'\\] | [\\][^])* ['];

    *    { -1 }
    $    { count }
    str  { lex_loop yyrecord (count + 1) }
    [ ]+ { lex_loop yyrecord count }
%}

and lex_loop st count =
    st.token <- st.yycursor;
    lex st count

let main () =
    let fname = "input" in

    (* Prepare input file. *)
    Out_channel.with_open_bin fname
        (fun oc -> for i = 1 to bufsize do
            output_string oc "'qu\x00tes' 'are' 'fine: \\'' "
        done);

    (* Run lexer on the prepared file. *)
    In_channel.with_open_bin fname
        (fun ic ->
            let yylimit = bufsize - 1 in
            let st = {
                file = ic;
                yyinput = create bufsize;
                yycursor = yylimit;
                yymarker = yylimit;
                yylimit = yylimit;
                token = yylimit;
                eof = false;
            } in if not (lex_loop st 0 = 3 * bufsize) then
                raise (Failure "error"));

    (* Cleanup. *)
    Sys.remove fname

let _ = main ()

🧸 YYFILL with padding 🧸

In the default case (when EOF rule is not used) YYFILL is a function-like primitive that accepts a single argument and does not return any value. YYFILL invocation is triggered by condition (YYLIMIT - YYCURSOR) < n in C pointer API and YYLESSTHAN(n) in generic API. The argument passed to YYFILL is the minimal number of characters that must be supplied. If it fails to do so, YYFILL must not return to the lexer (for that reason it is best implemented as a macro that returns from the calling function on failure). In case of a successful YYFILL invocation the limit position must be set either to one after the last input position in buffer, or to the end of YYMAXFILL padding (in case YYFILL has successfully read at least n characters, but not enough to fill the entire buffer).

fr fr, if that fill fails, do not come back to the lexer. just leave. exit. it's not your moment 🙈

The pictures below show the relative locations of input positions in buffer before and after YYFILL invocation (YYMAXFILL padding on the second picture is marked with # symbols).

             <-- shift -->                 <-- need -->
           >-A------------B---------C-----D-------E---F--------G->
           buffer       token    marker cursor  limit

>-A------------B---------C-----D-------E---F--------G->
           buffer,  marker cursor               limit
           token

             <-- shift -->                 <-- need -->
           >-A------------B---------C-----D-------E-F        (EOF)
           buffer       token    marker cursor  limit

>-A------------B---------C-----D-------E-F###############
           buffer,  marker cursor                   limit
           token                        <- YYMAXFILL ->

And here's an example of a program that reads input file input.txt in chunks of 4096 bytes and uses bounds-checking with padding 💖:

(* re2ocaml $INPUT -o $OUTPUT *)

open Bytes

%{max %}
let bufsize = 4096

exception Fill

type state = {
    file: in_channel;
    yyinput: bytes;
    mutable yycursor: int;
    mutable yymarker: int;
    mutable yylimit: int;
    mutable token: int;
    mutable eof: bool;
}

type status = Ok | Eof | LongLexeme

let fill (st: state) (need: int) : status =
    if st.eof then Eof else

    (* Error: lexeme too long. In real life could reallocate a larger buffer. *)
    if st.token < need then LongLexeme else (

    (* Shift buffer contents (discard everything up to the current token). *)
    blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
    st.yycursor <- st.yycursor - st.token;
    st.yymarker <- st.yymarker - st.token;
    st.yylimit <- st.yylimit - st.token;
    st.token <- 0;

    (* Fill free space at the end of buffer with new data from file. *)
    let n = input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in (* -1 for sentinel *)
    st.yylimit <- st.yylimit + n;

    (* If read zero characters, this is end of input => add zero padding
       so that the lexer can access characters at the end of buffer. *)
    if n = 0 then
        st.eof <- true; (* end of file *)
        for i = 0 to (yymaxfill - 1) do
            set st.yyinput (st.yylimit + i) '\x00';
            st.yylimit <- st.yylimit + yymaxfill
        done;

    Ok)

%{
    re2c:YYFN = ["lex;int", "yyrecord;state", "count;int"];
    re2c:YYFILL = "if not (fill yyrecord @@ = Ok) then raise Fill;";

    str = ['] ([^'\\] | [\\][^])* ['];

    [\x00] {
        (* check that it is the sentinel, not some unexpected null *)
        if yyrecord.token = yyrecord.yylimit - yymaxfill then count else -1
    }
    str  { lex_loop yyrecord (count + 1) }
    [ ]+ { lex_loop yyrecord count }
    *    { -1 }
%}

          and lex_loop st count =
              st.token <- st.yycursor;
              try lex st count with Fill -> -1

          let main () =
              let fname = "input" in

              (* Prepare input file. *)
              Out_channel.with_open_bin fname
                  (fun oc -> for i = 1 to bufsize do
                      output_string oc "'qu\x00tes' 'are' 'fine: \\'' "
                  done);

              (* Run lexer on the prepared file. *)
              In_channel.with_open_bin fname
                  (fun ic ->
                      let yylimit = bufsize - yymaxfill in
                      let st = {
                          file = ic;
                          yyinput = create bufsize;
                          yycursor = yylimit;
                          yymarker = yylimit;
                          yylimit = yylimit;
                          token = yylimit;
                          eof = false;
                      } in if not (lex_loop st 0 = 3 * bufsize) then
                          raise (Failure "error"));

              (* Cleanup. *)
              Sys.remove fname

          let _ = main ()

✨ FEATURES ✨

🧩 Multiple blocks

Okay so sometimes you literally need multiple interrelated lexers — like when there’s a high-level state machine that transitions between lexer modes, and one lexer alone is just not giving. 💅 This can be implemented using multiple connected re2ocaml blocks. Another option is to use start conditions. Both iconic, bestie. ✨

The way connections between blocks are implemented depends on the target language, no cap. In languages that have a goto statement (such as C/C++ and Go), you can have all blocks in one function, each of them prefixed with a label. Transition from one block to another is a simple goto. In languages that do not have goto (such as Rust), it is necessary to use a loop with a switch on a state variable, similar to the yystate loop/switch generated by re2ocaml, or else wrap each block in a function and use function calls. 🫶

The example below uses multiple blocks to parse binary, octal, decimal and hexadecimal numbers. Each base has its own block. The initial block determines base and dispatches to other blocks. Common configurations are defined in a separate block at the beginning of the program; they are inherited by the other blocks. Slay. 👑

          (* re2ocaml $INPUT -o $OUTPUT -i *)

          open Int64
          open Option
          open String

          type state = {
              yyinput: string;
              mutable yycursor: int;
              mutable yymarker: int;
          }

          let add (num: int option) (dgt: int) (base: int) : int option =
              match num with
                  | None -> None
                  | Some n ->
                      let n' = add (mul (of_int n) (of_int base)) (of_int dgt)
                      in if n' > (of_int32 Int32.max_int) then None else Some (to_int n')

          %{
              re2c:yyrecord = "st";
              re2c:yyfill:enable = 0;
          %}

          %{local
              re2c:YYFN = ["parse_bin;int option", "st;state", "num;int option"];
              [01] { parse_bin st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 2) }
              *    { num }
          %}

          %{local
              re2c:YYFN = ["parse_oct;int option", "st;state", "num;int option"];
              [0-7] { parse_oct st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 8) }
              *     { num }
          %}

          %{local
              re2c:YYFN = ["parse_dec;int option", "st;state", "num;int option"];
              [0-9] { parse_dec st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 10) }
              *     { num }
          %}

          %{local
              re2c:YYFN = ["parse_hex;int option", "st;state", "num;int option"];
              [0-9] { parse_hex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 16) }
              [a-f] { parse_hex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 87) 16) }
              [A-F] { parse_hex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 55) 16) }
              *     { num }
          %}

          %{local
              re2c:YYFN = ["parse;int option", "st;state"];
              '0b' / [01]        { parse_bin st (Some 0) }
              "0"                { parse_oct st (Some 0) }
              "" / [1-9]         { parse_dec st (Some 0) }
              '0x' / [0-9a-fA-F] { parse_hex st (Some 0) }
              *                  { None }
          %}

          let test (yyinput: string) (result: int option) =
              let st = {yyinput = yyinput; yycursor = 0; yymarker = 0} in
              if not (parse st = result) then raise (Failure "error")

          let main () =
              test "\x00" None;
              test "1234567890\x00" (Some 1234567890);
              test "0b1101\x00" (Some 13);
              test "0x7Fe\x00" (Some 2046);
              test "0644\x00" (Some 420);
              test "9999999999\x00" None

          let _ = main ()

🎛️ Start conditions

Start conditions are enabled with --start-conditions option. They provide a way to encode multiple interrelated automata within the same re2ocaml block. Main character energy fr fr. ✨

Each condition corresponds to a single automaton and has a unique name specified by the user and a unique internal number defined by re2ocaml. The numbers are used to switch between conditions: the generated code uses YYGETCOND and YYSETCOND primitives to get the current condition or set it to the given number. Use the conditions block, --header option or re2c:header configuration to generate numeric condition identifiers. Configuration re2c:cond:enumprefix specifies the generated identifier prefix. 🎀

In condition mode every rule must be prefixed with a list of comma-separated condition names in angle brackets, or a wildcard <*> to denote all conditions. The rule syntax is extended as follows:

• < condition-list > regular-expression code — a rule that is merged to every condition on the condition-list. It matches regular-expression and executes the associated code.
• < condition-list > regular-expression => condition code — a rule that is merged to every condition on the condition-list. It matches regular-expression, sets the current condition to condition and executes the associated code.
• < condition-list > regular-expression :=> condition — a rule that is merged to every condition on the condition-list. It matches regular-expression and immediately transitions to condition (there is no semantic action).
• < condition-list > !action code — a rule that binds code to the place defined by action in every condition on the condition-list (see the actions section for various types of actions).
• <! condition-list > code — a rule that prepends code to semantic actions of all rules for every condition on the condition-list. This syntax is deprecated and the !pre_rule action should be used instead (it does exactly the same).
• < > code — a rule that creates a special entry condition with number zero and name "0" that executes code before jumping to other conditions. This syntax is deprecated, and the !entry action should be used instead (it provides a more fine-grained control, as the code can be specified on a per-condition basis, and one can jump directly to condition start without going through condition dispatch).
• < > => condition code — same as the previous rule, except that it sets the next condition.
• < > :=> condition — same as the previous rule, except that it has no associated code and immediately jumps to condition.

The code re2ocaml generates for conditions depends on whether re2ocaml uses goto/label approach or loop/switch approach to encode the automata. Vibe check: language-dependent, bestie. 💅

In languages that have goto statement (such as C/C++ and Go) conditions are naturally implemented as blocks of code prefixed with labels of the form yyc_<cond>, where cond is a condition name (label prefix can be changed with re2c:cond:prefix). Transitions between conditions are implemented using goto and condition labels. Before all conditions re2ocaml generates an initial switch on YYGETSTATE that jumps to the start state of the current condition. The shortcut rules :=> bypass the initial switch and jump directly to the specified condition (re2c:cond:goto can be used to change the default behavior). The rules with semantic actions do not automatically jump to the next condition; this should be done by the user-defined action code. 👑

In languages that do not have goto (such as Rust) re2ocaml reuses the yystate variable to store condition numbers. Each condition gets a numeric identifier equal to the number of its start state, and a switch between conditions is no different than a switch between DFA states of a single condition. There is no need for a separate initial condition switch. (Since the same approach is used to implement storable states, YYGETCOND/YYSETCOND are redundant if both storable states and conditions are used). 🦋

The program below uses start conditions to parse binary, octal, decimal and hexadecimal numbers. There is a single block where each base has its own condition, and the initial condition is connected to all of them. User-defined variable cond stores the current condition number; it is initialized to the number of the initial condition generated with conditions block. It’s giving organized chaos and we’re obsessed. 🍓

          (* re2ocaml $INPUT -o $OUTPUT -ci *)

          open Int64
          open Option
          open String

          %{conditions %}

          type state = {
              yyinput: string;
              mutable yycursor: int;
              mutable yymarker: int;
              mutable yycond: yycondtype;
          }

          let add (num: int option) (dgt: int) (base: int) : int option =
              match num with
                  | None -> None
                  | Some n ->
                      let n' = add (mul (of_int n) (of_int base)) (of_int dgt)
                      in if n' > (of_int32 Int32.max_int) then None else Some (to_int n')

          %{
              re2c:YYFN = ["parse;int option", "st;state", "num;int option"];
              re2c:yyrecord = "st";
              re2c:yyfill:enable = 0;

              <init> '0b' / [01]        :=> bin
              <init> "0"                :=> oct
              <init> "" / [1-9]         :=> dec
              <init> '0x' / [0-9a-fA-F] :=> hex
              <init> * { None }

              <bin> [01]  { yyfnbin st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 2) }
              <oct> [0-7] { yyfnoct st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 8) }
              <dec> [0-9] { yyfndec st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 10) }
              <hex> [0-9] { yyfnhex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 48) 16) }
              <hex> [a-f] { yyfnhex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 87) 16) }
              <hex> [A-F] { yyfnhex st (add num (Char.code st.yyinput.[st.yycursor - 1] - 55) 16) }

              <bin, oct, dec, hex> * { num }
          %}

let test (yyinput: string) (result: int option) =
    let st = {yyinput = yyinput; yycursor = 0; yymarker = 0; yycond = YYC_init} in
    if not (parse st (Some 0) = result) then raise (Failure "error")

let main () =
    test "\x00" None;
    test "1234567890\x00" (Some 1234567890);
    test "0b1101\x00" (Some 13);
    test "0x7Fe\x00" (Some 2046);
    test "0644\x00" (Some 420);
    test "9999999999\x00" None

let _ = main ()

💾 Storable state

Okay so picture this: you pass --storable-state and re2ocaml hands you back a lexer that can save its whole entire vibe, dip back to the caller, and later pick up exactly where it left off. Iconic. 🎀

The default mode of operation in re2ocaml is a "pull" model, in which the lexer "pulls" more input whenever it needs it. That's lowkey a problem when the input shows up piece by piece — like when the lexer gets invoked by the parser, or when the lexer program is chatting over a socket protocol with some other program that has to wait for a reply from the lexer before it sends the next message 😭. The storable state feature is intended exactly for those cases: it lets you generate lexers that work in a "push" model. When the lexer needs more input, it stores its state and returns to the caller. Later, when more input becomes available, the caller resumes the lexer exactly where it stopped. 🫶

There are a few changes necessary compared to the "pull" model:

• Define YYSETSTATE() and YYGETSTATE(state) primitives. ✨
• Define yych, yyaccept (if used) and state variables as a part of persistent lexer state. The state variable should be initialized to -1. 🧠
• YYFILL should return to the outer program instead of trying to supply more input. Return code should indicate that lexer needs more input. 🛑
• The outer program should recognize situations when lexer needs more input and respond appropriately. 👀
• Optionally use getstate block to generate YYGETSTATE switch detached from the main lexer. This only works for languages that have goto (not in --loop-switch mode). 🤹
• Use re2c:eof and the sentinel with bounds checks method to handle the end of input. Padding-based method may not work because it is unclear when to append padding: the current end of input may not be the ultimate end of input, and appending padding too early may cut off a partially read greedy lexeme. Furthermore, due to high-level program logic getting more input may depend on processing the lexeme at the end of buffer (which already is blocked due to the end-of-input condition). 💀

Here's an example of a "push" model lexer that simulates reading packets from a socket. The lexer loops until it encounters the end of input and returns to the calling function. The calling function provides more input by "sending" the next packet and resumes lexing. This process stops when all the packets have been sent, or when there is an error. 🎀

(* re2ocaml $INPUT -o $OUTPUT -fi *)

open Bytes

(* Use a small buffer to cover the case when a lexeme doesn't fit.
   In real world use a larger buffer. *)
let bufsize = 10

let debug = false
let log format = (if debug then Printf.eprintf else Printf.ifprintf stderr) format

type state = {
    file: in_channel;
    yyinput: bytes;
    mutable yycursor: int;
    mutable yymarker: int;
    mutable yylimit: int;
    mutable token: int;
    mutable yystate: int;
    mutable recv: int;
}

type status = End | Ready | Waiting | BadPacket | BigPacket

let fill(st: state) : status =
    (* Error: lexeme too long. In real life could reallocate a larger buffer. *)
    if st.token < 1 then BigPacket else (

    (* Shift buffer contents (discard everything up to the current token). *)
    blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
    st.yycursor <- st.yycursor - st.token;
    st.yymarker <- st.yymarker - st.token;
    st.yylimit <- st.yylimit - st.token;
    st.token <- 0;

    (* Fill free space at the end of buffer with new data from file. *)
    let n = In_channel.input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in
    st.yylimit <- st.yylimit + n;
    set st.yyinput st.yylimit '\x00'; (* append sentinel *)

    Ready)

%{
    re2c:YYFN = ["lex;status", "yyrecord;state"];
    re2c:YYFILL = "Waiting";
    re2c:eof = 0;

    packet = [a-z]+[;];

    *      { BadPacket }
    $      { End }
    packet { yyrecord.recv <- yyrecord.recv + 1; lex_loop yyrecord }
%}

and lex_loop st =
    st.token <- st.yycursor;
    lex st

let test (packets: string list) (sts: status) =
    let fname = "pipe" in

    let oc = Out_channel.open_bin fname in
    let ic = In_channel.open_bin fname in

    let yylimit = bufsize - 1 in
    let st = {
        file = ic;
        (* Sentinel (at `yylimit` offset) is set to null, which triggers YYFILL. *)
        yyinput = create bufsize;
        yycursor = yylimit;
        yymarker = yylimit;
        yylimit = yylimit;
        token = yylimit;
        yystate = -1;
        recv = 0;
    } in

    let rec loop packets = match lex_loop st with
        | End ->
            log "done: got %d packets\n" st.recv;
            End
        | Waiting ->
            log "waiting...\n";
            let packets' = match packets with
                | [] -> []
                | p :: ps ->
                    log "sent packet '%s'\n" p;
                    Out_channel.output_string oc p;
                    Out_channel.flush oc; (* without `flush` write happens too late *)
                    ps
            in (match fill st with
                | BigPacket ->
                    log "error: packet too big\n";
                    BigPacket
                | Ready -> loop packets'
                | _ -> raise (Failure "unexpected status after fill"))
        | BadPacket ->
            log "error: ill-formed packet\n";
            BadPacket
        | _ -> raise (Failure "unexpected status")

    in if not (loop packets = sts) then
        raise (Failure "error");

    In_channel.close ic;
    Out_channel.close oc;
    Sys.remove fname

let main () =
    test [] End;
    test ["zero;"; "one;"; "two;"; "three;"; "four;"] End;
    test ["zer0;"] BadPacket;
    test ["goooooooooogle;"] BigPacket

let _ = main ()

🧩 Reusable blocks

Reusable blocks of the form /*!rules:re2c[:<name>] ... */ or %{rules[:<name>] ... %} can be reused any number of times and combined with other re2ocaml blocks. The <name> is optional. A rules block can be used in a use block or directive. The code for a rules block is generated at every point of use. ♻️

Use blocks are defined with /*!use:re2c[:<name>] ... */ or %{use[:<name>] ... %}. The <name> is optional: if it's not specified, the associated rules block is the most recent one (whether named or unnamed). A use block can add named definitions, configurations and rules of its own. An important use case for use blocks is a lexer that supports multiple input encodings: the same rules block is reused multiple times with encoding-specific configurations (see the example below). 🌈

In-block use directive !use:<name>; can be used from inside of a re2ocaml block. It merges the referenced block <name> into the current one. If some of the merged rules and configurations overlap with the previously defined ones, conflicts are resolved in the usual way: the earliest rule takes priority, and latest configuration overrides preceding ones. One exception are the special rules *, $ and (in condition mode) <!>, for which a block-local definition overrides any inherited ones. Use directive allows one to combine different re2ocaml blocks together in one block (see the example below). 🧚

Named blocks and in-block use directive were added in re2ocaml version 2.2. Since that version reusable blocks are allowed by default (no special option is needed). Before version 2.2 reuse mode was enabled with -r --reusable option. Before version 1.2 reusable blocks could not be mixed with normal blocks. ⏳

✨ Example of a !use directive ✨

(* re2ocaml $INPUT -o $OUTPUT *)
(* This example shows how to combine reusable re2c blocks: two blocks
   ('colors' and 'fish') are merged into one. The 'salmon' rule occurs
   in both blocks; the 'fish' block takes priority because it is used
   earlier. Default rule * occurs in all three blocks; the local (not
   inherited) definition takes priority. *)

open String

type answer = Color | Fish | Dunno

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable yymarker: int;
}

%{rules:colors
    *                            { raise (Failure "ah"); }
    "red" | "salmon" | "magenta" { Color }
%}

%{rules:fish
    *                            { raise (Failure "oh"); }
    "haddock" | "salmon" | "eel" { Fish }
%}

%{
    re2c:YYFN = ["lex;answer", "yyrecord;state"];
    re2c:yyfill:enable = 0;

    !use:fish;
    !use:colors;
    * { Dunno } // overrides inherited '*' rules
%}

let test(str, ans) =
    let st = {yyinput = str; yycursor = 0; yymarker = 0}
    in if not (lex st = ans) then raise (Failure "error")

let main () =
    test("salmon", Fish);
    test("what?", Dunno)

let _ = main ()

🌸 Example of a /*!use:re2c ... */ block 🌸

(* re2ocaml $INPUT -o $OUTPUT --input-encoding utf8 *)
(* This example supports multiple input encodings: UTF-8 and UTF-32.
   Both lexers are generated from the same rules block, and the use
   blocks add only encoding-specific configurations. *)

open Array

type 'a state = {
    yyinput: 'a array;
    mutable yycursor: int;
    mutable yymarker: int;
}

%{rules
    re2c:yyfill:enable = 0;

    "∀x ∃y" { Some yyrecord.yycursor }
    *       { None }
%}

%{use
    re2c:YYFN = ["lex8;int option", "yyrecord;char state"];
    re2c:encoding:utf8 = 1;
%}

%{use
    re2c:YYFN = ["lex32;int option", "yyrecord;int state"];
    re2c:encoding:utf32 = 1;
%}

Picking up right where the example left off, bestie — no cap, this is literally the next chunk:

          let main() =
              let st8 = {
                  yyinput = [|'\xe2'; '\x08'; '\x80'; '\x78'; '\x20'; '\xe2'; '\x88'; '\x83'; '\x79'|];
                  yycursor = 0;
                  yymarker = 0;
              } in if not (lex8 st8 = Some (Array.length st8.yyinput)) then raise (Failure "error");

              let st32 = {
                  yycursor = 0;
                  yymarker = 0;
                  yyinput = [|0x2200; 0x78; 0x20; 0x2203; 0x79|];
              } in if not (lex32 st32 = Some (Array.length st32.yyinput)) then raise (Failure "error");

✨ Submatch extraction ✨

Okay so re2ocaml has two options for submatch extraction, and both of them are kinda giving main-character energy.

🏷️ Tags 🏷️

The first option is to use standalone tags of the form @stag or #mtag, where stag and mtag are arbitrary user-defined names. Tags are enabled with -T, --tags option or re2c:tags = 1 configuration. Semantically tags are position markers: they can be inserted anywhere in a regular expression, and they bind to the corresponding position (or multiple positions) in the input string. S-tags bind to the last matching position, and m-tags bind to a list of positions (they may be used in repetition subexpressions, where a single position in a regular expression corresponds to multiple positions in the input string). All tags should be defined by the user, either manually or with the help of svars and mvars blocks. If there is more than one way tags can be matched against the input, ambiguity is resolved using leftmost greedy disambiguation strategy.

🎯 Captures 🎯

The second option is to use capturing groups. They are enabled with --captures option or re2c:captures = 1 configuration. There are two flavours for different disambiguation policies, --leftmost-captures (the default) is for leftmost greedy policy, and, --posix-captures is for POSIX longest-match policy. In this mode all parenthesized subexpressions are considered capturing groups, and a bang can be used to mark non-capturing groups: (! ... ). With --invert-captures option or re2c:invert-captures = 1 configuration the meaning of bang is inverted. The number of groups for the matching rule is stored in a variable yynmatch (the whole regular expression is group number zero), and submatch results are stored in yypmatch array. Both yynmatch and yypmatch should be defined by the user, and yypmatch size must be at least [yynmatch * 2]. Use maxnmatch block to define YYMAXNMATCH, a constant that equals to the maximum value of yynmatch among all rules.

🧸 Captvars 🧸

Another way to use capturing groups is the --captvars option or re2c:captvars = 1 configuration. The only difference with --captures is in the way the generated code stores submatch results: instead of yynmatch and yypmatch re2ocaml generates variables yytl<k> and yytr<k> for k-th capturing group (the user should declare these using an svars block). Captures with variables support two disambiguation policies: --leftmost-captvars or re2c:leftmost-captvars = 1 for leftmost greedy policy (the default one) and --posix-captvars or re2c:posix-captvars for POSIX longest-match policy.

Under the hood all these options translate into tags and Tagged Deterministic Finite Automata with Lookahead. The core idea of TDFA is to minimize the overhead on submatch extraction. In the extreme, if there're no tags or captures in a regular expression, TDFA is just an ordinary DFA. If the number of tags is moderate, the overhead is barely noticeable. The generated TDFA uses a number of tag variables which do not map directly to tags: a single variable may be used for different tags, and a tag may require multiple variables to hold all its possible values. Eventually ambiguity is resolved, and only one final variable per tag survives. Tag variables should be defined using stags or mtags blocks. If lexer state is stored, tag variables should be part of it. They also need to be updated by YYFILL.

S-tags support the following operations:

• save input position to an s-tag: t = YYCURSOR with C pointer API or a user-defined operation YYSTAGP(t) with generic API
• save default value to an s-tag: t = NULL with C pointer API or a user-defined operation YYSTAGN(t) with generic API
• copy one s-tag to another: t1 = t2

M-tags support the following operations:

• append input position to an m-tag: a user-defined operation YYMTAGP(t) with both default and generic API
• append default value to an m-tag: a user-defined operation YYMTAGN(t) with both default and generic API
• copy one m-tag to another: t1 = t2

S-tags can be implemented as scalar values (pointers or offsets). M-tags need a more complex representation, as they need to store a sequence of tag values. The most naive and inefficient representation of an m-tag is a list (array, vector) of tag values; a more efficient representation is to store all m-tags in a prefix-tree represented as array of nodes (v, p), where v is tag value and p is a pointer to parent node.

Here is a simple example of using s-tags to parse semantic versions consisting of three numeric components: major, minor, patch (the latter is optional). See below for a more complex example that uses YYFILL.

          (* re2ocaml $INPUT -o $OUTPUT *)

          open String

          type state = {
              yyinput: string;
              mutable yycursor: int;
              mutable yymarker: int;
              (* Final tag variables available in semantic action. *)
              %{svars format = "\n\tmutable @@{tag}: int;"; %}
              (* Intermediate tag variables used by the lexer (must be autogenerated). *)
              %{stags format = "\n\tmutable @@{tag}: int;"; %}
          }

          type semver = {
              major: int;
              minor: int;
              patch: int;
          }

          let s2n (str: string) (i1: int) (i2: int) : int =
              let rec f s i j n =
                  if i >= j then n else f s (i + 1) j (n * 10 + Char.code s.[i] - 48)
              in f str i1 i2 0

          %{local
              re2c:YYFN = ["parse;semver option", "st;state"];
              re2c:yyrecord = "st";
              re2c:tags = 1;
              re2c:yyfill:enable = 0;

              num = [0-9]+;

              @t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] {
                  Some {
                      major = s2n st.yyinput st.t1 st.t2;
                      minor = s2n st.yyinput st.t3 st.t4;
                      patch = if st.t5 = -1 then 0 else s2n st.yyinput st.t5 (st.yycursor - 1)
                  }
              }
              * { None }
          %}

          let test (str: string) (result: semver option) =
              let st = {
                  yyinput = str;
                  yycursor = 0;
                  yymarker = 0;
                  %{svars format = "\n\t\t@@{tag} = -1;"; %}
                  %{stags format = "\n\t\t@@{tag} = -1;"; %}
              }
              in if not (parse st = result) then raise (Failure "error")

          let main () =
              test "23.34" (Some {major = 23; minor = 34; patch = 0});
              test "1.2.99999" (Some {major = 1; minor = 2; patch = 99999});
              test "1.a" None

          let _ = main ()

Here is a more complex example of using s-tags with YYFILL to parse a file with newline-separated semantic versions. Tag variables are part of the lexer state, and they are adjusted in YYFILL like other input positions. Note that it is necessary for s-tags because their values are invalidated after shifting buffer contents. It may not be necessary in a custom implementation where tag variables store offsets relative to the start of the input string rather than the buffer, which may be the case with m-tags.

          (* re2ocaml $INPUT -o $OUTPUT *)

          open Bytes

          let bufsize = 4096

          type state = {
              file: in_channel;
              yyinput: bytes;
              mutable yycursor: int;
              mutable yymarker: int;
              mutable yylimit: int;
              mutable token: int;
              mutable eof: bool;
              (* Final tag variables available in semantic action. *)
              %{svars format = "\n\tmutable @@{tag}: int;"; %}
              (* Intermediate tag variables used by the lexer (must be autogenerated). *)
              %{stags format = "\n\tmutable @@{tag}: int;"; %}
          }

          type status = Ok | Eof | LongLexeme

          type semver = {
              major: int;
              minor: int;
              patch: int;
          }

          let s2n (str: bytes) (i1: int) (i2: int) : int =
              let rec f s i j n =
                  if i >= j then n else f s (i + 1) j (n * 10 + Char.code (get s i) - 48)
              in f str i1 i2 0

          let fill(st: state) : status =
              if st.eof then Eof else

              (* Error: lexeme too long. In real life could reallocate a larger buffer. *)
              if st.token < 1 then LongLexeme else (

              (* Shift buffer contents (discard everything up to the current token). *)
              blit st.yyinput st.token st.yyinput 0 (st.yylimit - st.token);
              st.yycursor <- st.yycursor - st.token;
              st.yymarker <- st.yymarker - st.token;
              st.yylimit <- st.yylimit - st.token;
              %{stags format = "\n\tst.@@ <- if st.@@ = -1 then -1 else st.@@ - st.token;"; %}
              st.token <- 0;

              (* Fill free space at the end of buffer with new data from file. *)
              let n = input st.file st.yyinput st.yylimit (bufsize - st.yylimit - 1) in (* -1 for sentinel *)
              st.yylimit <- st.yylimit + n;
              if n = 0 then
                  st.eof <- true; (* end of file *)
                  set st.yyinput st.yylimit '\x00'; (* append sentinel *)

              Ok)

          %{
              re2c:YYFN = ["lex;(semver list) option", "st;state", "vers;semver list"];
              re2c:YYFILL = "fill st = Ok";
              re2c:yyrecord = "st";
              re2c:tags = 1;
              re2c:eof = 0;

              num = [0-9]+;

              @t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\n] {
                  let ver = {
                      major = s2n st.yyinput st.t1 st.t2;
                      minor = s2n st.yyinput st.t3 st.t4;
                      patch = if st.t5 = -1 then 0 else s2n st.yyinput st.t5 (st.yycursor - 1)
                  } in lex_loop st (ver :: vers)
              }
              $ { Some (List.rev vers) }
              * { None }
          %}

          and lex_loop st vers =
              st.token <- st.yycursor;
              lex st vers

          let main () =
              let fname = "input" in

              (* Prepare input file. *)
              Out_channel.with_open_bin fname
                  (fun oc -> for i = 1 to bufsize do
                      output_string oc "1.22.333\n"
                  done);

              (* Construct the expected result to compare against. *)
              let expect = Some (List.init bufsize
                      (fun _ -> {major = 1; minor = 22; patch = 333;})) in

(* Run lexer on the prepared file. *)
In_channel.with_open_bin fname
    (fun ic ->
        let yylimit = bufsize - 1 in
        let st = {
            file = ic;
            yyinput = create bufsize;
            yycursor = yylimit;
            yymarker = yylimit;
            yylimit = yylimit;
            token = yylimit;
            eof = false;
            %{svars format = "\n\t\t@@{tag} = -1;"; %}
            %{stags format = "\n\t\t@@{tag} = -1;"; %}
        } in if (lex_loop st [] <> expect) then
            raise (Failure "error"));

(* Cleanup. *)
Sys.remove fname

let _ = main ()

Here’s an example of using capturing groups to parse semantic versions — yes bestie, tags doing their little tag thing 💅:

(* re2ocaml $INPUT -o $OUTPUT *)

open String

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable yymarker: int;
    (* Final tag variables available in semantic action. *)
    %{svars format = "\n\tmutable @@{tag}: int;"; %}
    (* Intermediate tag variables used by the lexer (must be autogenerated). *)
    %{stags format = "\n\tmutable @@{tag}: int;"; %}
}

type semver = {
    major: int;
    minor: int;
    patch: int;
}

let s2n (str: string) (i1: int) (i2: int) : int =
    let rec f s i j n =
        if i >= j then n else f s (i + 1) j (n * 10 + Char.code s.[i] - 48)
    in f str i1 i2 0

%{local
    re2c:YYFN = ["parse;semver option", "st;state"];
    re2c:yyrecord = "st";
    re2c:captvars = 1;
    re2c:yyfill:enable = 0;

    num = [0-9]+;

    (num) "." (num) ("." num)? [\x00] {
        Some {
            major = s2n st.yyinput st.yytl1 st.yytr1;
            minor = s2n st.yyinput st.yytl2 st.yytr2;
            patch = if st.yytl3 = -1 then 0 else s2n st.yyinput (st.yytl3 + 1) st.yytr3
        }
    }
    * { None }
%}

let test (str: string) (result: semver option) =
    let st = {
        yyinput = str;
        yycursor = 0;
        yymarker = 0;
        %{svars format = "\n\t\t@@{tag} = -1;"; %}
        %{stags format = "\n\t\t@@{tag} = -1;"; %}
    }
    in if not (parse st = result) then raise (Failure "error")

let main () =
    test "23.34" (Some {major = 23; minor = 34; patch = 0});
    test "1.2.99999" (Some {major = 1; minor = 2; patch = 99999});
    test "1.a" None

let _ = main ()

And here’s one with m-tags for a version that has a variable number of components. Tag variables are stored in a trie — it’s giving data-structure main character energy 🌸:

(* re2ocaml $INPUT -o $OUTPUT *)

open String

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable yymarker: int;
    (* Final tag variables available in semantic action. *)
    %{svars format = "\n\tmutable @@{tag}: int;"; %}
    %{mvars format = "\n\tmutable @@{tag}: int list;"; %}
    (* Intermediate tag variables used by the lexer (must be autogenerated). *)
    %{stags format = "\n\tmutable @@{tag}: int;"; %}
    %{mtags format = "\n\tmutable @@{tag}: int list;"; %}
}

let s2n (str: string) (i1: int) (i2: int) : int =
    let rec f s i j n =
        if i >= j then n else f s (i + 1) j (n * 10 + Char.code s.[i] - 48)
    in f str i1 i2 0

%{local
    re2c:YYFN = ["parse;(int list) option", "st;state"];
    re2c:YYMTAGP = "@@ <- st.yycursor :: @@;";
    re2c:YYMTAGN = ""; // alternatively could add `-1` to the list
    re2c:yyrecord = "st";
    re2c:tags = 1;
    re2c:yyfill:enable = 0;

    num = [0-9]+;

    @t1 num @t2 ("." #t3 num #t4)* [\x00] {
        let x = s2n st.yyinput st.t1 st.t2 in
        let xs = List.rev (List.map2 (fun x y -> s2n st.yyinput x y) st.t3 st.t4) in
        Some (x :: xs)
    }
    * { None }
%}

let test (str: string) (result: (int list) option) =
    let st = {
        yyinput = str;
        yycursor = 0;
        yymarker = 0;
        %{svars format = "\n\t\t@@{tag} = -1;"; %}
        %{mvars format = "\n\t\t@@{tag} = [];"; %}
        %{stags format = "\n\t\t@@{tag} = -1;"; %}
        %{mtags format = "\n\t\t@@{tag} = [];"; %}
    }
    in if not (parse st = result) then raise (Failure "error")

let main () =
    test "1" (Some [1]);
    test "1.2.3.4.5.6.7" (Some [1; 2; 3; 4; 5; 6; 7;]);
    test "1.2." None

let _ = main ()

Encoding support 🌈

Okay so you gotta understand the difference between code points and code units, bestie. A code point is a numeric identifier of a symbol. A code unit is the smallest unit of storage in the encoded text. A single code point may be represented with one or more code units. In a fixed-length encoding all code points are represented with the same number of code units. In a variable-length encoding code points may be represented with a different number of code units.

Note that the any rule [^] matches any code point, but not necessarily any code unit — the only way to match any code unit regardless of the encoding is the default rule *.

The generated lexer works with a stream of code units: yych stores a code unit, and YYCTYPE is the code unit type. Regular expressions, on the other hand, are specified in terms of code points. When re2ocaml compiles regular expressions to automata it translates code points to code units. This is generally not a simple mapping: in variable-length encodings a single code point range may get translated to a complex code unit graph.

The following encodings are supported:

• ASCII — enabled by default. It is a fixed-length encoding with code space [0-255] and 1-byte code points and code units.
• EBCDIC — enabled with --ebcdic or re2c:encoding:ebcdic. It is a fixed-length encoding with code space [0-255] and 1-byte code points and code units.
• UCS2 — enabled with --ucs2 or re2c:encoding:ucs2. It is a fixed-length encoding with code space [0-0xFFFF] and 2-byte code points and code units.
• UTF8 — enabled with --utf8 or re2c:encoding:utf8. It is a variable-length Unicode encoding. Code unit size is 1 byte. Code points are represented with 1 -- 4 code units.
• UTF16 — enabled with --utf16 or re2c:encoding:utf16. It is a variable-length Unicode encoding. Code unit size is 2 bytes. Code points are represented with 1 -- 2 code units.
• UTF32 — enabled with --utf32 or re2c:encoding:utf32. It is a fixed-length Unicode encoding with code space [0-0x10FFFF] and 4-byte code points and code units.

Include files include/unicode_blocks.re, include/unicode_categories.re and include/unicode_properties.re provide re2ocaml definitions for blocks, categories and properties described in the Unicode standard.

Option --input-encoding specifies source file encoding, which can be used to enable Unicode literals in regular expressions. For example --input-encoding utf8 tells re2ocaml that the source file is in UTF8 (it differs from --utf8 which sets input text encoding). Option --encoding-policy specifies the way re2ocaml handles Unicode surrogates (code points in range [0xD800-0xDFFF]).

Below is an example of a lexer for UTF8 encoded Unicode identifiers — international slay ✨:

(* re2ocaml $INPUT -o $OUTPUT --utf8 -i *)

open String

%{include "unicode_categories.re" %}

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable yymarker: int;
    mutable yyaccept: int;
}

%{
    re2c:YYFN = ["lex;bool", "yyrecord;state"];
    re2c:yyfill:enable = 0;

    // Simplified "Unicode Identifier and Pattern Syntax"
    // (see https://unicode.org/reports/tr31)
    id_start    = L | Nl | [$_];
    id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3];
    identifier  = id_start id_continue*;

    identifier { true }
    *          { false }
%}

let main () =
    let st = {
        yyinput = "_Ыдентификатор";
        yycursor = 0;
        yymarker = 0;
        yyaccept = 0;
    }
    in if not (lex st) then raise (Failure "error")

let _ = main ()

Include files 🎀

re2ocaml allows one to include other files using a block of the form /*!include:re2c FILE */ or %{include FILE %}, or an in-block directive !include FILE ;, where FILE is a path to the file to be included.

re2ocaml looks for include files in the directory of the including file and in include locations, which can be specified with the -I option.

Include blocks/directives in re2ocaml work in the same way as C/C++ #include: FILE contents are copy-pasted verbatim in place of the block/directive. Include files may have further includes of their own. Use --depfile option to track build dependencies of the output file on include files.

re2ocaml provides some predefined include files that can be found in the include/ subdirectory of the project. These files contain definitions that may be useful to other projects (such as Unicode categories) and form something like a standard library for re2ocaml.

Below is an example of using include files.

Include file 1 (definitions.ml) 💅

type number = Int | Float | NaN

%{
    number = [1-9][0-9]*;
%}

Include file 2 (extra_rules.re.inc) 🧚

// floating-point numbers
frac  = [0-9]* "." [0-9]+ | [0-9]+ ".";
exp   = 'e' [+-]? [0-9]+;
float = frac exp? | [0-9]+ exp;

float { Float }

Input file 🦋

(* re2ocaml $INPUT -o $OUTPUT -i *)

open String

%{include "definitions.ml" %}

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable yymarker: int;
    mutable yyaccept: int;
}

%{
    re2c:YYFN = ["lex;number", "yyrecord;state"];
    re2c:yyfill:enable = 0;

    *      { NaN }
    number { Int }
    !include "extra_rules.re.inc";
%}

let test(str, num) =
    let st = {yyinput = str; yycursor = 0; yymarker = 0; yyaccept = 0}
    in if not (lex st = num) then raise (Failure "error")

let main () =
    test("123", Int);
    test("123.4567", Float)

let _ = main ()

Header files 🎀

okay so bestie, re2ocaml can literally generate a header file for you straight from your input .re file — and you've got options, bestie has options 💅

• you can do it with the --header option, or
• with the re2c:header configuration, plus block pairs of the form /*!header:re2c:on*/ and /*!header:re2c:off*/, or %{header:on%} and %{header:off%}.

the first block marks the beginning of the header file, and the second block marks the end of it. everything between these blocks is processed by re2ocaml, and the generated code is written to the file you specified with --header option or re2c:header configuration (or stdout if neither option nor configuration is used — she'll just print it right at you 🫶).

and like, why would you even want an autogenerated header file? slay question, glad you asked: it may be needed in cases when re2ocaml is used to generate definitions that must be visible from other translation units. periodt ✨

here's an example of generating a header file that contains a definition of the lexer state with tag variables (the number of variables depends on the regular grammar and is unknown to the programmer — not me finding out at runtime 😭).

Input file

(* re2ocaml $INPUT -o $OUTPUT --header lexer/state.ml -i *)

open State
open String

%{header:on %}
type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable tag: int;
    %{stags format = "mutable @@: int;"; %}
}
%{header:off %}

%{
    re2c:YYFN = ["lex;int", "yyrecord;State.state"];
    re2c:tags = 1;
    re2c:yyfill:enable = 0;
    re2c:header = "lexer/state.ml";

    [a]* @tag [b]* { yyrecord.tag }
%}

let main () =
    let st = {
        yyinput = "ab";
        yycursor = 0;
        tag = 0;
        %{stags format = "\n\t@@ = 0;"; %}
    }
    in if not (lex st = 1) then raise (Failure "error")

let _ = main ()

Header file

(* Generated by re2c *)

type state = {
    yyinput: string;
    mutable yycursor: int;
    mutable tag: int;

mutable yyt1: int;
}

look at that, it literally wrote itself 💖

---

Skeleton programs 🦋

so with the -S, --skeleton option, re2ocaml ignores all non-re2ocaml code and generates a self-contained C program that can be further compiled and executed. main character energy, honestly.

the program consists of lexer code and input data. for each constructed DFA (block or condition) re2ocaml generates a standalone lexer and two files:

• an .input file with strings derived from the DFA, and
• a .keys file with expected match results.

the program runs each lexer on the corresponding .input file and compares results with the expectations. no cap, it checks itself 🧚

skeleton programs are genuinely very useful, for a number of reasons:

• they can check correctness of various re2ocaml optimizations (the data is generated early in the process, before any DFA transformations have taken place).
• generating a set of input data with good coverage may be useful for both testing and benchmarking.
• generating self-contained executable programs allows one to get minimized test cases (the original code may be large or have a lot of dependencies).

the difficulty with generating input data is that for all but the most trivial cases the number of possible input strings is too large (even if the string length is limited). re2ocaml solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. it uses the following algorithm:

first, it constructs a skeleton of the DFA. for encodings with 1-byte code unit size (such as ASCII, UTF-8 and EBCDIC) the skeleton is just an exact copy of the original DFA. for encodings with multibyte code units the skeleton is a copy of DFA with certain transitions omitted: namely, re2ocaml takes at most 256 code units for each disjoint continuous range that corresponds to a DFA transition. the chosen values are evenly distributed and include range bounds. instead of trying to cover all possible paths in the skeleton (which is infeasible) re2ocaml generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer. the algorithm implementation is limited by ~1Gb of transitions and consumes a constant amount of memory (re2ocaml writes data to file as soon as it is generated — she does not hoard, she ships 📦).

---

Visualization and debug 🔥

with the -D, --emit-dot option, re2ocaml does not generate code. instead, it dumps the generated DFA in DOT format. you can convert this dump to an image of the DFA using Graphviz or another library. aesthetic 😌

heads up bestie: this option shows the final DFA after it has gone through a number of optimizations and transformations. earlier stages can be dumped with various debug options, such as --dump-nfa, --dump-dfa-raw etc. (see the full list of options).

---

🌈 SEE ALSO 🌈

you can find more information about re2c at the official website: http://re2c.org. similar programs are flex(1), lex(1), quex (http://quex.sourceforge.net).

---

👑 AUTHORS 👑

re2ocaml was originally written by Peter Bumbulis (peter@csg.uwaterloo.ca) in 1993. Marcus Boerger and Dan Nuffer spent several years to turn the original idea into a production ready code generator. since then it has been maintained and developed by multiple volunteers, most notably:

• Brian Young (bayoung@acm.org)
• Marcus Boerger
• Dan Nuffer (nuffer@users.sourceforge.net)
• Ulya Trofimovich (skvadrik@gmail.com)
• Serghei Iakovlev
• Sergei Trofimovich
• Petr Skocik
• ligfx raekye
• and PolarGoose

legends, all of them, honestly 🫶💜

---

and that's the whole page, bestie!! that footer you saw at the bottom? that's RE2OCAML(1) — the manual page tag, doing its little signature move 🎀 so yeah, this is the RE2OCAML(1) page, fully read, fully slayed, no details skipped, no cap 💅✨ go write some lexers, you're gonna eat 🍓