leex(3)

📦

version 29.0.6 · 2 versions

29.0.6 · nixos 26.05, nixpkgs unstable, ubuntu stonking (latest)
28.5.0.8 · alpine 3.24

, in nixos 26.05, nixpkgs unstable, ubuntu stonking

✨ NAME ✨

leex — Lexical analyzer generator for Erlang. That's it, that's the name entry. And yeah, it's leex(3), filed under Erlang Module Definition, and it's genuinely iconic. 💅

💖 DESCRIPTION 💖

So basically, leex is a regular expression based lexical analyzer generator for Erlang, and it's the same vibe as lex or flex — like, similar to them, no cap. 🦋

🌸 Default Leex Options 🌸

Okay bestie, listen up ☕ — the (host operating system) environment variable ERL_COMPILER_OPTIONS can be used to give default Leex options. Its value must be a valid Erlang term. If the value is a list, it is used as is. If it is not a list, it is put into a list.

The list is appended to any options given to file/2.

And you can pull that list back out with compile:env_compiler_options/0. Slay.

🎀 Input File Format 🎀

Erlang style comments starting with a % are allowed in scanner files, which is lowkey great. A definition file has the following format:

<Header>

Definitions.

<Macro Definitions>

Rules.

<Token Rules>

Erlang code.

<Erlang code>

Now the important part: the Definitions., Rules., and Erlang code headings are mandatory and must start at the beginning of a source line. The <Header>, <Macro Definitions>, and <Erlang code> sections are allowed to be empty, but there must be at least one rule. No rules? Not the vibe, bestie. 💀

Macro definitions have the following format:

NAME = VALUE

and there must be spaces around =. Do not skip the spaces. 😭 Macros can be used in the regular expressions of rules by writing {NAME}.

NOTE: When macros are expanded in expressions, the macro calls are replaced by the macro value without any form of quoting or enclosing in parentheses. Periodt.

Rules have the following format:

<Regexp> : <Erlang code>.

The <Regexp> must occur at the start of a line and not include any blanks; use \t and \s to include TAB and SPACE characters in the regular expression. If <Regexp> matches then the corresponding <Erlang code> is evaluated to generate a token. And with that Erlang code, these predefined variables are right there for you:

TokenChars — A list of the characters in the matched token.
TokenLen — The number of characters in the matched token.
TokenLine — The line number where the token occurred.
TokenCol — The column number where the token occurred (column of the first character included in the token).
TokenLoc — Token location. Expands to {TokenLine,TokenCol} (even when error_location is set to line).

The code must return one of these:

{token,Token} — Return Token to the caller.
{end_token,Token} — Return Token and is last token in a tokens call.
skip_token — Skip this token completely.
{error,ErrString} — An error in the token, ErrString is a string describing the error.

It is also possible to push back characters into the input characters with the following returns:

{token,Token,PushBackList}
{end_token,Token,PushBackList}
{skip_token,PushBackList}

These have the same meanings as the normal returns but the characters in PushBackList will be prepended to the input characters and scanned for the next token. Just know that pushing back a newline will mean the line numbering will no longer be correct. 😬

NOTE: Pushing back characters gives you unexpected possibilities to cause the scanner to loop! Not the cute kind of loop. 💀

The following example would match a simple Erlang integer or float and return a token which could be sent to the Erlang parser:

D = [0-9]

{D}+ :
  {token,{integer,TokenLine,list_to_integer(TokenChars)}}.

{D}+\.{D}+((E|e)(\+|\-)?{D}+)? :
  {token,{float,TokenLine,list_to_float(TokenChars)}}.

And the Erlang code in the Erlang code. section is written into the output file directly after the module declaration and predefined exports declaration, making it possible to add extra exports, define imports, and other attributes, which are visible in the whole file. Chef's kiss. 👑

🦋 Regular Expressions 🦋

The regular expressions allowed here is a subset of the set found in egrep and in the AWK programming language, as defined in the book The AWK Programming Language by A. V. Aho, B. W. Kernighan, and P. J. Weinberger. They are composed of the following characters:

c — Matches the non-metacharacter c.
\c — Matches the escape sequence or literal character c.
. — Matches any character.
^ — Matches the beginning of a string.
$ — Matches the end of a string.
[abc...] — Character class, which matches any of the characters abc.... Character ranges are specified by a pair of characters separated by a -.
[^abc...] — Negated character class, which matches any character except abc....
r1 | r2 — Alternation. It matches either r1 or r2.
r1r2 — Concatenation. It matches r1 and then r2.
r+ — Matches one or more rs.
r* — Matches zero or more rs.
r? — Matches zero or one rs.
(r) — Grouping. It matches r.

The escape sequences allowed are the same as for Erlang strings:

\b — Backspace.
\f — Form feed.
\n — Newline (line feed).
\r — Carriage return.
\t — Tab.
\e — Escape.
\v — Vertical tab.
\s — Space.
\d — Delete.
\ddd — The octal value ddd.
\xhh — The hexadecimal value hh.
\x{h...} — The hexadecimal value h....
\c — Any other character literally, for example \\ for backslash, \" for ".

The following examples define simplified versions of a few Erlang data types:

Atoms a-z*

Variables A-Z_*

Floats (\+|-)?[0-9]+\.[0-9]+((E|e)(\+|-)?[0-9]+)?

NOTE: Anchoring a regular expression with ^ and $ is not implemented in the current version of leex and generates a parse error. So like… don't do it. 🫠

💜 DATA TYPES 💜

error_info() ::
          {erl_anno:line() | none, module(), ErrorDescriptor :: term()}.

The standard error_info/0 structure that is returned from all I/O modules. ErrorDescriptor is formattable by format_error/1.

error_ret() ::
          error | {error, Errors :: errors(), Warnings :: warnings()}.

errors() :: [{file:filename(), [error_info()]}].

leex_ret() :: ok_ret() | error_ret().

ok_ret() ::
          {ok, Scannerfile :: file:filename()} |
          {ok, Scannerfile :: file:filename(), warnings()}.

warnings() :: [{file:filename(), [error_info()]}].

💖 FUNCTIONS 💖

okay bestie, this is the part where the actual doing stuff happens. buckle up 🎀

🌸 `file(FileName, Options) -> leex_ret()`

here's the whole signature, character for character, no cap:

file(FileName, Options) -> leex_ret()
              when
                  FileName :: file:filename(),
                  Options :: Option | [Option],
                  Option ::
                      {dfa_graph, boolean()} |
                      {includefile, Includefile :: file:filename()} |
                      {report_errors, boolean()} |
                      {report_warnings, boolean()} |
                      {report, boolean()} |
                      {return_errors, boolean()} |
                      {return_warnings, boolean()} |
                      {return, boolean()} |
                      {scannerfile, Scannerfile :: file:filename()} |
                      {verbose, boolean()} |
                      {warnings_as_errors, boolean()} |
                      {deterministic, boolean()} |
                      {error_location, line | column} |
                      {tab_size, pos_integer()} |
                      dfa_graph | report_errors | report_warnings |
                      report | return_errors | return_warnings |
                      return | verbose | warnings_as_errors.

This bad boy generates a lexical analyzer from whatever definition is chillin' in your input file. iconic behavior fr fr ✨

The input file is supposed to have the extension .xrl. If you don't give it one, leex will just add it for you, no biggie. And the resulting module? That's your Xrl filename minus the .xrl extension. slay.

The current options are — and yes bestie, we're listing all of them 🫶:

dfa_graph — spits out a .dot file describing the DFA in a format you can peek at with Graphviz, www.graphviz.com. lowkey fun 💅
{includefile,Includefile} — lets you use a specific or customised prologue file instead of the default lib/parsetools/include/leexinc.hrl (which is what normally gets included otherwise).
{report_errors, boolean()} — makes errors print as they happen. Default is true.
{report_warnings, boolean()} — makes warnings print as they happen. Default is true.
{report, boolean()} — little short form that covers both report_errors and report_warnings at once. two birds, one flag 🦋
{return_errors, boolean()} — flip this on and you get {error, Errors, Warnings} returned whenever there are errors. Default is false.
{return_warnings, boolean()} — flip this on and an extra field holding Warnings gets added to the tuple you get back on success. Default is false.
{return, boolean()} — another short form, covering both return_errors and return_warnings.
{scannerfile, Scannerfile} — Scannerfile is the name of the file that's gonna hold the Erlang scanner code that gets generated. The default ("") is to tack the extension .erl onto FileName once you've stripped the .xrl extension off it. obsessed with the automaticity tbh
{verbose, boolean()} — outputs info from parsing the input file and generating the internal tables. it's giving narrator voice 📢
{warnings_as_errors, boolean()} — causes warnings to be treated as errors. dramatic but valid 💀
{deterministic, boolean()} — causes the generated -file() attributes to only include the basename of the file path. reproducible queen 👑
{error_location, line | column} — if you set this to column, error location becomes a {Line,Column} tuple instead of just Line. same deal for StartLoc and EndLoc in the string/2, token/3 and tokens/3 functions — they become {Line,Column} tuples too instead of just Line. Default is line. Big note: you can use TokenLoc for token location independently, even when error_location is set to line. ✨ Unicode characters are counted as however many columns as bytes it takes to represent them — so yeah, that's a thing you should know.
{tab_size, pos_integer()} — sets the width of the \t character. This only matters if error_location is set to column. Default is 8.

💡 Any of the Boolean options can be set to true by just stating the name of the option. For example, verbose is equivalent to {verbose, true}. effortless.

Also, heads up: Leex will add the extension .hrl to the Includefile name, and the extension .erl to the Scannerfile name — unless the extension is already there. it checks first, it's polite 🌸

🧚 `format_error(ErrorDescriptor) -> io_lib:chars()`

format_error(ErrorDescriptor) -> io_lib:chars()
                       when ErrorDescriptor :: term().

Returns a descriptive string in English of an error reason ErrorDescriptor, which you'd have gotten back from leex:file/1,2 when there's an error in a regular expression. it's the "explain urself" function 💬

🦋 `string(String, StartLoc) -> StringRet`

the whole shape of it:

string(String, StartLoc) -> StringRet
                 when
                     String :: string(),
                     StringRet :: {ok, Tokens, EndLoc} | ErrorInfo,
                     Tokens :: [Token],
                     Token :: term(),
                     ErrorInfo ::
                         {error, error_info(), erl_anno:location()},
                     StartLoc :: erl_anno:location(),
                     EndLoc :: erl_anno:location().

Scans String and hands you back either all the tokens in it or an error tuple. pick your fighter ✨

StartLoc and EndLoc are either erl_anno:line() or erl_anno:location(), depending on the error_location option.

⚠️ NOTE: It is an error if not all of the characters in String are consumed. literally every single one bestie, no leftovers allowed 😭

🌈 `token(Cont, Chars, StartLoc)`

token(Cont, Chars, StartLoc) ->
               {more, Cont1} | {done, TokenRet, RestChars}
               when
                   Cont :: [] | Cont1,
                   Cont1 :: tuple(),
                   Chars :: string() | eof,
                   RestChars :: string() | eof,
                   TokenRet ::
                       {ok, Token, EndLoc} | {eof, EndLoc} | ErrorInfo,
                   ErrorInfo ::
                       {error, error_info(), erl_anno:location()},
                   Token :: term(),
                   StartLoc :: erl_anno:location(),
                   EndLoc :: erl_anno:location().

This is a re-entrant call to try and scan a single token out of Chars. it's giving "we'll finish this later" energy 🙈

If there are enough characters in Chars to either scan a token or detect an error, then that gets returned with {done,...}. Otherwise you get {more,Cont} back, where Cont gets used in the next call to token() along with more characters to try an scan the token. This keeps going until a token has been scanned. Cont starts out as []. 🎀

It is not designed to be called directly by an application — it's meant to be used through the I/O system, where in an application it can typically be called like this:

io:request(InFile, {get_until,unicode,Prompt,Module,token,[Loc]})
  -> TokenRet

💜 `tokens(Cont, Chars, StartLoc)`

tokens(Cont, Chars, StartLoc) ->
                {more, Cont1} | {done, TokensRet, RestChars}
                when
                    Cont :: [] | Cont1,
                    Cont1 :: tuple(),
                    Chars :: string() | eof,
                    RestChars :: string() | eof,
                    TokensRet ::
                        {ok, Tokens, EndLoc} | {eof, EndLoc} |
ErrorInfo,
                    Tokens :: [Token],
                    Token :: term(),
                    ErrorInfo ::
                        {error, error_info(), erl_anno:location()},
                    StartLoc :: erl_anno:location(),
                    EndLoc :: erl_anno:location().

Another re-entrant call, but this time we're trying to scan tokens (plural, the more the merrier) from Chars. 🔥

If there are enough characters in Chars to either scan tokens or detect an error, then that comes back with {done,...}. Otherwise {more,Cont} is returned, and Cont gets used in the next call to tokens() with more characters to try an scan the tokens. This continues until all tokens have been scanned. Cont is initially [].

The way this differs from token is that it keeps scanning tokens up to and including an {end_token,Token} being scanned (see next section 👀). Then it returns all the tokens. This is typically used for scanning grammars like Erlang, where there's an explicit end token, '.'. If no end token is found, then the whole file gets scanned and returned. And if an error occurs? Then all tokens up to and including the next end token get skipped. brutal but fair 🍓

It is not designed to be called directly by an application, but used through the I/O system, where it can typically be called in an application like this:

io:request(InFile, {get_until,unicode,Prompt,Module,tokens,[Loc]})
  -> TokensRet

and that's the whole page bestie, you ate it 💅✨ this icon was brought to you by Ericsson AB, parsetools version 2.8, and the leex(3) manpage — go scan something gorgeous, ily 🫶🎀🦋

📦 where this page comes from

nixos 26.05: erlang 29.1.2
nixpkgs unstable: erlang 29.1.2
ubuntu stonking: erlang-doc 1:29.0.6+dfsg-1 · Apache-2.0
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms