erl_scan(3erl)

📦
version 25.3.2.8 · 5 versions
, in ubuntu noble

💅 erl_scan — the Erlang token scanner, but make it make sense ✨

okay so bestie, this is the erl_scan module from the Erlang standard library docs (stdlib), and it's literally the module that turns your raw characters into Erlang tokens. no cap, it's the main character of the whole lexing situation. let's get into it 🫶

🌸 NAME 🌸

erl_scan — The Erlang token scanner. that's it, that's the name, iconic.

🦋 SYNOPSIS 🦋

this page is about the module, so no synopsis line here — just know that everything below is exported by erl_scan and you call it from your Erlang code like a normal person 💖

✨ DESCRIPTION ✨

This module contains functions for tokenizing (scanning) characters into Erlang tokens. so yeah — you feed it characters, it hands you back tokens. literal glow-up for your strings 👑

🎀 DATA TYPES 🎀

girl, this is the type-alias section, and it's giving very "here's the vocabulary before the vibes":

category() = atom()

error_description() = term()

error_info() =
    {erl_anno:location(), module(), error_description()}

option() =
    return | return_white_spaces | return_comments | text |
    {reserved_word_fun, resword_fun()} |
    {text_fun, text_fun()}

options() = option() | [option()]

symbol() = atom() | float() | integer() | string()

resword_fun() = fun((atom()) -> boolean())

token() =
    {category(), Anno :: erl_anno:anno(), symbol()} |
    {category(), Anno :: erl_anno:anno()}

tokens() = [token()]

tokens_result() =
    {ok, Tokens :: tokens(), EndLocation :: erl_anno:location()} |
    {eof, EndLocation :: erl_anno:location()} |
    {error,
     ErrorInfo :: error_info(),
     EndLocation :: erl_anno:location()}

text_fun() = fun((atom(), string()) -> boolean())

copy these exactly, bestie, they're load-bearing 💅

🔥 EXPORTS 🔥

category(Token) -> category()

Types:

Token = token()

Returns the category of Token. one job, does it slay.

column(Token) -> erl_anno:column() | undefined

Types:

Token = token()

Returns the column of Token's collection of annotations. cute little column check 💖

end_location(Token) -> erl_anno:location() | undefined

Types:

Token = token()

Returns the end location of the text of Token's collection of annotations. If there is no text, undefined is returned. no text? no location. periodt.

format_error(ErrorDescriptor) -> string()

Types:

ErrorDescriptor = error_description()

Uses an ErrorDescriptor and returns a string that describes the error or warning. This function is usually called implicitly when an ErrorInfo structure is processed (see section Error Information). so like, you don't usually call it yourself, it just happens 🧚

line(Token) -> erl_anno:line()

Types:

Token = token()

Returns the line of Token's collection of annotations. line go brrr 📍

location(Token) -> erl_anno:location()

Types:

Token = token()

Returns the location of Token's collection of annotations. the where-is-it of it all 🌈

reserved_word(Atom :: atom()) -> boolean()

Returns true if Atom is an Erlang reserved word, otherwise false. simple, clean, no notes 💅

string(String) -> Return

string(String, StartLocation) -> Return

string(String, StartLocation, Options) -> Return

Types:

String = string()
Options = options()
Return =
    {ok, Tokens :: tokens(), EndLocation} |
    {error, ErrorInfo :: error_info(), ErrorLocation}
StartLocation = EndLocation = ErrorLocation =
erl_anno:location()

Takes the list of characters String and tries to scan (tokenize) them. Returns one of the following:

  • {ok, Tokens, EndLocation}: Tokens are the Erlang tokens from String. EndLocation is the first location after the last token. slay, you got tokens ✨
  • {error, ErrorInfo, ErrorLocation}: An error occurred. ErrorLocation is the first location after the erroneous token. 😭 rude but informative.

string(String) is equivalent to string(String, 1), and string(String, StartLocation) is equivalent to string(String, StartLocation, []). free shortcuts, we love to see it.

StartLocation indicates the initial location when scanning starts. If StartLocation is a line, Anno, EndLocation, and ErrorLocation are lines. If StartLocation is a pair of a line and a column, Anno takes the form of an opaque compound data type, and EndLocation and ErrorLocation are pairs of a line and a column. The token annotations contain information about the column and the line where the token begins, as well as the text of the token (if option text is specified), all of which can be accessed by calling column/1, line/1, location/1, and text/1. okay so annotations are the receipts of where everything came from, got it 🧾

A token is a tuple containing information about syntactic category, the token annotations, and the terminal symbol. For punctuation characters (such as ; and |) and reserved words, the category and the symbol coincide, and the token is represented by a two-tuple. Three-tuples have one of the following forms:

  • {atom, Anno, atom()}
  • {char, Anno, char()}
  • {comment, Anno, string()}
  • {float, Anno, float()}
  • {integer, Anno, integer()}
  • {var, Anno, atom()}
  • {white_space, Anno, string()}

Valid options — okay bestie, this is the options menu, read it all:

  • {reserved_word_fun, reserved_word_fun()}: A callback function that is called when the scanner has found an unquoted atom. If the function returns true, the unquoted atom itself becomes the category of the token. If the function returns false, atom becomes the category of the unquoted atom. you decide the fate of the atom 👑
  • return_comments: Return comment tokens. give me my comments back 🙏
  • return_white_spaces: Return white space tokens. By convention, a newline character, if present, is always the first character of the text (there cannot be more than one newline in a white space token). whitespace girlies rise 🌸
  • return: Short for [return_comments, return_white_spaces]. the combo meal deal.
  • text: Include the token text in the token annotation. The text is the part of the input corresponding to the token. See also text_fun. keep the receipts 📝
  • {text_fun, text_fun()}: A callback function used to determine whether the full text for the token shall be included in the token annotation. Arguments of the function are the category of the token and the full token string. This is only used when text is not present. If neither are present the text will not be saved in the token annotation. so: text wins, then text_fun, then nothing. iykyk.

symbol(Token) -> symbol()

Types:

Token = token()

Returns the symbol of Token. the actual meat of the token 💖

text(Token) -> erl_anno:text() | undefined

Types:

Token = token()

Returns the text of Token's collection of annotations. If there is no text, undefined is returned. same energy as end_location, no text no problem 🌈

tokens(Continuation, CharSpec, StartLocation) -> Return

tokens(Continuation, CharSpec, StartLocation, Options) -> Return

Types:

Continuation = return_cont() | []
CharSpec = char_spec()
StartLocation = erl_anno:location()
Options = options()
Return =
    {done,
     Result :: tokens_result(),
     LeftOverChars :: char_spec()} |
    {more, Continuation1 :: return_cont()}
char_spec() = string() | eof
return_cont()
  An opaque continuation.

This is the re-entrant scanner, which scans characters until either a dot ('.' followed by a white space) or eof is reached. It returns:

  • {done, Result, LeftOverChars}: Indicates that there is sufficient input data to get a result. Result is:
    • {ok, Tokens, EndLocation}: The scanning was successful. Tokens is the list of tokens including dot. 🎉
    • {eof, EndLocation}: End of file was encountered before any more tokens. that's all folks.
    • {error, ErrorInfo, EndLocation}: An error occurred. LeftOverChars is the remaining characters of the input data, starting from EndLocation. leftovers, but make it lexing 🍓
  • {more, Continuation1}: More data is required for building a term. Continuation1 must be passed in a new call to tokens/3,4 when more data is available. come back when you have more, bestie.

The CharSpec eof signals end of file. LeftOverChars then takes the value eof as well.

tokens(Continuation, CharSpec, StartLocation) is equivalent to tokens(Continuation, CharSpec, StartLocation, []). yes girl, the shortcut again.

For a description of the options, see string/3. go back up, it's all there 💅

🧚 ERROR INFORMATION 🧚

ErrorInfo is the standard ErrorInfo structure that is returned from all I/O modules. The format is as follows:

{ErrorLocation, Module, ErrorDescriptor}

A string describing the error is obtained with the following call:

Module:format_error(ErrorDescriptor)

that's the whole error-handling vibe, no notes 😭

⭐ NOTES ⭐

The continuation of the first call to the re-entrant input functions must be []. like, literally must be. For a complete description of how the re-entrant input scheme works, see Armstrong, Virding and Williams: 'Concurrent Programming in Erlang', Chapter 13. homework assignment, but make it iconic 📚

bestie, real talk: that [] on the first call isn't optional, it's the entry fee 💀

🦋 SEE ALSO 🦋

erl_anno(3erl), erl_parse(3erl), io(3erl)

go read those too, they're the supporting cast and they ate 💜

and that's the whole page, bestie — written by Ericsson AB, stdlib version 4.3.1.3, dated alongside this erl_scan(3erl) manual page. mwah 💋✨🎀

📦 where this page comes from