interop(3)

📦 version 3.4, in fedora 43

interop(3) 💅✨

the ANTLR3C manual, bestie edition — yes it's interop(3), and no it's not about being "interoperable" in the boring corporate way, it's about how you actually talk to your generated code. 🦋

✨ Introduction ✨

okay so. the main character way you interact with your generated code is through action code — that's the stuff you drop inside { and } in your rules. 🎀

and like, real talk: keep the code you embed in those actions, and the grammar itself, to an absolute minimum. do not be out here writing a whole novel inside your grammar. instead you should build an API and call it from the actions in your grammar. that way your grammar stays clean and maintainable, and your code generators / other code stay separate from the definition of the grammar itself. it's giving separation of concerns, and we love her. 💖

but — and this is the part everyone needs — when you wanna call your API functions, or you just gotta insert a tiny lil piece of code that does not warrant a whole external function, you're gonna need to reach into:

elements of tokens 🍓
return elements from parser rules
and maybe the internals of the recognizer itself 🙈

and the C runtime has your back, because it provides a bunch of MACROs you can use inside your action code. it also ships a bunch of performant structures that are genuinely so useful for building symbol tables, lists, tries, stacks, arrays and so on — and all of them are managed, so your memory allocation problems are, like, minimized. slay. 🔥

🫶 Parameters and Returns from Parser Rules 🫶

ngl, the C target does not differ from the Java target in any major ways here. so go consult the standard documentation for how to use parameters on rules and the returns clause. 📚

but bestie, be aware: the rules generate C function calls, which means the input and returns clauses are subject to the constraints of C scoping. that's just how it is. 💀

and here's the tea 🍵 — if your parser rule returns more than a single entity, then the return type of the generated rule function is a struct, which is returned by value. same deal if your rule is part of a tree building grammar (uses the output=AST; option.

that unclosed paren is in the original, babes. we don't fix, we preserve. periodt. 💅

other than the notes above? you can use any pre-declared type as an input or output parameter for your rule. easy. ✨

💖 Memory Management 💖

listen. you are responsible for allocating and freeing any memory used by your own constructs. ANTLR will track and release any memory allocated internally for tokens, trees, stacks, scopes and so on. 🧚

that memory gets returned to the malloc pool when you call the free method of any ANTLR3 produced structure. so like, don't forget to call it. 😭

for performance reasons, and to avoid thrashing the malloc allocation system, memory for amy elements of your generated parser is allocated in chunks and parcelled out by factories. for instance, memory for tokens is created as an array of tokens, and a token factory hands out the next available slot to the lexer. when you free the lexer, the allocated memory is returned to the pool. same applies to 'strings' that contain the token text and various other text elements accessed within the lexer. 🎀

the only side effect of this is: after your parse and analysis is complete, if you wish to retain anything generated automatically, you must copy it before freeing the recognizer structures. in practice it's usually practical to retain the recognizer context objects until your processing is complete, or to use your own allocation scheme for generating output etc. whichever vibe. 🌸

the advantage of using object factories is of course that memory leaks and accessing de-allocated memory are bugs that rarely occur within the ANTLR3 C runtime. further, allocating memory for tokens, trees and so on is very fast. we love a fast queen. 👑

🌈 The CTX Macro 🌈

the CTX macro is a fundamental parameter that gets passed as the first parameter to any generated function concerned with your lexer, parser, or tree parser. 🦋

it's the context pointer for your generated recognizer, and it's how you invoke the generated functions and access the data embedded within your generated recognizer. while you can use it to directly access stacks, scopes and so on, that's not really recommended — you should use the $xxx references that are available generically within ANTLR grammars. iykyk. 💜

the context pointer is used because this removes the need for any global/static variables at all — either within the generated code, or the C runtime. this is of course fundamental to creating free threading recognizers. 🔥

wherever a function call or rule call required the ctx parameter, you either reference it via the CTX macro, or the ctx parameter is in fact the return type from calling the 'constructor' function for your parser/lexer/tree parser (see code example in "How to build Generated Code" .)

yes the spacing before that period is cursed. it's canon. 🙈

🎀 Macro Changes 🎀

while the author is not fond of using C MACROs to hide code or structure access, in the case of generated code, they serve two useful purposes:

to simplify the references to internal constructs
to facilitate the change of any internal interface without requiring you to port grammars from earlier versions (just regenerate and recompile)

as of release 3.1, these macros are stable and will only change their usage interface in the event of bugs being discovered. you are encouraged to use these macros in your code, rather than access the raw interface. 💖

\bNB: Macros that act like statements must be terminated with a ';'. The macro body does not supply this, nor should it. Macros that call functions are declared with () even if they have no parameters, macros that reference fields do not have a () declaration.

🌸 more coming, bestie — this page is long and we're only getting started 🌸

✨ Lexer Macros ✨

Okay bestie, there are a bunch of macros that are only useful inside lexer rules. There are also extra macros that every recognizer shares — those live in the Common Macros section.

🦋 LEXER

The LEXER macro hands you a pointer to the base lexer object, which is of type pANTLR3_LEXER. Important: this is not the pointer to your generated lexer — that one comes from the CTX macro. This is the common implementation of a lexer interface, and it's supplied to all generated lexers.

🦋 LEXSTATE

Gives you a pointer to the lexer shared state structure, where the tokens for a rule get constructed and the status elements of the lexer hang out. This pointer is of type #pANTLR3_RECOGNIZER_SHARED_STATE. In general you should only touch elements of this structure if there isn't already another MACRO or standard $xxxx ANTLR reference that points to it. iykyk.

🦋 LA(n)

The LA macro returns the character at index n from the current input stream index. Return type: ANTLR3_UINT32. So LA(1) gives you the character at the current input position (the one that's about to be consumed), LA(-1) gives you the character that was just consumed, and so on. LA(n) is super useful for building semantic predicates in lexer rules. But warning: LA(0) is undefined and will cause an error in your lexer. not me using LA(0) 💀

🦋 GETCHARINDEX()

The GETCHARINDEX macro returns the index of the current character position as a 0 based offset from the start of the input stream. Return type: ANTLR3_UINT32.

🦋 GETLINE()

The GETLINE macro returns the line number of the current character (LA(1)) in the input stream. Return type: ANTLR3_UINT32. Note that the line number is incremented automatically by an input stream when it sees the input character ' (yes, literally that character).

🦋 GETTEXT()

The GETTEXT macro returns the text currently matched by the lexer rule. In general you should use the generic $text reference in ANTLR to grab this. Return type is a reference type of pANTLR3_STRING, which lets you manipulate the text you've retrieved. NB: this does not change the input stream — only the text you copy from the input stream when you use this MACRO or $text.

The reference $text->chars or GETTEXT()->chars will reference a pointer to the '\0' terminated character string that the ANTLR3 pANTLR3_STRING represents. String space is allocated automatically, along with the structure that holds the string. The pANTLR3_STRING_FACTORY associated with the lexer handles this, and when you close the lexer, it will automatically free any space allocated for strings and their structures. auto-cleanup slay.

🦋 GETCHARPOSITIONINLINE()

GETCHARPOSITIONINLINE returns the zero based offset of character LA(1) from the start of the current input line. See the GETLINE macro for details on what the line number means.

🦋 EMIT()

The EMIT macro causes the text range currently matched to the lexer rule to be emitted immediately as the token for the rule. Subsequent text is matched but ignored. The type used for the token is the name of the lexer rule — or, if you've changed it with $type = XXX;, then type XXX is used.

🦋 EMITNEW(t)

The EMITNEW macro causes the supplied token reference t to be used as the token emitted by the rule. The parameter t must be of type pANTLR3_COMMON_TOKEN.

🦋 INDEX()

The INDEX macro returns the current input position according to the input stream. It is not guaranteed to be the character offset in the input stream, but is instead used as a value for marking and rewinding to specific points in the input stream. Use GETCHARINDEX() to find out the position of LA(1) in the input stream.

🦋 PUSHSTREAM(str)

The PUSHSTREAM macro, together with the POPSTREAM macro (usually called internally by the runtime), can stack many input streams onto the lexer and implement constructs like the C pre-processor #include directive.

An input stream pushed onto the stack becomes the current input stream for the lexer, and the state of the previous stream is automatically saved. The input stream will be automatically popped from the stack when it's exhausted by the lexer. You can use the POPSTREAM macro to return to the previous input stream before exhausting the currently stacked input stream.

Here's an example of using the macro in a lexer to implement the C #include pre-processor directive:

fragment
STRING_GUTS :   (~('\\'|'"') )* ;

LINE_COMMAND
: '#' (' ' | '\t')*
    (
        'include' (' ' | '\t')+ '"' file = STRING_GUTS '"' (' ' | '\t')* '\r'? '\n'
        {
            pANTLR3_STRING      fName;
            pANTLR3_INPUT_STREAM    in;

            // Create an initial string, then take a substring
            // We can do this by messing with the start and end
            // pointers of tokens and so on. This shows a reasonable way to
            // manipulate strings.
            //
            fName = $file.text;
            printf("Including file '\%s'\n", fName->chars);

            // Create a new input stream and take advantage of built in stream stacking
            // in C target runtime.
            //
            in = antlr38BitFileStreamNew(fName->chars);
            PUSHSTREAM(in);

            // Note that the input stream is not closed when it EOFs, I don't bother
            // to do it here, but it is up to you to track streams created like this
            // and destroy them when the whole parse session is complete. Remember that you
            // don't want to do this until all tokens have been manipulated all the way through
            // your tree parsers etc as the token does not store the text it just refers
            // back to the input stream and trying to get the text for it will abort if you
            // close the input stream too early.
            //

        }
            | (('0'..'9')=>('0'..'9'))+ ~('\n'|'\r')* '\r'? '\n'
        )
     {$channel=HIDDEN;}
    ;

🦋 POPSTREAM()

Assuming you've stacked an input stream using the PUSHSTREAM macro, you can remove it from the stream stack and revert to the previous input stream. Be careful to pop the stream at an appropriate point in your lexer action, so you don't match characters from one stream with those from another in the same rule — unless that's what you want to do. no accidental cross-stream chaos, bestie.

🦋 SETTEXT(str)

A token manufactured by the lexer doesn't actually physically store the text from the input stream it matches. The token string is instead created only if you ask for the text. But if you want to change the text that the token represents, you can use this macro to set it explicitly. Note that this does not change the input stream text — it associates the supplied pANTLR3_STRING with the token. This string is then returned when parser and tree parser reference the tokens via the $xxx.text reference.

🦋 USER1 USER2 USER3 and CUSTOM

While you can create your own custom token class and have the lexer deal with it, that's a lot of work compared to the trivial inheritance you can achieve in the Java target. In many cases, though, all you need is to add a few data items like an integer or a pointer. Rather than force C programmers to create complicated structures just to add a few data items, the C target provides a few custom fields in the standard token, which will fulfil the needs of most lexers and parsers. practical queen behavior.

The token fields user1, user2, and user3 are all value types of #ANTLR_UINT32. In the parser you can reference these fields directly from the token: x=TOKNAME { $x->user1 ... but when you're building the token in the lexer, you must assign to the fields using the macros USER1, USER2, or USER3. As in:

LEXTOK: 'AAAAA' { USER1 = 99; } ;

✨ Parser and Tree Parser Macros ✨

🦋 PARSER

The PARSER macro returns a pointer to the base parser or tree parser object, which is of type pANTLR3_PARSER or pANTLR3_TREE_PARSER. This is not the pointer to your generated parser — that's supplied by the CTX macro. This is the common implementation of a parser or tree parser interface, supplied to all generated parsers.

🦋 INDEX()

When used in the parser, the INDEX macro returns the position of the current token (LT(1)) in the input token stream. It can be used for MARK and REWIND operations.

🦋 LT(n) and LA(n)

In the parser, the macro LT(n) returns the pANTLR3_COMMON_TOKEN at offset n from the current token stream input position. The macro LA(n) returns the token type of the token at position n. The value n cannot be zero, and such a reference will return NULL and possibly cause an error. LA(1) is the token that is about to be recognized, and LA(-1) is the token that has just been recognized. Values of n that exceed the limits of the token stream boundaries will return NULL.

🦋 PSRSTATE

Returns the shared state pointer of type pANTLR3_RECOGNIZER_SHARED_STATE. This is not generally useful to the grammar programmer, because the useful elements have generic $xxx references built into ANTLR.

🦋 ADAPTOR

When building an AST via a parser, the work of constructing and manipulating trees is done by a supplied adaptor class. The default class is usually fine for most tree operations, but if you want to build your own specialized linked/tree structure, you may need to reference the adaptor you supply directly. The ADAPTOR macro returns the reference to the tree adaptor, which is always of type pANTLR3_BASE_TREE_ADAPTOR, even if it's your custom adapter. custom but still that type, periodt.

🫶 Macros Common to All Recognizers 🫶

okay bestie, these are the ✨ universal ✨ ones — they work no matter what kind of recognizer you're holding, so no matter the vibe, they've got you 💅

`RECOGNIZER` 👑

Returns a reference type of pANTRL3_BASE_RECOGNIZER, which is literally the base functionality handed to every single recognizer — lexers, parsers, tree parsers, the whole friend group 🌈

And yes, you can override methods in this interface by installing your own function pointers 🛠️ — but only once you know what you are doing, okay? that's the original's words and honestly? respect the warning 💀

`INPUT` 🎀

Returns a reference to the input stream of the appropriate type for whatever recognizer you're in, and the type changes depending on who's asking:

In a lexer, this macro returns a reference type of pANTLR3_INPUT_STREAM
In a parser, it's type pANTLR3_TOKEN_STREAM
In a tree parser, it's type pANTLR3_COMMON_TREE_NODE_STREAM

And of course you can provide your own implementations of any of these interfaces — main character energy, your rules 💅

`MARK()` 📍

This macro makes the input stream for the current recognizer get marked with a checkpoint ✨ It returns a value type of ANTLR3_MARKER, which you can then use as the parameter to a REWIND macro to teleport back to that marked point in the input 🦋

And if you already know you're only ever gonna rewind to the last MARK? then bestie you can straight up ignore the return value of this macro and just use the REWINDLAST macro to go back to the last MARK that was set in the input stream 🎯 no cap.

`REWIND(m)` ⏪

Rewinds the appropriate input stream back to the marked checkpoint that was returned from a prior MARK macro call, supplied as the parameter m to the REWIND(m) macro 🌸

`REWINDLAST()` 🔙

Rewinds the current input stream (characters, tokens, tree nodes — all of it) back to the last checkpoint marker created by a MARK macro call 💖

⚠️ and here's the thing: it fails silently if there was no prior MARK call. like, no error, no drama, it just… doesn't. so keep track of your marks, babe 😭

`SEEK(n)` 🧭

Causes the input stream to position itself directly at offset n in the stream ✨ and it works for all input stream types — lexer, parser and tree parser alike 👑

and that's the whole page, bestie 🫶 you made it to the end of interop(3) — this was Version 3.3.1, dated Tue Jul 29 2025, and honestly? iconic. go forth and interoperate 💖✨🎀🦋

📦 where this page comes from

fedora 43: antlr3-C-devel 1:3.4-3.5.3.14.fc43 · BSD-3-Clause
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-14: a language model's rewrite of the original, under its terms

interop(3)

interop(3) 💅✨

✨ Introduction ✨

🫶 Parameters and Returns from Parser Rules 🫶

💖 Memory Management 💖

🌈 The CTX Macro 🌈

🎀 Macro Changes 🎀

✨ Lexer Macros ✨

🦋 LEXER

🦋 LEXSTATE

🦋 LA(n)

🦋 GETCHARINDEX()

🦋 GETLINE()

🦋 GETTEXT()

🦋 GETCHARPOSITIONINLINE()

🦋 EMIT()

🦋 EMITNEW(t)

🦋 INDEX()

🦋 PUSHSTREAM(str)

🦋 POPSTREAM()

🦋 SETTEXT(str)

🦋 USER1 USER2 USER3 and CUSTOM

✨ Parser and Tree Parser Macros ✨

🦋 PARSER

🦋 INDEX()

🦋 LT(n) and LA(n)

🦋 PSRSTATE

🦋 ADAPTOR

🫶 Macros Common to All Recognizers 🫶

RECOGNIZER 👑

INPUT 🎀

MARK() 📍

REWIND(m) ⏪

REWINDLAST() 🔙

SEEK(n) 🧭

`RECOGNIZER` 👑

`INPUT` 🎀

`MARK()` 📍

`REWIND(m)` ⏪

`REWINDLAST()` 🔙

`SEEK(n)` 🧭