ANTLR3_LEXER_struct(3)

📦 version 3.4, in fedora 43

💖 ANTLR3_LEXER_struct — the lexer struct, bestie edition 💖

okay so like. this is a page from the ANTLR3C manual (that's the C runtime for ANTLR 3, iykyk), and today we're doing a full vibe check on ANTLR3_LEXER_struct. it's a struct. it's a big one. grab a strawberry 🍓 and let's go.

✨ NAME ✨

ANTLR3_LEXER_struct — the struct that IS the lexer interface. she's the main character fr fr 👑

🌸 SYNOPSIS 🌸

#include <antlr3lexer.h>

🎀 Data Fields 🎀

  • void * ctx — okay so a generated lexer has this mTokens() function, right? and that function needs the context pointer of the generated lexer, NOT the base lexer interface. so we stash it here 💅. it gets initialized by the generated code, or manually if you built the lexer by hand. no judgment, we love a DIY moment.
  • pANTLR3_COMMON_TOKEN(* emit )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that constructs a brand new token out of the info the lexer already has stored. new token, who dis ✨
  • void(* emitNew )(struct ANTLR3_LEXER_struct *lexer, pANTLR3_COMMON_TOKEN token) — pointer to a function that emits the token you hand it as the next token in the stream. here bestie, take this token, put it in the stream 🫶
  • void(* free )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that knows how to free a lexer's resources. she's the cleanup queen, we stan 🧹
  • ANTLR3_MARKER(* getCharIndex )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that returns the current char index. yes queen, tell us where we are 📍
  • ANTLR3_UINT32(* getCharPositionInLine )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that returns the current character position in the line. how far along are we? she'll tell you 💖
  • ANTLR3_UINT32(* getLine )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that returns the current line number in the input stream. what line are we on bestie 📖
  • pANTLR3_STRING(* getText )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that returns the text so far for the token currently being generated. the token's whole origin story, basically 😭
  • pANTLR3_INPUT_STREAM input — a pointer to the character stream where this lexer is getting her characters from. the source, the origin, the moment 🌊
  • void(* matchAny )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that matches the next token/char in the input stream no matter what it actually is. zero standards, we love that for her 💀 (note: the original spells it "actaully" and honestly? iconic typo, we respect it)
  • ANTLR3_BOOLEAN(* matchc )(struct ANTLR3_LEXER_struct *lexer, ANTLR3_UCHAR c) — pointer to a function that matches and consumes the specified character from the input stream. just the one char. specific. focused. slay.
  • ANTLR3_BOOLEAN(* matchRange )(struct ANTLR3_LEXER_struct *lexer, ANTLR3_UCHAR low, ANTLR3_UCHAR high) — pointer to a function that matches any character in the supplied range. and like, technically it could be a token range too? but that would only be useful if the tokens were in some guaranteed order, which only happens with a hand crafted token set. so. you know. range energy 🎯
  • ANTLR3_BOOLEAN(* matchs )(struct ANTLR3_LEXER_struct *lexer, ANTLR3_UCHAR *string) — pointer to a function that tries to match and consume the specified string from the input stream. strings have to be passed as terminated arrays of ANTLR3_UCHAR, and they're terminated with 0xFFFFFFFF, which is an invalid UTF32 character. so don't get cute with your terminators, bestie 🚫
  • void(* mTokens )(void *ctx) — pointer to the user provided function (either written by hand or generated by code generation) that makes the lexer rules run and produce the next token if there is one. this gets called from nextToken() in the pANTLR3_TOKEN_SOURCE. and heads up: the input parameter for this function is the generated lexer context (that's the thing stored in ctx in this interface). it's a generated function, so it expects the context to be the generated lexer. don't hand her the wrong context, she WILL know 🙈
  • void(* popCharStream )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that abandons the current input stream — whether it's empty or not, no questions asked — and reverts to the previous stacked input stream. bye felicia, we're going back to the last one 👋
  • void(* pushCharStream )(struct ANTLR3_LEXER_struct *lexer, pANTLR3_INPUT_STREAM input) — pointer to a function that switches the current character input stream to a new one, saving the old one so we can revert to it at the end of this new one. stacking streams like a little hoarder, and honestly? valid 🧺
  • pANTLR3_BASE_RECOGNIZER rec — pointer to the implementation of a base recognizer, which the lexer creates and then overrides with her own lexer oriented functions (the default implementation is parser oriented, so she's basically giving it a glow up). this also contains a token source interface, which the lexer instance hands to anything that needs it — which is anything else that implements a base recognizer, like a parser 💫
  • void(* recover )(struct ANTLR3_LEXER_struct *lexer) — pointer to a function that recovers from an error found in the input stream. usually it'll be a #ANTLR3_EXCEPTION_NOVIABLE_ALT, but it could also come from a mismatched token that the (*match)() couldn't recover from. she's resilient, she gets back up 💪
  • void(* setCharStream )(struct ANTLR3_LEXER_struct *lexer, pANTLR3_INPUT_STREAM input) — pointer to a function that sets the charstream source for the lexer and causes it to be reset. new stream, fresh start, clean slate ✨
  • void * super — if there's a super structure implementing the lexer, you can store a pointer to it here, in case implementing functions get overridden by this super structure. yes she has a super. she's someone's super too. growth 🌱

🔥 Field Documentation 🔥

💜 void* ANTLR3_LEXER_struct::ctx

A generated lexer has an mTokens() function, which needs the context pointer of the generated lexer, not the base lexer interface. this is stored here and initialized by the generated code (or manually if this is a manually built lexer.

Referenced by nextTokenStr().

💜 pANTLR3_COMMON_TOKEN(* ANTLR3_LEXER_struct::emit) (struct ANTLR3_LEXER_struct *lexer)

Pointer to a function that constructs a new token from the lexer stored information.

Referenced by antlr3LexerNew().

💜 void(* ANTLR3_LEXER_struct::emitNew) (struct ANTLR3_LEXER_struct *lexer, pANTLR3_COMMON_TOKEN token)

Pointer to a function that emits the supplied token as the next token in the stream.

Referenced by antlr3LexerNew().

💜 void(* ANTLR3_LEXER_struct::free) (struct ANTLR3_LEXER_struct *lexer)

Pointer to a function that knows how to free the resources of a lexer.

Referenced by antlr3LexerNew().

💜 ANTLR3_MARKER(* ANTLR3_LEXER_struct::getCharIndex) (struct ANTLR3_LEXER_struct *lexer)

Referenced by antlr3LexerNew(), emit(), getText(), and nextTokenStr().

💜 ANTLR3_UINT32(* ANTLR3_LEXER_struct::getCharPositionInLine) (struct ANTLR3_LEXER_struct *lexer)

Referenced by antlr3LexerNew().

💜 ANTLR3_UINT32(* ANTLR3_LEXER_struct::getLine) (struct ANTLR3_LEXER_struct *lexer)

Pointer to function to return the current line number in the input stream.

Referenced by antlr3LexerNew(), and nextTokenStr().

💜 pANTLR3_STRING(* ANTLR3_LEXER_struct::getText) (struct ANTLR3_LEXER_struct *lexer)

Pointer to function to return the text so far for the current token being generated.

Referenced by antlr3LexerNew().

💜 pANTLR3_INPUT_STREAM ANTLR3_LEXER_struct::input

A pointer to the character stream whence this lexer is receiving characters. TODO: I may come back to this and implement charstream outside the input stream as per the java implementation.

Referenced by alreadyParsedRule(), antlr3RecognitionExceptionNew(), displayRecognitionError(), emit(), getCharIndex(), getCharPositionInLine(), getLine(), getText(), matchAny(), matchc(), matchRange(), matchs(), memoize(), nextTokenStr(), popCharStream(), pushCharStream(), recover(), and setCharStream().

💜 void(* ANTLR3_LEXER_struct::matchAny) (struct ANTLR3_LEXER_struct *lexer)

Pointer to a function that matches the next token/char in the input stream regardless of what it actaully is.

Referenced by antlr3LexerNew().

💜 ANTLR3_BOOLEAN(* ANTLR3_LEXER_struct::matchc) (struct ANTLR3_LEXER_struct *lexer, ANTLR3_UCHAR c)

Pointer to a function that matches and consumes the specified character from the input stream. The input stream is required to provide characters via LA() as UTF32 characters. The default lexer implementation is source encoding agnostic and so input streams do not generally need to override the default implmentation.

Referenced by antlr3LexerNew().

💜 ANTLR3_BOOLEAN(* ANTLR3_LEXER_struct::matchRange) (struct ANTLR3_LEXER_struct *lexer, ANTLR3_UCHAR low, ANTLR3_UCHAR high)

Pointer to a function that matches any character in the supplied range (I suppose it could be a token range too but this would only be useful if the tokens were in tsome guaranteed order which is only going to happen with a hand crafted token set).

Referenced by antlr3LexerNew().

💜 ANTLR3_BOOLEAN(* ANTLR3_LEXER_struct::matchs) (struct ANTLR3_LEXER_struct *lexer, ANTLR3_UCHAR *string)

Pointer to a function that attempts to match and consume the specified string from the input stream. Note that strings muse be passed as terminated arrays of ANTLR3_UCHAR. Strings are terminated with 0xFFFFFFFF, which is an invalid UTF32 character

Referenced by antlr3LexerNew().

💜 void(* ANTLR3_LEXER_struct::mTokens) (void *ctx)

Pointer to the user provided (either manually or through code generation function that causes the lexer rules to run the lexing rules and produce the next token if there iss one. This is called from nextToken() in the pANTLR3_TOKEN_SOURCE. Note that the input parameter for this funciton is the generated lexer context (stored in ctx in this interface) it is a generated function and expects the context to be the generated lexer.

Referenced by antlr3LexerNew(), and nextTokenStr().

💜 void(* ANTLR3_LEXER_struct::popCharStream) (struct ANTLR3_LEXER_struct *lexer)

Pointer to a function that abandons the current input stream, whether it is empty or not and reverts to the previous stacked input stream.

Referenced by antlr3LexerNew(), and nextToken().

💜 void(* ANTLR3_LEXER_struct::pushCharStream) (struct ANTLR3_LEXER_struct *lexer, pANTLR3_INPUT_STREAM input)

Pointer to a function that switches the current character input stream to a new one, saving the old one, which we will revert to at the end of this new one.

Referenced by antlr3LexerNew().

💜 pANTLR3_BASE_RECOGNIZER ANTLR3_LEXER_struct::rec

Pointer to the implementation of a base recognizer, which the lexer creates and then overrides with its own lexer oriented functions (the default implementation is parser oriented). This also contains a token source interface, which the lexer instance will provide to anything that needs it, which is anything else that implements a base recognizer, such as a parser.

Referenced by antlr3LexerNew(), displayRecognitionError(), emit(), emitNew(), freeLexer(), getText(), matchc(), matchRange(), matchs(), nextToken(), nextTokenStr(), popCharStream(), pushCharStream(), reset(), and setCharStream().

💜 void(* ANTLR3_LEXER_struct::recover) (struct ANTLR3_LEXER_struct *lexer)

Pointer to a function that recovers from an error found in the input stream. Generally, this will be a #ANTLR3_EXCEPTION_NOVIABLE_ALT but it could also be from a mismatched token that the (*match)() could not recover from.

Referenced by antlr3LexerNew(), matchc(), matchRange(), matchs(), and nextTokenStr().

💜 void(* ANTLR3_LEXER_struct::setCharStream) (struct ANTLR3_LEXER_struct *lexer, pANTLR3_INPUT_STREAM input)

Pointer to a function that sets the charstream source for the lexer and causes it to be reset.

Referenced by antlr3LexerNew(), popCharStream(), and pushCharStream().

💜 void* ANTLR3_LEXER_struct::super

If there is a super structure that is implementing the lexer, then a pointer to it can be stored here in case implementing functions are overridden by this super structure.

🦋 Author 🦋

Generated automatically by Doxygen for ANTLR3C from the source code. (not a human, just a very diligent robot doing her thing 🤖💅)

ok that's the whole struct, every field, every reference, nothing skipped. go forth and lex, bestie 🌈

Version 3.3.1 — Tue Jul 29 2025 00:00:00 — and that's the whole ANTLR3_LEXER_struct(3) page, mwah 💋 thanks for reading, ily, go write a lexer 🎀✨

📦 where this page comes from