ANTLR3_COMMON_TOKEN_struct(3)

📦 version 3.4, in fedora 43

💖 `ANTLR3_COMMON_TOKEN_struct` 💖

okay bestie, sit down, hydrate, because this is ANTLR3_COMMON_TOKEN_struct(3) — straight outta the ANTLR3C manual, section 3, and it is lowkey a whole personality. This is the struct page. The big one. The main character. ✨

🎀 NAME 🎀

ANTLR3_COMMON_TOKEN_struct — the definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure. 💅

🌸 SYNOPSIS 🌸

#include <antlr3commontoken.h>

🦋 Data Fields 🦋

ANTLR3_UINT32 channel — the virtual channel that this token exists in. 📡
ANTLR3_INT32 charPosition — the character position in the line that this token was derived from.
void * custom — pointer to a custom element that the ANTLR3 programmer may define and install. it's giving "bring your own struct" 🫶
ANTLR3_BOOLEAN factoryMade — indicates that a token was produced from the token factory, and therefore the freeToken() method should not do anything itself, because the token factory is responsible for deleting it. no cap.
void(* freeCustom )(void *custom) — pointer to a function that knows how to free the custom structure when the token is destroyed. 🧹
ANTLR3_UINT32(* getChannel )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels. ⚠️ yes, that parenthesis is the original's, it never closes, we love her anyway
ANTLR3_INT32(* getCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that gets the offset in the line where this token exists.
ANTLR3_UINT32(* getLine )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that gets the 'line' number where this token resides.
ANTLR3_MARKER(* getStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that gets the start index in the input stream for this token.
ANTLR3_MARKER(* getStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that gets the stop index in the input stream for this token.
pANTLR3_STRING(* getText )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token. (note: it's "pointer to function", not "pointer to a function. we respect the typo. 💀)
ANTLR3_MARKER(* getTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that returns an index 0...n-1 of the token in the token input stream.
ANTLR3_UINT32(* getType )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that returns the token type of this token.
ANTLR3_MARKER index — what the index of this token is: 0, 1, .., n-2, n-1 tokens. 🔢
pANTLR3_INPUT_STREAM input — pointer to the input stream that this token originated in.
ANTLR3_UINT32 line — the line number in the input stream where this token was derived from.
void * lineStart — the offset into the input stream that the line in which this token resides starts.
void(* setChannel )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel) — pointer to a function that sets the channel that this token should belong to.
void(* setCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos) — pointer to a function that sets the offset in the line where this token exists.
void(* setLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line) — pointer to a function that sets the 'line' number where this token reside. (yes. "reside". iconic. 👑)
void(* setStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index) — pointer to a function that sets the start index in the input stream for this token.
void(* setStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index) — pointer to a function that sets the stop index in the input stream for this token.
void(* setText )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text) — pointer to a function that 'might' be able to set the text associated with a token. might!! she's not making promises!!
void(* setText8 )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text) — pointer to a function that 'might' be able to set the text associated with a token.
void(* setTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER) — pointer to a function that can set the token index of this token in the token input stream.
void(* setType )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype) — pointer to a function that sets the type of this token.
ANTLR3_MARKER start — the character offset in the input stream where the text for this token starts.
ANTLR3_MARKER stop — the character offset in the input stream where the text for this token stops.
pANTLR3_STRING_FACTORY strFactory — a string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.
ANTLR3_UINT32 textState — indicates whether this token is carrying:
the union tokText, and inside her:

union {
   pANTLR3_UCHAR chars
   pANTLR3_STRING text
} tokText

pANTLR3_UCHAR chars — pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar.
pANTLR3_STRING text — some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.

pANTLR3_STRING(* toString )(struct ANTLR3_COMMON_TOKEN_struct *token) — pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as \n replaced with the printable sequence "\\n". This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations. " (the stray quote at the end is literally in the original, I'm just the messenger 🫡)
ANTLR3_UINT32 type — the actual type of this token.
ANTLR3_UINT32 user1 — because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. 🫶
ANTLR3_UINT32 user2 — same energy, same reason, different field. because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user3 — and the third one, for the same reason: because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.

💜 Detailed Description 💜

The definition of an ANTLR3_COMMON_TOKEN structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure. periodt.

🌈 Remarks 🌈

Token streams are in essence provided by lexers or other programs that serve as lexers. ✨

🧚 Field Documentation 🧚

okay this is the part where every single field gets her own little spotlight. obsessed. 💖

🍓 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::channel`

The virtual channel that this token exists in.

Referenced by emit(), getChannel(), setChannel(), skipOffTokenChannels(), and skipOffTokenChannelsReverse().

🍓 `ANTLR3_INT32 ANTLR3_COMMON_TOKEN_struct::charPosition`

The character position in the line that this token was derived from.

Referenced by emit(), getCharPositionInLine(), and setCharPositionInLine().

🍓 `pANTLR3_UCHAR ANTLR3_COMMON_TOKEN_struct::chars`

Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar. No sense in constructing a pANTLR3_STRING just for that, as mostly the text will not be accessed — if it is, then we will build a pANTLR3_STRING for it a that point. (fr fr, lazy is efficient 💅)

Referenced by antlr3CommonTreeNodeStreamNew(), createTokenFromToken(), getText(), newDownNode(), newUpNode(), and setText8().

🍓 `void* ANTLR3_COMMON_TOKEN_struct::custom`

Pointer to a custom element that the ANTLR3 programmer may define and install.

Referenced by antlr3LexerNew(), emit(), factoryClose(), and getMissingSymbol().

🍓 `ANTLR3_BOOLEAN ANTLR3_COMMON_TOKEN_struct::factoryMade`

Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.

Referenced by antlr3LexerNew(), antlr3TokenFactoryNew(), newPoolToken(), newToken(), and nextTokenStr().

🍓 `void(* ANTLR3_COMMON_TOKEN_struct::freeCustom) (void *custom)`

Pointer to a function that knows how to free the custom structure when the token is destroyed.

Referenced by factoryClose().

🍓 `ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getChannel) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), fillBuffer(), serializeToken(), and toString().

🍓 `ANTLR3_INT32(* ANTLR3_COMMON_TOKEN_struct::getCharPositionInLine) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that gets the offset in the line where this token exists.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getCharPositionInLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().

🍓 `ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getLine) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that gets the 'line' number where this token resides.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().

🍓 `ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStartIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that gets the start index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), getText(), and toString().

🍓 `ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStopIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that gets the stop index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), getText(), and toString().

🍓 `pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::getText) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), getMissingSymbol(), serializeToken(), toString(), and toStringSS(). (yes, getMissingSymbol() is listed twice. the original said it twice, so I say it twice 🫡)

🍓 `ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getTokenIndex) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that returns an index 0...n-1 of the token in the token input stream.

Referenced by antlr3SetTokenAPI(), createNodeTok(), createTokenFromToken(), dbgSetTokenBoundaries(), serializeToken(), setTokenBoundaries(), toString(), and toStringTT().

🍓 `ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getType) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that returns the token type of this token.

⭐ to be continued, bestie — part 2 has the rest of the field docs and she's just as loud ⭐

🗺️ `ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::index`

What the index of this token is, fr — like 0, 1, .., n-2, n-1 tokens. It's giving roll-call energy 💅

Referenced by getTokenIndex(), and setTokenIndex().

🌊 `pANTLR3_INPUT_STREAM ANTLR3_COMMON_TOKEN_struct::input`

Pointer to the input stream that this token originated in. Where it came from, bestie, its whole origin story 🦋

Referenced by antlr3RecognitionExceptionNew(), getMissingSymbol(), getStartIndex(), getText(), newPoolToken(), and setInputStream().

📍 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::line`

The line number in the input stream where this token was derived from. Yes literally the line, like line 47 of ur file 🎀

Referenced by emit(), getLine(), and setLine().

⬅️ `void* ANTLR3_COMMON_TOKEN_struct::lineStart`

The offset into the input stream that the line in which this token resides starts. So where the line began, not where the token began, ok subtle but important ✨

Referenced by emit(), and getMissingSymbol().

📡 `void(* ANTLR3_COMMON_TOKEN_struct::setChannel) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)`

Pointer to a function that sets the channel that this token should belong to. Pick ur channel bestie, default or hidden 📺

Referenced by antlr3SetTokenAPI(), fillBuffer(), and getMissingSymbol().

📐 `void(* ANTLR3_COMMON_TOKEN_struct::setCharPositionInLine) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)`

Pointer to a function that sets the offset in the line where this token exists. Column vibes 🎯

Referenced by antlr3SetTokenAPI(), and getMissingSymbol().

🧾 `void(* ANTLR3_COMMON_TOKEN_struct::setLine) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)`

Pointer to a function that sets the 'line' number where this token reside. Reside. It said reside. Iconic word choice fr fr 👑

Referenced by antlr3SetTokenAPI(), getMissingSymbol(), and nextTokenStr().

🚩 `void(* ANTLR3_COMMON_TOKEN_struct::setStartIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)`

Pointer to a function that sets the start index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), newPoolToken(), nextTokenStr(), and tokLT().

🏁 `void(* ANTLR3_COMMON_TOKEN_struct::setStopIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)`

Pointer to a function that sets the stop index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), nextTokenStr(), and tokLT().

✍️ `void(* ANTLR3_COMMON_TOKEN_struct::setText) (struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)`

Pointer to a function that 'might' be able to set the text associated with a token. Read that "might" twice, bestie, it's doing a LOT of heavy lifting 💀 Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically. Yeah. Drastically. No cap, do not test this 🙈

Referenced by antlr3SetTokenAPI().

✍️ `void(* ANTLR3_COMMON_TOKEN_struct::setText8) (struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)`

Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.

Referenced by antlr3SetTokenAPI(), createTypeTokenText(), getMissingSymbol(), and getMissingSymbol().

Not me noticing getMissingSymbol() is listed twice — that's what the original says, so we keep it, no editing, periodt 💅

🔢 `void(* ANTLR3_COMMON_TOKEN_struct::setTokenIndex) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)`

Pointer to a function that can set the token index of this token in the token input stream.

Referenced by antlr3SetTokenAPI(), and fillBuffer().

🏷️ `void(* ANTLR3_COMMON_TOKEN_struct::setType) (struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)`

Pointer to a function that sets the type of this token.

Referenced by antlr3CommonTokenNew(), antlr3LexerNew(), antlr3SetTokenAPI(), createTypeToken(), createTypeTokenText(), and getMissingSymbol().

🎬 `ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::start`

The character offset in the input stream where the text for this token starts.

Referenced by emit(), getStartIndex(), and setStartIndex().

🛑 `ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::stop`

The character offset in the input stream where the text for this token stops.

Referenced by emit(), getStopIndex(), and setStopIndex().

🏭 `pANTLR3_STRING_FACTORY ANTLR3_COMMON_TOKEN_struct::strFactory`

A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING. We're literally building the string on demand, custom order, artisanal ✨

Referenced by ANTLR3_TREE_ADAPTORNew(), antlr3CommonTreeNodeStreamNew(), antlr3CommonTreeNodeStreamNewStream(), antlr3LexerNew(), getText(), newPoolToken(), setCharStream(), and setInputStream().

💬 `pANTLR3_STRING ANTLR3_COMMON_TOKEN_struct::text`

Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL. Some tokens are self-sufficient like that 💅

Referenced by emit(), getText(), setText(), and setText8().

🔮 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::textState`

Indicates whether this token is carrying:

0 | Nothing (neither rewrite text, nor setText)
1 | char * to user supplied rewrite text
2 | pANTLR3_STRING because of setText or similar action

Affects the union structure tokText below (uses 32 bit so alignment is always good) alignment stays blessed, we love to see it 🙏

Referenced by antlr3CommonTreeNodeStreamNew(), antlr3LexerNew(), createTokenFromToken(), emit(), getText(), newDownNode(), newUpNode(), setText(), and setText8().

🧺 `union { ... } ANTLR3_COMMON_TOKEN_struct::tokText`

Referenced by antlr3CommonTreeNodeStreamNew(), createTokenFromToken(), emit(), getText(), newDownNode(), newUpNode(), setText(), and setText8().

🖨️ `pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::toString) (struct ANTLR3_COMMON_TOKEN_struct *token)`

Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as replaced with the printable sequence "\\n" This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations. Making ugly control codes printable so ur terminal doesn't have a meltdown 🫶

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and displayRecognitionError().

🎭 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::type`

The actual type of this token. The real one. The main character 💖

Referenced by displayRecognitionError(), emit(), fillBuffer(), getText(), getType(), nextToken(), setType(), and toString().

🧑‍💻 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user1`

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them. Ur little corner of the struct, do whatever, it's urs 🌈

Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().

🧑‍💻 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user2`

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.

Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().

🧑‍💻 `ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user3`

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.

Referenced by antlr3LexerNew(), emit(), and getMissingSymbol().

👑 AUTHOR

Generated automatically by Doxygen for ANTLR3C from the source code. No human wrote this one, it just appeared, fully formed, like a biblically accurate manpage 🧚

And that's the whole page, bestie — you read all of it, which honestly? Iconic behaviour. 🌸

This was Version 3.3.1, dated Tue Jul 29 2025 00:00, and it's the ANTLR3_COMMON_TOKEN_struct(3) page 💜 Go forth and tokenize, mwah 🫶✨

📦 where this page comes from

fedora 43: antlr3-C-devel 1:3.4-3.5.3.14.fc43 · BSD-3-Clause
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-14: a language model's rewrite of the original, under its terms

ANTLR3_COMMON_TOKEN_struct(3)