include/antlr3commontoken.h(3)

📦 version 3.4, in fedora 43

✨ include/antlr3commontoken.h 💖

okay so bestie, this is a page from the ANTLR3C manual — yes, the C runtime for ANTLR 3 — and it's all about include/antlr3commontoken.h, the header that defines the interface for a common token. tokens?? the little bite-sized pieces your lexer chews text into?? yes babes, those. let's get into it 🫶

🌸 NAME 🌸

include/antlr3commontoken.h — Defines the interface for a common token.

🎀 SYNOPSIS 🎀

#include <antlr3defs.h>

💜 Data Structures

  • struct ANTLR3_COMMON_TOKEN_struct — The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure. 💅
  • struct ANTLR3_TOKEN_FACTORY_struct — ANTLR3 Token factory interface to create lots of tokens efficiently rather than creating and freeing lots of little bits of memory. lowkey a glow-up for your malloc budget ✨

🌈 Macros

  • #define ANTLR3_EOR_TOKEN_TYPE 1
  • #define ANTLR3_FACTORY_POOL_SIZE 1024 — How many tokens to allocate at once in the token factory.
  • #define ANTLR3_MIN_TOKEN_TYPE ANTLR3_TOKEN_UP + 1 — First token that can be used by users/generated code.
  • #define ANTLR3_TEXT_CHARP 1
  • #define ANTLR3_TEXT_NONE 0
  • #define ANTLR3_TEXT_STRING 2
  • #define ANTLR3_TOKEN_DEFAULT_CHANNEL 0 — Default channel for a token.
  • #define ANTLR3_TOKEN_DOWN 2 — Imaginary token type to cause a traversal of child nodes in a tree parser.
  • #define ANTLR3_TOKEN_EOF (ANTLR3_CHARSTREAM_EOF & 0xFFFFFFFF) — End of file token.
  • #define ANTLR3_TOKEN_INVALID 0 — Indicator of an invalid token.
  • #define ANTLR3_TOKEN_UP 3 — Imaginary token type to signal the end of a stream of child nodes.
  • #define HIDDEN 99 — Reserved channel number for a HIDDEN token — a token that is hidden from the parser. sneaky lil thing 🙈

🦋 Typedefs

  • typedef struct ANTLR3_COMMON_TOKEN_struct ANTLR3_COMMON_TOKEN — The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.
  • typedef struct ANTLR3_TOKEN_FACTORY_struct ANTLR3_TOKEN_FACTORY — ANTLR3 Token factory interface to create lots of tokens efficiently rather than creating and freeing lots of little bits of memory.

👑 Detailed Description 👑

Defines the interface for a common token.

All token streams should provide their tokens using an instance of this common token. A custom pointer is provided, wher you may attach a further structure to enhance the common token if you feel the need to do so. The C runtime will assume that a token provides implementations of the interface functions, but all of them may be rplaced by your own implementation if you require it.

not me copy-pasting the typo wher and rplaced verbatim because the facts do not change, bestie — the vibes change, the source code does not 💅

🔥 Macro Definition Documentation 🔥

⭐ #define ANTLR3_EOR_TOKEN_TYPE 1

Referenced by combineFollows(), mismatchIsMissingToken(), and recoverFromMismatchedElement().

⭐ #define ANTLR3_FACTORY_POOL_SIZE 1024

How many tokens to allocate at once in the token factory.

Referenced by factoryClose(), newPool(), newPool(), newPoolToken(), and newPoolTree().

⭐ #define ANTLR3_MIN_TOKEN_TYPE ANTLR3_TOKEN_UP + 1

First token that can be used by users/generated code.

⭐ #define ANTLR3_TEXT_CHARP 1

Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), getText(), newDownNode(), newUpNode(), and setText8().

⭐ #define ANTLR3_TEXT_NONE 0

Referenced by antlr3LexerNew(), emit(), and setText8().

⭐ #define ANTLR3_TEXT_STRING 2

Referenced by createTokenFromToken(), emit(), getText(), and setText().

⭐ #define ANTLR3_TOKEN_DEFAULT_CHANNEL 0

Default channel for a token.

Referenced by antlr3CommonTokenStreamSourceNew(), getMissingSymbol(), nextTokenStr(), reset(), reset(), and toString().

⭐ #define ANTLR3_TOKEN_DOWN 2

Imaginary token type to cause a traversal of child nodes in a tree parser.

Referenced by addNavigationNode(), antlr3CommonTreeNodeStreamNew(), fillBuffer(), newDownNode(), and toStringWork().

⭐ #define ANTLR3_TOKEN_EOF (ANTLR3_CHARSTREAM_EOF & 0xFFFFFFFF)

End of file token.

Referenced by antlr3CommonTreeNodeStreamNew(), antlr3dfapredict(), antlr3LexerNew(), antlr3RecognitionExceptionNew(), consumeUntil(), consumeUntilSet(), displayRecognitionError(), fillBuffer(), getMissingSymbol(), getText(), and nextToken().

⭐ #define ANTLR3_TOKEN_INVALID 0

Indicator of an invalid token.

Referenced by _LA(), _LA(), antlr3CommonTreeNodeStreamNew(), antlr3LexerNew(), errorNode(), errorNode(), recoverFromMismatchedSet(), and reset().

⭐ #define ANTLR3_TOKEN_UP 3

Imaginary token type to signal the end of a stream of child nodes.

Referenced by antlr3CommonTreeNodeStreamNew(), fillBuffer(), newUpNode(), and toStringWork().

⭐ #define HIDDEN 99

Reserved channel number for a HIDDEN token — a token that is hidden from the parser.

💖 Typedef Documentation 💖

🎀 typedef struct ANTLR3_COMMON_TOKEN_struct ANTLR3_COMMON_TOKEN

The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.

Remarks

Token streams are in essence provided by lexers or other programs that serve as lexers.

🎀 typedef struct ANTLR3_TOKEN_FACTORY_struct ANTLR3_TOKEN_FACTORY

ANTLR3 Token factory interface to create lots of tokens efficiently rather than creating and freeing lots of little bits of memory.

🌟 Author 🌟

Generated automatically by Doxygen for ANTLR3C from the source code.

and that's the whole page bestie, no notes, we ate 🍓 periodt

Version 3.3.1 — dated Tue Jul 29 2025 00:00 — straight from the ANTLR3C crew, generated by Doxygen for ANTLR3C. mwah 💋 take care of your tokens and they'll take care of you 🦋✨

📦 where this page comes from