src/antlr3tokenstream.c(3)

📦 version 3.4, in fedora 43

💖 src/antlr3tokenstream.c(3) — ANTLR3C 💖

okay bestie, sit down, hydrate, because we're reading the src/antlr3tokenstream.c(3) manual page together and it's genuinely a lot. this is the C source file page for the ANTLR3 C runtime, and it's all about the default implementation of CommonTokenStream. slay. ✨

🎀 NAME 🎀

src/antlr3tokenstream.c — the default implementation of CommonTokenStream. that's it, that's the vibe. 💅

🦋 SYNOPSIS 🦋

#include <antlr3tokenstream.h>

🌸 Functions 🌸

here's the whole roster, no cap, every single one:

  • static ANTLR3_UINT32 _LA (pANTLR3_INT_STREAM is, ANTLR3_INT32 i)
  • ANTLR3_API pANTLR3_COMMON_TOKEN_STREAM antlr3CommonTokenDebugStreamSourceNew (ANTLR3_UINT32 hint, pANTLR3_TOKEN_SOURCE source, pANTLR3_DEBUG_EVENT_LISTENER debugger)
  • ANTLR3_API pANTLR3_COMMON_TOKEN_STREAM antlr3CommonTokenStreamNew (ANTLR3_UINT32 hint)
  • ANTLR3_API pANTLR3_COMMON_TOKEN_STREAM antlr3CommonTokenStreamSourceNew (ANTLR3_UINT32 hint, pANTLR3_TOKEN_SOURCE source)
  • static void antlr3CTSFree (pANTLR3_COMMON_TOKEN_STREAM stream)
  • static void antlr3TokenStreamFree (pANTLR3_TOKEN_STREAM stream)
  • ANTLR3_API pANTLR3_TOKEN_STREAM antlr3TokenStreamNew ()
  • static void consume (pANTLR3_INT_STREAM is) — moves the input pointer to the next incoming token. 👉
  • static void consumeInitialHiddenTokens (pANTLR3_INT_STREAM is) — a debug-only method, its whole job is flagging that we consumed the initial off-channel tokens in the input stream. 🕵️
  • static void dbgConsume (pANTLR3_INT_STREAM is) — same energy as the ordinary consume, but it also tells the debugger about hidden tokens and so on.
  • static ANTLR3_UINT32 dbgLA (pANTLR3_INT_STREAM is, ANTLR3_INT32 i) — literally _LA() but for debug mode.
  • static ANTLR3_MARKER dbgMark (pANTLR3_INT_STREAM is) — literally mark() but with a lil call that tells the debugger we're doing this.
  • static void dbgRewindLast (pANTLR3_INT_STREAM is)
  • static void dbgRewindStream (pANTLR3_INT_STREAM is, ANTLR3_MARKER marker)
  • static void dbgSeek (pANTLR3_INT_STREAM is, ANTLR3_MARKER index)
  • static pANTLR3_COMMON_TOKEN dbgTokLT (pANTLR3_TOKEN_STREAM ts, ANTLR3_INT32 k) — like the normal tokLT except it sends information over to the debugger.
  • static void discardOffChannel (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_BOOLEAN discard)
  • static void discardTokenType (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_INT32 ttype)
  • static void fillBuffer (pANTLR3_COMMON_TOKEN_STREAM tokenStream)
  • ANTLR3_API void fillBufferExt (pANTLR3_COMMON_TOKEN_STREAM tokenStream)
  • static pANTLR3_COMMON_TOKEN get (pANTLR3_TOKEN_STREAM ts, ANTLR3_UINT32 i)
  • static pANTLR3_STRING getSourceName (pANTLR3_INT_STREAM is) — returns a string that represents the name associated with the input source. (the original spells it "assoicated", iconic typo, we love her 💀)
  • static pANTLR3_LIST getTokenRange (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)
  • static pANTLR3_VECTOR getTokens (pANTLR3_COMMON_TOKEN_STREAM cts)
  • static pANTLR3_LIST getTokensList (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list)
  • static pANTLR3_TOKEN_SOURCE getTokenSource (pANTLR3_TOKEN_STREAM ts)
  • static pANTLR3_LIST getTokensSet (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types) — given a start and stop index, it returns a List of all tokens in the token type BitSet.
  • static pANTLR3_LIST getTokensType (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type)
  • static pANTLR3_COMMON_TOKEN LB (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_INT32 i)
  • static ANTLR3_MARKER mark (pANTLR3_INT_STREAM is)
  • static void release (pANTLR3_INT_STREAM is, ANTLR3_MARKER mark)
  • static void reset (pANTLR3_COMMON_TOKEN_STREAM cts)
  • static void rewindLast (pANTLR3_INT_STREAM is)
  • static void rewindStream (pANTLR3_INT_STREAM is, ANTLR3_MARKER marker)
  • static void seek (pANTLR3_INT_STREAM is, ANTLR3_MARKER index)
  • static void setDebugListener (pANTLR3_TOKEN_STREAM ts, pANTLR3_DEBUG_EVENT_LISTENER debugger)
  • static void setTokenSource (pANTLR3_TOKEN_STREAM ts, pANTLR3_TOKEN_SOURCE tokenSource)
  • static void setTokenTypeChannel (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel) — a simple filter mechanism where you can tell this token stream to force all tokens of type ttype to be on channel.
  • static ANTLR3_UINT32 size (pANTLR3_INT_STREAM is)
  • static ANTLR3_UINT32 skipOffTokenChannels (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_INT32 i) — given a starting index, return the index of the first on-channel token.
  • static ANTLR3_UINT32 skipOffTokenChannelsReverse (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_INT32 i)
  • static ANTLR3_MARKER tindex (pANTLR3_INT_STREAM is)
  • static pANTLR3_COMMON_TOKEN tokLT (pANTLR3_TOKEN_STREAM ts, ANTLR3_INT32 k) — grab the *i*th token from the current position, 1..n, where k=1 is the first symbol of lookahead.
  • static pANTLR3_STRING toString (pANTLR3_TOKEN_STREAM ts)
  • static pANTLR3_STRING toStringSS (pANTLR3_TOKEN_STREAM ts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)
  • static pANTLR3_STRING toStringTT (pANTLR3_TOKEN_STREAM ts, pANTLR3_COMMON_TOKEN start, pANTLR3_COMMON_TOKEN stop)

✨ Detailed Description ✨

Default implementation of CommonTokenStream. that's the whole description, she's concise. 👑

👑 Function Documentation 👑

🌈 ANTLR3_UINT32 _LA (pANTLR3_INT_STREAM is, ANTLR3_INT32 i) [static] 🌈

this one is out here referencing ANTLR3_TOKEN_STREAM_struct::_LT, ANTLR3_TOKEN_INVALID, ANTLR3_COMMON_TOKEN_struct::getType, and ANTLR3_INT_STREAM_struct::super.

and who's referencing her? dbgLA(). periodt.

🌈 ANTLR3_API pANTLR3_COMMON_TOKEN_STREAM antlr3CommonTokenDebugStreamSourceNew (ANTLR3_UINT32 hint, pANTLR3_TOKEN_SOURCE source, pANTLR3_DEBUG_EVENT_LISTENER debugger) 🌈

the references, in full, no skipping: ANTLR3_INT_STREAM_struct::_LA, ANTLR3_TOKEN_STREAM_struct::_LT, ANTLR3_FALSE, antlr3CommonTokenStreamSourceNew(), ANTLR3_INT_STREAM_struct::consume, dbgConsume(), dbgLA(), dbgMark(), dbgRewindLast(), dbgRewindStream(), dbgSeek(), dbgTokLT(), ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_TOKEN_STREAM_struct::initialStreamState, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_INT_STREAM_struct::mark, ANTLR3_INT_STREAM_struct::rewind, ANTLR3_INT_STREAM_struct::rewindLast, ANTLR3_INT_STREAM_struct::seek, and ANTLR3_COMMON_TOKEN_STREAM_struct::tstream.

🌈 ANTLR3_API pANTLR3_COMMON_TOKEN_STREAM antlr3CommonTokenStreamNew (ANTLR3_UINT32 hint) 🌈

the reference list is giving maximalist, and we respect it: _LA(), ANTLR3_INT_STREAM_struct::_LA, ANTLR3_TOKEN_STREAM_struct::_LT, ANTLR3_TOKENSTREAM, antlr3IntStreamNew(), antlr3TokenStreamNew(), antlr3VectorNew(), ANTLR3_INT_STREAM_struct::consume, consume(), discardOffChannel(), ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannelToks, ANTLR3_COMMON_TOKEN_STREAM_struct::discardTokenType, discardTokenType(), ANTLR3_TOKEN_STREAM_struct::get, get(), ANTLR3_INT_STREAM_struct::getSourceName, getSourceName(), ANTLR3_COMMON_TOKEN_STREAM_struct::getTokenRange, getTokenRange(), ANTLR3_COMMON_TOKEN_STREAM_struct::getTokens, getTokens(), ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensList, getTokensList(), ANTLR3_TOKEN_STREAM_struct::getTokenSource, getTokenSource(), ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensSet, getTokensSet(), ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensType, getTokensType(), ANTLR3_INT_STREAM_struct::index, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_INT_STREAM_struct::mark, mark(), ANTLR3_COMMON_TOKEN_STREAM_struct::p, ANTLR3_INT_STREAM_struct::release, release(), ANTLR3_COMMON_TOKEN_STREAM_struct::reset, reset(), ANTLR3_INT_STREAM_struct::rewind, ANTLR3_INT_STREAM_struct::rewindLast, rewindLast(), rewindStream(), ANTLR3_INT_STREAM_struct::seek, seek(), ANTLR3_TOKEN_STREAM_struct::setDebugListener, setDebugListener(), ANTLR3_TOKEN_STREAM_struct::setTokenSource, setTokenSource(), ANTLR3_COMMON_TOKEN_STREAM_struct::setTokenTypeChannel, setTokenTypeChannel(), ANTLR3_INT_STREAM_struct::size, size(), ANTLR3_INT_STREAM_struct::super, ANTLR3_TOKEN_STREAM_struct::super, tindex(), ANTLR3_COMMON_TOKEN_STREAM_struct::tokens, tokLT(), ANTLR3_TOKEN_STREAM_struct::toString, toString(), ANTLR3_TOKEN_STREAM_struct::toStringSS, toStringSS(), ANTLR3_TOKEN_STREAM_struct::toStringTT, toStringTT(), ANTLR3_COMMON_TOKEN_STREAM_struct::tstream, and ANTLR3_INT_STREAM_struct::type.

who's referencing this one? antlr3CommonTokenStreamSourceNew(). 🫶

🌈 ANTLR3_API pANTLR3_COMMON_TOKEN_STREAM antlr3CommonTokenStreamSourceNew (ANTLR3_UINT32 hint, pANTLR3_TOKEN_SOURCE source) 🌈

references: ANTLR3_FALSE, ANTLR3_TOKEN_DEFAULT_CHANNEL, antlr3CommonTokenStreamNew(), antlr3CTSFree(), ANTLR3_COMMON_TOKEN_STREAM_struct::channel, ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverrides, ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannel, ANTLR3_COMMON_TOKEN_STREAM_struct::discardSet, ANTLR3_COMMON_TOKEN_STREAM_struct::free, ANTLR3_TOKEN_STREAM_struct::setTokenSource, and ANTLR3_COMMON_TOKEN_STREAM_struct::tstream.

referenced by antlr3CommonTokenDebugStreamSourceNew().

🌈 void antlr3CTSFree (pANTLR3_COMMON_TOKEN_STREAM stream) [static] 🌈

references: ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverrides, ANTLR3_COMMON_TOKEN_STREAM_struct::discardSet, ANTLR3_INT_STREAM_struct::free, ANTLR3_LIST_struct::free, ANTLR3_TOKEN_STREAM_struct::free, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_INT_STREAM_struct::super, ANTLR3_TOKEN_STREAM_struct::super, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens, and ANTLR3_COMMON_TOKEN_STREAM_struct::tstream.

referenced by antlr3CommonTokenStreamSourceNew().

🌈 void antlr3TokenStreamFree (pANTLR3_TOKEN_STREAM stream) [static] 🌈

referenced by antlr3TokenStreamNew(). short and sweet, she said little but she meant it. 💅

🌈 ANTLR3_API pANTLR3_TOKEN_STREAM antlr3TokenStreamNew () 🌈

references antlr3TokenStreamFree(), and ANTLR3_TOKEN_STREAM_struct::free.

referenced by antlr3CommonTokenStreamNew().

🌈 void consume (pANTLR3_INT_STREAM is) [static] 🌈

okay so this one moves the input pointer to the next incoming token. the stream has to become active with LT(1) available. consume() simply moves the input pointer so that LT(1) points at the next input symbol. consume at least one token. ✅

it also walks past any token that's not on the channel the parser is listening to. bye girl, not on my channel. 🚶‍♀️

references: ANTLR3_VECTOR_struct::count, ANTLR3_COMMON_TOKEN_STREAM_struct::p, skipOffTokenChannels(), ANTLR3_INT_STREAM_struct::super, ANTLR3_TOKEN_STREAM_struct::super, and ANTLR3_COMMON_TOKEN_STREAM_struct::tokens.

referenced by dbgConsume().

🌈 void consumeInitialHiddenTokens (pANTLR3_INT_STREAM is) [static] 🌈

debug-only method to flag consumption of the initial off-channel tokens in the input stream. it's the little gremlin that watches the hidden ones. 🙈

references: ANTLR3_FALSE, ANTLR3_DEBUG_EVENT_LISTENER_struct::consumeHiddenToken, ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_TOKEN_STREAM_struct::get, ANTLR3_INT_STREAM_struct::index, ANTLR3_TOKEN_STREAM_struct::initialStreamState, and ANTLR3_INT_STREAM_struct::super.

referenced by dbgConsume(), dbgLA(), and dbgTokLT().

🌈 void dbgConsume (pANTLR3_INT_STREAM is) [static] 🌈

as per ordinary consume, but it notifies the debugger about hidden tokens and so on. same girl, more gossip. 🗣️

So that last one wraps up with these connections, bestie — nothing skipped, promise:

  • References: ANTLR3_TOKEN_STREAM_struct::_LT, ANTLR3_TRUE, consume(), ANTLR3_DEBUG_EVENT_LISTENER_struct::consumeHiddenToken, consumeInitialHiddenTokens(), ANTLR3_DEBUG_EVENT_LISTENER_struct::consumeToken, ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_TOKEN_STREAM_struct::get, ANTLR3_INT_STREAM_struct::index, ANTLR3_TOKEN_STREAM_struct::initialStreamState, ANTLR3_INT_STREAM_struct::super
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

🎀 dbgLA 🎀

ANTLR3_UINT32 dbgLA (pANTLR3_INT_STREAM is, ANTLR3_INT32 i) [static]

Literally the same energy as _LA() but it's doing it for debug mode. Same lookahead, extra snitching to the debugger 💅

  • References: _LA(), ANTLR3_TRUE, consumeInitialHiddenTokens(), ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_TOKEN_STREAM_struct::initialStreamState, ANTLR3_DEBUG_EVENT_LISTENER_struct::LT, ANTLR3_INT_STREAM_struct::super, tokLT()
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

🦋 dbgMark 🦋

ANTLR3_MARKER dbgMark (pANTLR3_INT_STREAM is) [static]

Same as mark() except it also tells the debugger "hey bestie, I'm marking this spot rn" ✨

  • References: ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_INT_STREAM_struct::index, ANTLR3_INT_STREAM_struct::lastMarker, ANTLR3_DEBUG_EVENT_LISTENER_struct::mark, ANTLR3_INT_STREAM_struct::super
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

🌸 dbgRewindLast 🌸

void dbgRewindLast (pANTLR3_INT_STREAM is) [static]
  • References: ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_INT_STREAM_struct::lastMarker, ANTLR3_INT_STREAM_struct::rewind, ANTLR3_DEBUG_EVENT_LISTENER_struct::rewindLast, ANTLR3_INT_STREAM_struct::super
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

💜 dbgRewindStream 💜

void dbgRewindStream (pANTLR3_INT_STREAM is, ANTLR3_MARKER marker) [static]
  • References: ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_DEBUG_EVENT_LISTENER_struct::rewind, ANTLR3_INT_STREAM_struct::seek, ANTLR3_INT_STREAM_struct::super
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

🍓 dbgSeek 🍓

void dbgSeek (pANTLR3_INT_STREAM is, ANTLR3_MARKER index) [static]
  • References: seek()
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

🧚 dbgTokLT 🧚

pANTLR3_COMMON_TOKEN dbgTokLT (pANTLR3_TOKEN_STREAM ts, ANTLR3_INT32 k)
[static]

Exactly like the regular tokLT except it spills all the tea to the debugger 📣

  • References: ANTLR3_TRUE, consumeInitialHiddenTokens(), ANTLR3_TOKEN_STREAM_struct::initialStreamState, ANTLR3_TOKEN_STREAM_struct::istream, tokLT()
  • Referenced by: antlr3CommonTokenDebugStreamSourceNew(), setDebugListener()

👑 discardOffChannel 👑

void discardOffChannel (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_BOOLEAN discard) [static]
  • References: ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannel
  • Referenced by: antlr3CommonTokenStreamNew()

✨ discardTokenType ✨

void discardTokenType (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_INT32 ttype)
[static]
  • References: ANTLR3_FUNC_PTR, antlr3ListNew(), ANTLR3_COMMON_TOKEN_STREAM_struct::discardSet, ANTLR3_LIST_struct::put
  • Referenced by: antlr3CommonTokenStreamNew()

🔥 fillBuffer 🔥

void fillBuffer (pANTLR3_COMMON_TOKEN_STREAM tokenStream) [static]

The one that does the actual work of stuffing tokens in. Iconic behaviour, no cap.

  • References: ANTLR3_VECTOR_struct::add, ANTLR3_FALSE, ANTLR3_TOKEN_EOF, ANTLR3_TRUE, ANTLR3_UINT32_CAST, ANTLR3_INT_STREAM_struct::cachedSize, ANTLR3_COMMON_TOKEN_STREAM_struct::channel, ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverrides, ANTLR3_VECTOR_struct::count, ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannel, ANTLR3_COMMON_TOKEN_STREAM_struct::discardSet, ANTLR3_LIST_struct::get, ANTLR3_COMMON_TOKEN_struct::getChannel, ANTLR3_COMMON_TOKEN_struct::getType, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_TOKEN_SOURCE_struct::nextToken, ANTLR3_COMMON_TOKEN_STREAM_struct::p, ANTLR3_COMMON_TOKEN_struct::setChannel, ANTLR3_COMMON_TOKEN_struct::setTokenIndex, skipOffTokenChannels(), ANTLR3_COMMON_TOKEN_STREAM_struct::tokens, ANTLR3_TOKEN_STREAM_struct::tokenSource, ANTLR3_COMMON_TOKEN_STREAM_struct::tstream, ANTLR3_COMMON_TOKEN_struct::type
  • Referenced by: fillBufferExt(), getTokens(), getTokensSet(), LB(), tokLT(), toString(), toStringSS()

🌈 fillBufferExt 🌈

ANTLR3_API void fillBufferExt (pANTLR3_COMMON_TOKEN_STREAM tokenStream)
  • References: fillBuffer()

💖 get 💖

pANTLR3_COMMON_TOKEN get (pANTLR3_TOKEN_STREAM ts, ANTLR3_UINT32 i) [static]
  • References: ANTLR3_VECTOR_struct::get, ANTLR3_TOKEN_STREAM_struct::super, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens

🎀 getSourceName 🎀

pANTLR3_STRING getSourceName (pANTLR3_INT_STREAM is) [static]

Hands you back a string that represents the name assoicated with the input source. (Yes, "assoicated" — that's the original spelling, bestie, we don't fix the vibes of history 💀)

/param[in] is — The ANTLR3_INT_STREAM interface that is representing this token stream.

/returns /implements ANTLR3_INT_STREAM_struct::getSourceName()

  • References: ANTLR3_INT_STREAM_struct::super
  • Referenced by: antlr3CommonTokenStreamNew()

🦋 getTokenRange 🦋

pANTLR3_LIST getTokenRange (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop) [static]
  • References: ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensSet
  • Referenced by: antlr3CommonTokenStreamNew()

🌸 getTokens 🌸

pANTLR3_VECTOR getTokens (pANTLR3_COMMON_TOKEN_STREAM cts) [static]
  • References: fillBuffer(), ANTLR3_COMMON_TOKEN_STREAM_struct::p, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens
  • Referenced by: antlr3CommonTokenStreamNew()

💜 getTokensList 💜

pANTLR3_LIST getTokensList (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list) [static]
  • References: antlr3BitsetList(), ANTLR3_BITSET_struct::free, ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensSet, ANTLR3_LIST_struct::table
  • Referenced by: antlr3CommonTokenStreamNew()

🍓 getTokenSource 🍓

pANTLR3_TOKEN_SOURCE getTokenSource (pANTLR3_TOKEN_STREAM ts) [static]
  • References: ANTLR3_TOKEN_STREAM_struct::tokenSource
  • Referenced by: antlr3CommonTokenStreamNew()

🧚 getTokensSet 🧚

pANTLR3_LIST getTokensSet (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types) [static]

Give it a start and stop index and it returns a List of all tokens sitting in the token type BitSet. If nothing turned up, you get null back. And the slay detail: this one peeks at both on and off channel tokens 👀

  • References: ANTLR3_TRUE, antlr3ListNew(), fillBuffer(), ANTLR3_LIST_struct::free, ANTLR3_TOKEN_STREAM_struct::get, ANTLR3_COMMON_TOKEN_struct::getType, ANTLR3_BITSET_struct::isMember, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_COMMON_TOKEN_STREAM_struct::p, ANTLR3_LIST_struct::put, ANTLR3_INT_STREAM_struct::size, ANTLR3_LIST_struct::size, ANTLR3_COMMON_TOKEN_STREAM_struct::tstream
  • Referenced by: antlr3CommonTokenStreamNew()

👑 getTokensType 👑

pANTLR3_LIST getTokensType (pANTLR3_COMMON_TOKEN_STREAM cts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type) [static]
  • References: antlr3BitsetOf(), ANTLR3_BITSET_struct::free, ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensSet
  • Referenced by: antlr3CommonTokenStreamNew()

✨ LB ✨

pANTLR3_COMMON_TOKEN LB (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_INT32 i) [static]
  • References: ANTLR3_VECTOR_ELEMENT_struct::element, ANTLR3_VECTOR_struct::elements, fillBuffer(), ANTLR3_COMMON_TOKEN_STREAM_struct::p, skipOffTokenChannelsReverse(), ANTLR3_COMMON_TOKEN_STREAM_struct::tokens
  • Referenced by: tokLT()

🔥 mark 🔥

ANTLR3_MARKER mark (pANTLR3_INT_STREAM is) [static]
  • References: ANTLR3_INT_STREAM_struct::index, ANTLR3_INT_STREAM_struct::lastMarker

🌈 release 🌈

void release (pANTLR3_INT_STREAM is, ANTLR3_MARKER mark) [static]
  • References: mark()

💖 reset 💖

void reset (pANTLR3_COMMON_TOKEN_STREAM cts) [static]
  • References: ANTLR3_FALSE, ANTLR3_TOKEN_DEFAULT_CHANNEL, antlr3VectorNew(), ANTLR3_COMMON_TOKEN_STREAM_struct::channel, ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverrides, ANTLR3_VECTOR_struct::clear, ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannel, ANTLR3_COMMON_TOKEN_STREAM_struct::discardSet, ANTLR3_LIST_struct::free, ANTLR3_COMMON_TOKEN_STREAM_struct::p, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens

🎀 rewindLast 🎀

void rewindLast (pANTLR3_INT_STREAM is) [static]
  • References: ANTLR3_INT_STREAM_struct::lastMarker, ANTLR3_INT_STREAM_struct::rewind

🦋 rewindStream 🦋

void rewindStream (pANTLR3_INT_STREAM is, ANTLR3_MARKER marker) [static]
  • References: ANTLR3_INT_STREAM_struct::seek
  • Referenced by: antlr3CommonTokenStreamNew()

🌸 seek 🌸

void seek (pANTLR3_INT_STREAM is, ANTLR3_MARKER index) [static]
  • References: ANTLR3_COMMON_TOKEN_STREAM_struct::p, ANTLR3_INT_STREAM_struct::super, ANTLR3_TOKEN_STREAM_struct::super
  • Referenced by: dbgSeek()

💜 setDebugListener 💜

void setDebugListener (pANTLR3_TOKEN_STREAM ts, pANTLR3_DEBUG_EVENT_LISTENER debugger) [static]
  • References: ANTLR3_INT_STREAM_struct::_LA, ANTLR3_TOKEN_STREAM_struct::_LT, ANTLR3_FALSE, ANTLR3_INT_STREAM_struct::consume, dbgConsume(), dbgLA(), dbgMark(), dbgRewindLast(), dbgRewindStream(), dbgSeek(), dbgTokLT(), ANTLR3_TOKEN_STREAM_struct::debugger, ANTLR3_TOKEN_STREAM_struct::initialStreamState, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_INT_STREAM_struct::mark, ANTLR3_INT_STREAM_struct::rewind, ANTLR3_INT_STREAM_struct::rewindLast, ANTLR3_INT_STREAM_struct::seek

🍓 setTokenSource 🍓

void setTokenSource (pANTLR3_TOKEN_STREAM ts, pANTLR3_TOKEN_SOURCE tokenSource) [static]
  • References: ANTLR3_TOKEN_STREAM_struct::tokenSource
  • Referenced by: antlr3CommonTokenStreamNew()

🧚 setTokenTypeChannel 🧚

void setTokenTypeChannel (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel) [static]

A lil filter mechanism where you get to tell this token stream to force all tokens of type ttype onto a specific channel. Example time: when interpreting, we can't execute actions, so we need to tell the stream to force all WS and NEWLINE onto a different, ignored, channel 🫶

  • References: ANTLR3_FUNC_PTR, antlr3ListNew(), ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverrides, ANTLR3_LIST_struct::put
  • Referenced by: antlr3CommonTokenStreamNew()

👑 size 👑

ANTLR3_UINT32 size (pANTLR3_INT_STREAM is) [static]
  • References: ANTLR3_INT_STREAM_struct::cachedSize, ANTLR3_VECTOR_struct::count, ANTLR3_INT_STREAM_struct::super, ANTLR3_TOKEN_STREAM_struct::super, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens

✨ skipOffTokenChannels ✨

ANTLR3_UINT32 skipOffTokenChannels (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_INT32 i) [static]

Hand it a starting index and it hands you back the index of the first on-channel token. Very "skip the noise, find the main character" of it 💅

  • References: ANTLR3_INT_STREAM_struct::cachedSize, ANTLR3_COMMON_TOKEN_STREAM_struct::channel, ANTLR3_COMMON_TOKEN_struct::channel, ANTLR3_VECTOR_ELEMENT_struct::element, ANTLR3_VECTOR_struct::elements, ANTLR3_TOKEN_STREAM_struct::istream, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens, ANTLR3_COMMON_TOKEN_STREAM_struct::tstream
  • Referenced by: consume(), fillBuffer(), tokLT()

🔥 skipOffTokenChannelsReverse 🔥

ANTLR3_UINT32 skipOffTokenChannelsReverse (pANTLR3_COMMON_TOKEN_STREAM tokenStream, ANTLR3_INT32 i) [static]
  • References: ANTLR3_COMMON_TOKEN_STREAM_struct::channel, ANTLR3_COMMON_TOKEN_struct::channel, ANTLR3_VECTOR_ELEMENT_struct::element, ANTLR3_VECTOR_struct::elements, ANTLR3_COMMON_TOKEN_STREAM_struct::tokens
  • Referenced by: LB()

ANTLR3_MARKER tindex (pANTLR3_INT_STREAM is)</p> [static] 💅

okay so this one is a lil internal helper, bestie — it's marked [static] so it's just doing its thing behind the scenes, no cap.

it touches these:

  • ANTLR3_COMMON_TOKEN_STREAM_struct::p
  • ANTLR3_INT_STREAM_struct::super
  • ANTLR3_TOKEN_STREAM_struct::super

pANTLR3_COMMON_TOKEN tokLT (pANTLR3_TOKEN_STREAM ts, ANTLR3_INT32 k)</p> [static] 👑

this is the lookahead girlie. she grabs you the ith token counting from wherever you're standing right now, in the range 1..n, where k=1 is literally the very first symbol of lookahead. so k=1 = the next thing, k=2 = the thing after that, etc etc. main character energy fr fr ✨

she's reaching into all of this:

  • ANTLR3_INT_STREAM_struct::cachedSize
  • ANTLR3_VECTOR_ELEMENT_struct::element
  • ANTLR3_VECTOR_struct::elements
  • ANTLR3_TOKEN_SOURCE_struct::eofToken
  • fillBuffer()
  • ANTLR3_INT_STREAM_struct::index
  • ANTLR3_TOKEN_STREAM_struct::istream
  • LB()
  • ANTLR3_COMMON_TOKEN_STREAM_struct::p
  • ANTLR3_COMMON_TOKEN_struct::setStartIndex
  • ANTLR3_COMMON_TOKEN_struct::setStopIndex
  • skipOffTokenChannels()
  • ANTLR3_TOKEN_STREAM_struct::super
  • ANTLR3_COMMON_TOKEN_STREAM_struct::tokens
  • ANTLR3_TOKEN_STREAM_struct::tokenSource

and the girlies who call her? iconic trio:

  • antlr3CommonTokenStreamNew()
  • dbgLA()
  • dbgTokLT()

pANTLR3_STRING toString (pANTLR3_TOKEN_STREAM ts)</p> [static] 🎀

she turns the whole stream into a string. that's it, that's the vibe. obsessed.

her little reference circle:

  • fillBuffer()
  • ANTLR3_TOKEN_STREAM_struct::istream
  • ANTLR3_COMMON_TOKEN_STREAM_struct::p
  • ANTLR3_INT_STREAM_struct::size
  • ANTLR3_TOKEN_STREAM_struct::super
  • ANTLR3_TOKEN_STREAM_struct::toStringSS

pANTLR3_STRING toStringSS (pANTLR3_TOKEN_STREAM ts, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)</p> [static] 🦋

the "string between two spots" queen 🫶 you hand her a start and a stop (ANTLR3_UINT32 both, keep them exact bestie) and she gives you back the text in between. periodt.

things she's leaning on:

  • fillBuffer()
  • ANTLR3_TOKEN_STREAM_struct::get
  • ANTLR3_COMMON_TOKEN_struct::getText
  • ANTLR3_TOKEN_STREAM_struct::getTokenSource
  • ANTLR3_TOKEN_STREAM_struct::istream
  • ANTLR3_STRING_FACTORY_struct::newRaw
  • ANTLR3_COMMON_TOKEN_STREAM_struct::p
  • ANTLR3_INT_STREAM_struct::size
  • ANTLR3_TOKEN_SOURCE_struct::strFactory
  • ANTLR3_TOKEN_STREAM_struct::super
  • ANTLR3_COMMON_TOKEN_STREAM_struct::tokens

pANTLR3_STRING toStringTT (pANTLR3_TOKEN_STREAM ts, pANTLR3_COMMON_TOKEN start, pANTLR3_COMMON_TOKEN stop)</p> [static] 🌸

same energy as above but this time you're passing actual tokens instead of indexes — a start token and a stop token. so slay.

she only needs these two:

  • ANTLR3_COMMON_TOKEN_struct::getTokenIndex
  • ANTLR3_TOKEN_STREAM_struct::toStringSS

and she gets called by:

  • antlr3CommonTokenStreamNew()

Author ✨

Generated automatically by Doxygen for ANTLR3C from the source code. so like… not a human author, a robot wrote this and honestly? she ate. 💀

okay bestie that's the whole page, we made it 🥹 this was Version 3.3.1, dated Tue Jul 29 2025 00:00:00, straight out of src/antlr3tokenstream.c(3) — go forth and tokenize, i'm so proud of you, mwah 💖🌈⭐

📦 where this page comes from