LaTeXM...:Token(3pm)

🎀 LaTeXM...:Token(3pm) 🎀

from the User Contributed Perl Documentation — okay bestie, let's get into it 🫶

🌸 NAME 🌸

LaTeXML::Core::Token — representation of a Token: a pair of character and category code (catcode); It extends LaTeXML::Common::Object.

---

✨ Exported functions ✨

• $catcode = CC_ESCAPE; — the page gives you this line for the category-code constants, and here's the whole squad:

CC_BEGIN, CC_END, CC_MATH, CC_ALIGN, CC_EOL,
CC_PARAM, CC_SUPER, CC_SUB, CC_IGNORE,
CC_SPACE, CC_LETTER, CC_OTHER, CC_ACTIVE,
CC_COMMENT, CC_INVALID, CC_CS.

tiny note, fr fr: [The last 2 are (apparent) extensions, with catcodes 16 and 17, respectively].

• $token = Token($string,$cc); — creates a LaTeXML::Core::Token with the given content and catcode. The following shorthand versions are also exported for convenience:

T_BEGIN, T_END, T_MATH, T_ALIGN, T_PARAM,
T_SUB, T_SUPER, T_SPACE, T_LETTER($letter),
T_OTHER($char), T_ACTIVE($char),
T_COMMENT($comment), T_CS($cs)

• @tokens = Explode($string); — returns a list of the tokens corresponding to the characters in $string. All tokens have catcode CC_OTHER, except for spaces which have catcode CC_SPACE. periodt 💅
• @tokens = ExplodeText($string); — returns a list of the tokens corresponding to the characters in $string. All (roman) letters have catcode CC_LETTER, all others have catcode CC_OTHER, except for spaces which have catcode CC_SPACE. 🌸
• UnTeX($object, $suppress_linebreaks); — converts $object to a string containing TeX that created it (or could have). Note that this is not necessarily the original TeX code; expansions or other substitutions may have taken place. 👀

Line-breaking of the generated TeX can be explicitly requested or disabled by passing 0 or 1 as the second $suppress_linebreaks argument. The default behavior of line-breaking is controlled by the global State value SUPPRESS_UNTEX_LINEBREAKS. iykyk.

---

💅 Methods 💅

• @tokens = $object->unlist; — return a list of the tokens making up this $object. slay.
• $string = $object->toString; — return a string representing $object.
• $string = $token->getCSName; — return the string or character part of the $token; for the special category codes, returns the standard string (eg. T_BEGIN->getCSName returns "{"). cute.
• $string = $token->getString; — return the string or character part of the $token.
• $code = $token->getCharcode; — return the character code of the character part of the $token, or 256 if it is a control sequence. 👑
• $code = $token->getCatcode; — return the catcode of the $token.

---

💖 AUTHOR 💖

pBruce Miller <bruce.miller@nist.gov>

yes bestie, pBruce is literally what the page says 💀 but no hate, we respect the icon.

---

👑 COPYRIGHT 👑

Public domain software, produced as part of work done by the United States Government & not subject to copyright in the US.

---

mwah 💋 that's the whole page. perl v5.40.1, dated 2025-11-16. periodt. 🦋