XTOKID(1)

💖 XTOKID 💖

okay bestie, welcome to xtokid, a user command, and honestly? she's giving very specific tool energy and I'm obsessed. no cap.

✨ NAME ✨

xtokid — print all tokens found in a source file 🎀

🌸 SYNOPSIS 🌸

xtokid [OPTION]... [FILE]...

🦋 DESCRIPTION 🦋

Print all tokens found in a source file. that's it, that's the whole job, and she does it flawlessly 💅

the options are like, the whole personality here, so let's go one by one. and yeah, some of these flags get reused down in the language-specific sections, so read the section headers, bestie — context is everything 🫶

• -i, --include=LANGS — include the languages in LANGS (default: "C C++ asm") ⭐
• -x, --exclude=LANGS — exclude the languages in LANGS 🙈
• -l, --lang-option=L:OPT — pass OPT as a default for language L (see below, okay? the below is important, don't skip it 👀)
• -m, --lang-map=MAPFILE — use MAPFILE to map file names onto source language 🗺️
• -d, --default-lang=LANG — make LANG the default source language 👑
• -p, --prune=NAMES — exclude the named files and/or directories ✂️
• --files0-from=F — tokenize only the files specified by NUL-terminated names in file F 💀 (NUL-terminated, not newline. respect the NUL.)
• --help — display this help and exit 🆘
• --version — output version information and exit 🌈

---

The following arguments apply to the language-specific scanners: 🧚

🌟 C language: 🌟

• -k, --keep=CHARS — allow CHARS in single-token strings, keep the result
• -i, --ignore=CHARS — allow CHARS in single-token strings, toss the result
• -u, --strip-underscore — strip a leading underscore from single-token strings

💜 C++ language: 💜

• -k, --keep=CHARS — allow CHARS in single-token strings, keep the result
• -i, --ignore=CHARS — allow CHARS in single-token strings, toss the result
• -u, --strip-underscore — strip a leading underscore from single-token strings

🍓 Java language: 🍓

• -k, --keep=CHARS — allow CHARS in single-token strings, keep the result
• -i, --ignore=CHARS — allow CHARS in single-token strings, toss the result
• -u, --strip-underscore — strip a leading underscore from single-token strings

🔥 Assembly language: 🔥

• -c, --comment=CHARS — any of CHARS starts a comment until end-of-line
• -k, --keep=CHARS — allow CHARS in tokens, and keep the result
• -i, --ignore=CHARS — allow CHARS in tokens, and toss the result
• -u, --strip-underscore — strip a leading underscore from tokens
• -n, --no-cpp — don't handle C pre-processor directives

🎀 Text language: 🎀

• -i, --include=CHAR-CLASS — treat characters of CHAR-CLASS as token constituents
• -x, --exclude=CHAR-CLASS — treat characters of CHAR-CLASS as token delimiters

🌈 Perl language: 🌈

• -i, --include=CHAR-CLASS — treat characters of CHAR-CLASS as token constituents
• -x, --exclude=CHAR-CLASS — treat characters of CHAR-CLASS as token delimiters
• -d, --dtags — include documentation tags

✨ Lisp language: ✨

okay so. Lisp language: …and then there's just nothing listed under it. no options. no bullets. nothing. she said "Lisp" and moved on 💀 iconic behaviour honestly, the mystery is the moment.

---

🙈 REPORTING BUGS 🙈

Report bugs to bug-idutils@gnu.org — be nice, they're doing their best, and please don't just send "it broke" with no details, bestie, that's not the vibe 😭

🫶 SEE ALSO 🫶

The full documentation for the id-utils package is maintained as a Texinfo manual. If the info and id-utils packages are properly installed, the command

info 'ID database'

should give you access to the complete manual. go read it, it's lowkey the main character of this whole thing 👑

---

and that's the whole page, bestie!! 💖 this one's from xtokid — 4.6.28-15da, dated May 2020. slay, periodt, go tokenize something 🦋✨