string::token(3tcl)

💖 string::token — bestie edition 💖

okay so like. this is a Tcl manual page, string::token(3tcl), living in the Text and string utilities section. it's giving "regex based iterative lexing" and honestly? iconic. let me explain everything, no cap, nothing skipped. 🎀

---

✨ NAME ✨

string::token — regex based iterative lexing. that's it. that's the whole vibe. 💅

---

🌸 SYNOPSIS 🌸

package require Tcl 8.5 9

package require string::token ?1.1?

package require fileutil

::string token text lex string

::string token file lex path

::string token chomp lex startvar string resultvar

like yes bestie, package require Tcl 8.5 9 — she needs Tcl 8.5, and 9 is fine too, she's flexible 💜. and package require string::token ?1.1? — the ?1.1? is optional, she's chill like that. and don't forget package require fileutil, she borrows from her friend. 🫶

---

🦋 DESCRIPTION 🦋

so basically this package hands you commands for regular expression based lexing — aka tokenization — of strings. she takes a big ol' string and chops it up into tokens. slay.

the complete set of procedures is described below. every single one. no skips, periodt. 👑

🌈 ::string token text lex string

okay so this one takes an ordered dictionary lex that maps regular expressions to labels, and tokenizes the string according to that dictionary. dictionary says jump, the string says how high. 💖

the result of the command is a list of tokens, where each token is a 3-element list of:

• the label
• the start-index in the string
• the end-index in the string

⚠️ the command will throw an error if it is not able to tokenize the whole string. she does not do half-finished work, bestie. it's all or nothing. 😭

🌈 ::string token file lex path

this one is a convenience wrapper around ::string token text above, plus fileutil::cat, so you can tokenize whole files without crying. 💅

⚠️ note that this command loads the file wholly into memory before it even starts processing. so if your file is a monster, this is not the move.

if the file is too large for this mode of operation, you'll need a command directly based on ::string token chomp below. okay? okay. 🫶

🌈 ::string token chomp lex startvar string resultvar

this is the work horse behind ::string token text above. the main character, fr fr. she's exposed to the public so you can write your own lexers — ones that, for example, apply different lexing dictionaries according to some internal state. very custom, very you. ✨

the command takes:

• an ordered dictionary lex mapping regular expressions to labels
• a variable startvar, which indicates where to start lexing in the input string
• a result variable resultvar, to extend

the result of the command is a tri-state numeric code indicating one of:

1. 0 — No token found. 💀
2. 1 — Token found. 🔥
3. 2 — End of string reached. 🌈

note that recognition of a token from lex is started at the character index in startvar.

if a token was recognized (status 1), the command will:

• update the index in startvar to point to the first character of the string past the recognized token, and
• further extend resultvar with a 3-element list containing:
  • the label associated with the regular expression of the token,
  • the start-character-index of the token in string,
  • the end-character-index of the token in string.

neither startvar nor resultvar will be updated if no token is recognized at all. like, zero edits, she simply did not see it. 🙈

💡 note that the regular expressions are applied (tested) in the order they are specified in lex, and the first matching pattern stops the process. because of this it is recommended to specify the patterns to lex with from the most specific to the most general. order matters bestie, always has. 👑

💡 further note that all regex patterns are implicitly prefixed with the constraint escape A to ensure that a match starts exactly at the character index found in startvar. no wandering off. she starts where you told her to start. 🎀

---

💀 BUGS, IDEAS, FEEDBACK 💀

ngl, this document, and the package it describes, will undoubtedly contain bugs and other problems. it's giving humbling, but honestly that's every software ever. 😭

please report such in the category textutil of the Tcllib Trackers at:

http://core.tcl.tk/tcllib/reportlist

please also report any ideas for enhancements you may have for either the package and/or the documentation. we want them. we love them. 🫶

when proposing code changes, please provide unified diffs, i.e. the output of diff -u.

⚠️ note further that attachments are strongly preferred over inlined patches. attachments can be made by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar. iykyk. 💅

---

🎀 KEYWORDS 🎀

lexing, regex, string, tokenization

---

🌸 CATEGORY 🌸

Text processing

---

and that's the whole page bestie!! this little cutie is from tcllib, version 1.1, manual page string::token(3tcl). go forth and tokenize responsibly, mwah 💖✨🦋💅