FUNCTION::TOKENIZE(3stap)

✨ NAME ✨

function::tokenize — bestie, this one hands you back the next non-empty token in a string 💖

okay so real quick before we dive in: this page lives in the 3stap manual, aka the SystemTap tapset reference. you're in the right place, you're doing amazing, let's go 🦋

---

🦋 SYNOPSIS 🦋

1) tokenize:string(delim:string)

2) tokenize:string(input:string,delim:string)

two whole forms, two whole vibes, pick your fighter 🌸

---

🎀 ARGUMENTS 🎀

• delim — the set of characters that delimit the tokens. like, the lil scissors that decide where one token ends and the next one starts ✂️
• input — the string you wanna tokenize. and here's the cute part: if it's empty, it returns the next non-empty token in the string that got passed in the previous call to tokenize. it remembers. main character memory, honestly 👑

---

💅 DESCRIPTION 💅

🌈 1) the one with just delim

so this version returns the next token in the string that was passed in the previous call to tokenize. if no delimiter is found, the entire remaining input string is * returned. and when there are no more tokens left to give? it returns empty. that's it. that's the whole thing. no cap 💀

🌈 2) the one with input AND delim

this version returns the next non-empty token in the given input string, where the tokens are delimited by characters in the delim string. here's the breakdown bestie, stay with me:

• if the input string is non-empty, it returns the first token.
• if the input string is empty, it returns the next token in the string passed in the previous call to tokenize. (same trick as form 1, she's consistent 💅)
• if no delimiter is found, the entire remaining input string is returned.
• and it returns empty when no more tokens are available. periodt ✨

---

👑 SEE ALSO 👑

• tapset::tokenize(3stap)

---

and that's the whole page, bestie!! 🫶 this lil number came straight outta the SystemTap Tapset Reference, dated September 2026, living its best life in the 3stap manual. go tokenize something iconic today, you ate 💅🎀✨