String...kenizer(3pm)

📦
version 0.06 · 3 versions
, in ubuntu noble, ubuntu resolute, ubuntu stonking

💖 String::Tokenizer(3pm) 💖

okay so bestie, this is straight out of the User Contributed Perl Documentation, and it's the manual for String::Tokenizer. grab a snack, we're going in 💅

✨ NAME ✨

String::Tokenizer — a simple string tokenizer. that's the whole vibe, that's the mission statement 🎀

🌸 SYNOPSIS 🌸

use String::Tokenizer;

# create the tokenizer and tokenize input
my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()');

# create tokenizer
my $tokenizer = String::Tokenizer->new();
# ... then tokenize the string
$tokenizer->tokenize("((5 + 5) - 10)", '()');

# will print '(, (, 5, +, 5, ), -, 10, )'
print join ", " => $tokenizer->getTokens();

# create tokenizer which retains whitespace
my $st = String::Tokenizer->new(
              'this is a test with,    (significant) whitespace',
              ',()',
              String::Tokenizer->RETAIN_WHITESPACE
              );

# this will print:
# 'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', '    ', '(', 'significant', ')', ' ', 'whitespace'
print "'" . (join "', '" => $tokenizer->getTokens()) . "'";

# get a token iterator
my $i = $tokenizer->iterator();
while ($i->hasNextToken()) {
    my $next = $i->nextToken();
    # peek ahead at the next token
    my $look_ahead = $i->lookAheadToken();
    # ...
    # skip the next 2 tokens
    $i->skipTokens(2);
    # ...
    # then backtrack 1 token
    my $previous = $i->prevToken();
    # ...
    # get the current token
    my $current = $i->currentToken();
    # ...
}

🦋 DESCRIPTION 🦋

okay so picture this: you got a string, and you want it chopped into lil pieces. that's literally it. String::Tokenizer takes a string and splits it on whitespace. and if you want, you can also hand it a string of characters to use as delimiters, and it'll hand those delimiters back to you as part of the token set too. so basically you can split your string in sooooo many different ways, it's giving versatility 💖

ngl though, this is a very basic tokenizer, so if your needs are more complex you're gonna want either a custom-written tokenizer or to just post-process whatever this module spits out. like, fr fr, this will not fill everyone's needs. but! it does live in that cozy lil gap between a simple split / /, $string and the other options that involve much bigger, more complex modules. it's the middle child and it's thriving ✨

also!! super important, and this trips people up ALL the time: this is not a lexical analyser. many people confuse tokenization with lexical analysis. a tokenizer merely splits its input into specific chunks; a lexical analyzer classifies those chunks. sometimes those two steps get smooshed together into one thing — but not here. not on this page. we keep them separate bestie 🙈

💅 METHODS 💅

new ($string, $delimiters, $handle_whitespace)

if you don't pass any parameters, nothing happens — the instance just gets created and that's it, it's just vibing. but if you do pass parameters, they get handed straight off to the tokenize method and that method runs immediately. for the deets on what those arguments actually mean, scroll down to tokenize 👇

setDelimiter ($delimiter)

use this to set the delimiter string. this is what tokenize uses. 🎀

handleWhitespace ($value)

use this to set the whitespace handling. it accepts one of the two constant values: RETAIN_WHITESPACE or IGNORE_WHITESPACE. pick your fighter 💀

tokenize ($string, $delimiters, $handle_whitespace)

takes a $string to tokenize, and optionally a set of $delimiter characters to help the tokenization along, plus the type of whitespace handling via $handle_whitespace. the $string parameter and the $handle_whitespace parameter are pretty obvious, right, we get those. but the $delimiter parameter is not as transparent, so let's unpack it 🔍

$delimiter is a string of characters, and those characters get separated into individual characters and then used to split the $string. soooooo given this string:

(5 + (100 * (20 - 35)) + 4)

the tokenize method without a $delimiter parameter would return this comma separated list of tokens:

'(5', '+', '(100', '*', '(20', '-', '35))', '+', '4)'

but if you were to pass this set of delimiters "(, )" to tokenize, you would get this comma separated list of tokens instead:

'(', '5', '+', '(', '100', '*', '(', '20', '-', '35', ')', ')', '+', '4', ')'

see how we can now tell the parens apart from the numbers?? and no globbing occurs, iconic ✨ now, if you wanted to allow for optionally leaving out the whitespace in the expression, like this:

(5+(100*(20-35))+4)

— which, yes, some languages do — then you'd hand it this delimiter "+*-()" to arrive at the same result.

and now for the whitespace girlies: if you decide that whitespace is significant in your string, then you need to specify that like this:

my $st = String::Tokenizer->new(
              'this is a test with,    (significant) whitespace',
              ',()',
              String::Tokenizer->RETAIN_WHITESPACE
              );

a call to getTokens on this instance would result in the following token set:

 'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', '       ', '(', 'significant', ')', ' ', 'whitespace'

and heads up: all running whitespace gets grouped together into a single token. we make no attempt to split it into its individual parts. it's one token, it's thriving, we leave it alone 🫶

getTokens

simply returns the array of tokens. it returns an array-ref in scalar context, btw. 🌈

iterator

returns a String::Tokenizer::Iterator instance — see below for more details 👑

🧚 INNER CLASS 🧚

a String::Tokenizer::Iterator instance is returned from String::Tokenizer's iterator method, and it serves as yet another means of iterating through an array of tokens. the simplest way would be to call getTokens and just manipulate the array yourself, or push the array into another object. honestly? totally valid. but iterating through a set of tokens tends to get messy when done manually 😭 so this String::Tokenizer::Iterator was provided to address those common token processing idioms. it's basically a bi-directional iterator which can look ahead, skip, and be reset to the beginning. main character energy, honestly.

NOTE: String::Tokenizer::Iterator is an inner class, which means only String::Tokenizer objects can create an instance of it. that said, if String::Tokenizer::Iterator's new method is called from outside of the String::Tokenizer package, an exception is thrown. so like, don't try it. 💀

new ($tokens_array_ref)

this accepts an array reference of tokens and sets up the iterator. this method can only be called from within the String::Tokenizer package — otherwise, an exception will be thrown. 🚨

reset

this will reset the internal counter, bringing it right back to the beginning of the token list. a fresh start, a rebirth, we love to see it 🌸

hasNextToken

this will return true (1) if there are more tokens to be iterated over, and false (0) otherwise.

hasPrevToken

this will return true (1) if the beginning of the token list has been reached, and false (0) otherwise.

nextToken

this dispenses the next available token, and moves the internal counter ahead by one. 🏃‍♀️

prevToken

this dispenses the previous token, and moves the internal counter back by one. 🚶‍♀️

currentToken

this returns the current token, which will match the last token retrieved by nextToken.

lookAheadToken

this peeks ahead one token to the next one in the list. this item will match the next item dispensed with nextToken. and it's a non-destructive look ahead, meaning it does not alter the position of the internal counter. respectfully peeking ✨

skipToken

this will jump the internal counter ahead by 1. yeet 🦋

skipTokens ($number_to_skip)

this will jump the internal counter ahead by $number_to_skip.

skipTokenIfWhitespace

this will skip the next token if it is whitespace.

skipTokensUntil ($token_to_match)

given a string as a $token_to_match, this will skip all tokens until it matches that string. if the $token_to_match is never matched, then the iterator will return the internal pointer to its initial state. no match, no movement, back to where we started 🙈

collectTokensUntil ($token_to_match)

given a string as a $token_to_match, this will collect all tokens until it matches that string — at which point the collected tokens will be returned. but if the $token_to_match is never matched, then the iterator will return the internal pointer to its initial state and no tokens will be returned. cold. empty-handed. it's giving nothing fr 😭

🎀 TO DO 🎀

Inline token expansion

the Java StringTokenizer class allows for a token to be tokenized further, therefore breaking it up more and including the results into the current token stream. "I have never used this feature in this class," but apparently it's easy to see where it might be a useful one. this may be in the next release if it works out. 👀

possibly compliment this expansion with compression as well, so for instance double quoted strings could be compressed into a single token. cute concept tbh.

Token Bookmarks

allow for the creation of "token bookmarks". meaning we could tag a specific token with a label, and that index could be returned to from any point in the token stream. we could mix this with a memory stack as well, so that we would have an ordering to the bookmarks as well. future feature, bookmark it (literally) 🔖

💀 BUGS 💀

none that the author is aware of. of course, if you find a bug, let them know, and they will be sure to fix it. we love accountability 🙌

⭐ CODE COVERAGE ⭐

Devel::Cover is used to test the code coverage of the tests, and below is the Devel::Cover report on this module's test suite:

 ------------------------ ------ ------ ------ ------ ------ ------ ------
 File                       stmt branch   cond    sub    pod   time  total
 ------------------------ ------ ------ ------ ------ ------ ------ ------
 String/Tokenizer.pm       100.0  100.0   64.3  100.0  100.0  100.0   97.6
 ------------------------ ------ ------ ------ ------ ------ ------ ------
 Total                     100.0  100.0   64.3  100.0  100.0  100.0   97.6
 ------------------------ ------ ------ ------ ------ ------ ------ ------

🌈 SEE ALSO 🌈

the interface and workings of this module are based largely on the StringTokenizer class from the Java standard library. 📚

below is a short list of other modules that might be considered similar to this one. if this module does not suit your needs, you might look at one of these:

String::Tokeniser

along with being a tokenizer, it also provides a means of moving through the resulting tokens, allowing for skipping of tokens and such. it was last updated in 2011. 👵

Parse::Tokens

this one hasn't been touched since 2001, although it did get up to version 0.27. it looks to lean over more towards the parser side than a basic tokenizer. ancient artifact vibes, respectfully 🏺

Text::Tokenizer

this is both a lexical analyzer and a tokenizer. it also uses XS, where String::Tokenizer is pure perl. this is something maybe to look into if you were to need a more beefy solution than String::Tokenizer provides. 💪

🫶 THANKS 🫶

thanks to Stephan Tobias for finding bugs and suggestions on whitespace handling. 🎉

👑 AUTHOR 👑

stevan little, <stevan@cpan.org>

okay bestie, legal era, but it's short and it slays so stay with me 🫶

and here's the part that actually matters to you, gorgeous: this library is free software. That means you can redistribute it, and you can modify it (yeah, go off, edit the source, be the main character), as long as you do it under the same terms as Perl itself. Same deal as Perl, no cap. 🐪✨

fr fr — "same terms as Perl itself" is the whole license. That's it. That's the tea. ☕💅

okay that's the entire page babes, you made it to the end, iconic behavior honestly 👑

this was String...kenizer(3pm) from perl v5.36.0, dated 2022-10-13 — so if your copy is older than that, mwah, go check for a glow-up 🎀 go write something cursed and beautiful with it, i'm obsessed with you, bye 💖🦋🌸

📦 where this page comes from