PPIx::R...okenizer(3)

📦
version 0.092 · 13 versions
, in fedora 43, fedora 44

💖 PPIx::Regexp::Tokenizer — the manual page, but make it cute 💖

okay so bestie, gather round 🌸 this is the manual for PPIx::Regexp::Tokenizer, which lives in the Perl user-contributed docs (that's the PPIx::R...okenizer(3) situation up top, iykyk). It's the thing that tokenizes a regular expression — like, it takes your regex and chops it into lil pieces so the rest of the machinery can vibe with it. 🦋

✨ NAME ✨

PPIx::Regexp::Tokenizer — literally just: Tokenize a regular expression. That's the whole job. That's the assignment. 💅

✨ SYNOPSIS ✨

The way you'd actually touch this thing:

use PPIx::Regexp::Dumper;
PPIx::Regexp::Dumper->new( 'qr{foo}smx' )
    ->print();

short, sweet, iconic ✨

✨ INHERITANCE ✨

Okay so the family tree situation:

  • PPIx::Regexp::Tokenizer is a PPIx::Regexp::Support. 🫶 That's its parent.
  • PPIx::Regexp::Tokenizer has no descendants. It's the end of the line, main character with no sequels. 😭

✨ DESCRIPTION ✨

This class provides the tokenization of the regular expression. That's it, that's the whole glow-up. 🎀

✨ METHODS ✨

This class provides the following public methods. 💖

Real talk though: any method that isn't documented here (or down below under "EXTERNAL TOKENIZERS") is private. And "private" means unsupported in the sense that the author reserves the right to change or remove them without notice. So like… don't build your whole personality on them, bestie. 🚩

🌈 new

my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' );

This is a static method and it instantiates the tokenizer. ✨

You must hand it the regular expression to be parsed — either as a string or as a PPI::Element of some sort. You can also pass optional name/value pairs of arguments. Heads up: the option names are specified without a leading dash. 🎀

Here are the supported options, one by one:

  • default_modifiers array_reference — this argument specifies default statement modifiers. It's optional, but if you specify it, it must be an array reference. For the deets, go look at the PPIx::Regexp new() documentation. 🦋
  • encoding name — this option specifies the encoding of the string to be tokenized. If you set it, an Encode::decode is done on the string (or on the content of the PPI class) before it gets tokenized. 🌸
  • index_locations — this Boolean option says that the locations of the generated tokens are to be computed. 📍
  • strict boolean — this option specifies whether tokenization should assume use re 'strict'; is in effect.
⚠️ The 'strict' pragma was introduced in Perl 5.22, and its documentation says it is experimental, and that there is no commitment to backward compatibility. The same energy applies to the tokenization you get when this option is asserted. So like, it might change on you. 💀
  • trace number — specifying a positive value for this option causes a trace of the tokenization. 🕵️ This option is unsupported in the sense that the author reserves the right to alter it without notice.

If this option is unspecified, the value comes from the environment variable PPIX_REGEXP_TOKENIZER_TRACE (scroll to "ENVIRONMENT VARIABLES" for that whole situation). And if that environment variable doesn't exist either? The default is 0. 🫶

🚨 Undocumented options are unsupported. No cap.

The returned value is the instantiated tokenizer… or undef if instantiation failed. 😭 And if it did fail, a call to errstr will hand you the reason why.

🌈 content

print $tokenizer->content();

This method returns the string being tokenized. If the object was instantiated with a PPI::Element, this will be whatever the PPI::Element->content() method gives you. ✨

🌈 default_modifiers

print join ', ', @{ $tokenizer->default_modifiers() };

Returns a reference to a copy of the array that you passed to the default_modifiers argument of new(). 🎀 And if you never used that argument to instantiate the object? You get a reference to an empty array. Cool cool cool.

🌈 encoding

This method returns the encoding of the data being parsed, if one was set when the class was instantiated. Otherwise it just returns undef. That's it. 🦋

🌈 errstr

my $tokenizer = PPIx::Regexp::Tokenizer->new( 'xyzzy' )
    or die PPIx::Regexp::Tokenizer->errstr();

A static method that returns an error description if tokenizer instantiation failed. Basically the "what went wrong" receipt. 🧾

🌈 failures

print $tokenizer->failures(), " tokenization failures\n";

This returns the number of tokenization failures encountered. And how does a tokenization failure show up in the output token stream? As a PPIx::Regexp::Token::Unknown. 💀

🌈 modifier

$tokenizer->modifier( 'x' )
    and print "Tokenizing an extended regular expression\n";

This method returns true if the given modifier character was found on the end of the regular expression, and false otherwise. ✅❌

Now, the plot twist 🌈 — starting with version 0.036_01: if the argument is a single-character modifier followed by an asterisk (the asterisk is intended as a wild card character), the return is the number of times that modifier appears. But heads up, an exception will be thrown if you specify a multi-character modifier (like 'ee*'), or if you specify one of the match semantics modifiers (like 'a*'). 🚩

And one more thing: if it's called by an external tokenizer, this method returns true if the given modifier was true at the current point in the tokenization. 👑

🌈 next_token

my $token = $tokenizer->next_token();

Returns the next token in the token stream… or nothing if there are no more tokens. Simple as. ✨

🌈 significant

This method exists simply for the convenience of PPIx::Regexp::Dumper. It always returns true. That's the whole method. It's just there being supportive. 🫶

🌈 tokens

my @tokens = $tokenizer->tokens();

Returns all remaining tokens in the token stream. 🎀

okay that's part one bestie, keep scrolling, we're not done yet 💖

💖 EXTERNAL TOKENIZERS 💖

Okay so bestie, plot twist: this class does very little of its own tokenization. Instead, the token classes contain external tokenization routines, whose name is '__PPIX_TOKENIZER__' concatenated with the current mode of the tokenizer ('regexp' for regular expressions, 'repl' for the replacement string).

These external tokenizers are called as static methods, and passed the PPIx::Regexp::Tokenizer object and the current character in the character stream.

If the external tokenizer wants to make one or more tokens, it returns an array containing either length in characters for tokens of the tokenizer's own class, or the results of one or more make_token calls for tokens of an arbitrary class.

If the external tokenizer is not interested in the characters starting at the current position it simply returns.

The following methods are for the use of external tokenizers, and are not part of the public interface to this class.

🎀 capture 🎀

if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
    foreach ( $tokenizer->capture() ) {
        print "$_\n";
    }
}

This method returns all the contents of any capture buffers from the previous call to find_regexp. The first element of the array (i.e. element 0) corresponds to $1, and so on.

The captures are cleared by make_token, as well as by another call to find_regexp.

$tokenizer->cookie( foo => sub { 1 } );
my $cookie = $tokenizer->cookie( 'foo' );
my $old_hint = $tokenizer->cookie( foo => undef );

This method either creates, deletes, or accesses a cookie.

A cookie is a code reference which is called whenever the tokenizer makes a token. If it returns a false value, it is deleted. Explicitly setting the cookie to undef also deletes it.

When you call $tokenizer->cookie( 'foo' ), the current cookie is returned. If you pass a new value of undef to delete the token, the deleted cookie (if any) is returned.

When the make_token method calls a cookie, it passes it the tokenizer and the token just made. If a token calls a cookie, it is recommended that it merely pass the tokenizer, though of course the token can do whatever it wants.

The cookie mechanism seems to be a bit of a crock, but it appeared to be more work to fix things up in the lexer after the tokenizer got something wrong.

The recommended way to write a cookie is to use a closure to store any necessary data, and have a call to the cookie return the data; otherwise the ultimate consumer of the cookie has no way to access the data. Of course, it may be that the presence of the cookie at a certain point in the parse is all that is required.

🌟 expect 🌟

$tokenizer->expect( 'PPIx::Regexp::Token::Code' );

This method inserts a given class at the head of the token scan, for the next iteration only. More than one class can be specified. Class names can be abbreviated by removing the leading 'PPIx::Regexp::'.

If no class is specified, this method does nothing.

The expectation lasts from the next time get_token is called until the next time make_token makes a significant token, or until the next expect call if that is done sooner.

🔍 find_regexp 🔍

my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
my ( $begin, $end ) = $tokenizer->find_regexp(
    qr{ \A \w+ }smx );

This method finds the given regular expression in the content, starting at the current position. If called in scalar context, the offset from the current position to the end of the matched string is returned. If called in list context, the offsets to both the beginning and the end of the matched string are returned.

🧩 find_matching_delimiter 🧩

my $offset = $tokenizer->find_matching_delimiter();

This method is used by tokenizers to find the delimiter matching the character at the current position in the content string. If the delimiter is an opening bracket of some sort, bracket nesting will be taken into account.

When searching for the matching delimiter, the back slash character is considered to escape the following character, so back-slashed delimiters will be ignored. No other quoting mechanisms are recognized, though, so delimiters inside quotes still count. This is actually the way Perl works, as

$ perl -e 'qr<(?{ print "}" })>'

demonstrates.

This method returns the offset from the current position in the content string to the matching delimiter (which will always be positive), or undef if no match can be found.

🌀 get_mode 🌀

This method returns the name of the current mode of the tokenizer.

🎬 get_start_delimiter 🎬

my $start_delimiter = $tokenizer->get_start_delimiter();

This method is used by tokenizers to access the start delimiter for the regular expression.

🎟️ get_token 🎟️

my $token = $tokenizer->make_token( 3 );
my @tokens = $tokenizer->get_token();

This method returns the next token that can be made from the input stream. It is not part of the external interface, but is intended for the use of an external tokenizer which calls it after making and retaining its own token to look at the next token ( if any ) in the input stream.

🚨 Fatal error alert: If any external tokenizer calls get_token without first calling make_token, a fatal error occurs; this is better than the infinite recursion which would occur if the condition were not trapped.

An external tokenizer must return anything returned by get_token; otherwise tokens get lost.

🫧 interpolates 🫧

This method returns true if the top-level structure being tokenized interpolates; that is, if the delimiter is not a single quote.

🏭 make_token 🏭

return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );

This method is used by this class (and possibly by individual tokenizers) to manufacture a token. Its arguments are the number of characters to include in the token, and optionally the class of the token. If no class name is given, the caller's class is used. Class names may be shortened by removing the initial 'PPIx::Regexp::', which will be restored by this method.

The token will be manufactured from the given number of characters starting at the current cursor position, which will be adjusted.

If the given length would include characters past the end of the string being tokenized, the length is reduced appropriately. If this means a token with no characters, nothing is returned.

🪞 match 🪞

if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
    print $tokenizer->match(), "\n";
}

This method returns the string matched by the previous call to find_regexp.

The match is set to undef by make_token, as well as by another call to find_regexp.

🧬 modifier_duplicate 🧬

$tokenizer->modifier_duplicate();

This method duplicates the modifiers on the top of the modifier stack, with the intent of creating a locally-scoped copy of the modifiers. This should only be called by an external tokenizer that is actually creating a modifier scope. In other words, only when creating a PPIx::Regexp::Token::Structure token whose content is '('.

🎛️ modifier_modify 🎛️

$tokenizer->modifier_modify( name => $value ... );

This method sets new values for the modifiers in the local scope. Only the modifiers whose names are actually passed have their values changed.

This method is intended to be called after manufacturing a PPIx::Regexp::Token::Modifier token, and passed the results of its modifiers method.

🫳 modifier_pop 🫳

$tokenizer->modifier_pop();

This method removes the modifiers on the top of the modifier stack. This should only be called by an external tokenizer that is ending a modifier scope. In other words, only when creating a PPIx::Regexp::Token::Structure token whose content is ')'.

Note that this method will never pop the last modifier item off the stack, to guard against unmatched right parentheses.

👀 modifier_seen 👀

$tokenizer->modifier_seen( 'i' )
    and print "/i was seen at some point.\n";

Unlike modifier(), this method returns a true value if the given modifier has been seen in any scope visible from the current location in the parse. There is no magic for group match semantics ( /a, /aa, /d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e and /ee.

👁️ peek 👁️

my $character = $tokenizer->peek();
my $next_char = $tokenizer->peek( 1 );

This method returns the character at the given non-negative offset from the current position. If no offset is given, an offset of 0 is used.

If you ask for a negative offset or an offset off the end of the sting, undef is returned.

📄 ppi_document 📄

This method makes a PPI document out of the remainder of the string, and returns it.

⏮️ prior_significant_token ⏮️

$tokenizer->prior_significant_token( 'can_be_quantified' )
   and print "The prior token can be quantified.\n";

This method calls the named method on the most-recently-instantiated significant token, and returns the result. Any arguments subsequent to the method name will be passed to the method.

Because this method is designed to be used within the tokenizing system, it will die horribly if the named method does not exist.

If called with no arguments at all the most-recently-instantiated significant token is returned.

📏 strict 📏

say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';

This method simply returns true or false, depending on whether the 'strict' option to new() was true or false.

🌈 ENVIRONMENT VARIABLES 🌈

A tokenizer trace can be requested by setting environment variable PPIX_REGEXP_TOKENIZER_TRACE to a numeric value other than 0. Use of this environment variable is unsupported in the same sense that the "trace" option of "new" is unsupported. Explicitly specifying the "trace" option to "new" overrides the environment variable.

The real reason this is documented is to give the user a way to troubleshoot funny output from the tokenizer.

🫶 SUPPORT 🫶

Support is by the author. Please file bug reports at https://github.com/trwyant/perl-PPIx-Regexp/issues or in electronic mail to the author.

👑 AUTHOR 👑

Thomas R. Wyant, III harryfmudd at comcast dot net

Copyright (C) 2009-2023, 2025-2026 by Thomas R. Wyant, III

This program is free software; you can redistribute it and/or modify it under the same terms as Perl 5.10.0. For more details, see the full text of the licenses in the directory LICENSES.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.

And that's the whole vibe, bestie! 💖✨ This page is from perl v5.42.2, dated 2026-06-15, and it's the PPIx::R...okenizer(3) manual. Periodt. 💅🌸

📦 where this page comes from