Text::T...okenizer(3)

💅 Text::T...okenizer 💅

okay so bestie, welcome to a page from the User Contributed Perl Documentation collection, and the page is literally called Text::T...okenizer(3). yes the title is abbreviated like that. no I don't know why. we move. 🫶

---

✨ NAME ✨

Text::Template::Simple::Tokenizer — the tokenizer, the moment, the main character 🎀

---

🌈 VERSION 🌈

version 0.91 ✨

---

🦋 SYNOPSIS 🦋

use strict;
use warnings;
use Text::Template::Simple::Constants qw( :token );
use Text::Template::Simple::Tokenizer;
my $t = Text::Template::Simple::Tokenizer->new( $start_delim, $end_delim );
foreach my $token ( @{ $t->tokenize( $raw_data ) } ) {
   printf "Token type: %s\n", $token->[TOKEN_ID];
   printf "Token data: %s\n", $token->[TOKEN_STR];
}

like. look at her. building an object, tokenizing the raw data, printing the TOKEN_ID and the TOKEN_STR of every token. that's the whole vibe. slay. 💅

---

💜 DESCRIPTION 💜

so basically it splits the input into tokens using the delimiter pair you defined. that's it. that's the job. she takes your big messy string and hands you back lil token pieces. periodt. ✨

---

💖 NAME 💖

Text::Template::Simple::Tokenizer - "Tokenizer"

(yes bestie, a second NAME section. the POD said so. we don't question the POD. 🙈)

---

👑 METHODS 👑

🧚 new 🧚

the object constructor, the moment of creation, the birth of the tokenizer 💫

she accepts two parameters, and the order matters fr fr:

1. start_delimiter
2. end_delimiter

that's the order. don't flip it. 💀

🍓 tokenize 🍓

splits the input into tokens using the delimiter pair you supplied when you made the object ✂️

she accepts a single parameter: the raw template string. that's it. one (1) argument. no more. she's low maintenance like that. 🎀

🌸 ESCAPE METHODS 🌸

the escape girls. they escape things. iconic behaviour honestly. 🫶

• tilde — escapes the tilde character. that's the ~ bestie. she said not today. ✨
• quote — escapes double quotes. the " ones. she's got them handled too. 💅

---

🌟 AUTHOR 🌟

Burak Gursoy <burak@cpan.org> 👑

---

💜 COPYRIGHT AND LICENSE 💜

This software is copyright (c) 2004 by Burak Gursoy.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. 🎀

---

aaaand that's the page bestie 💖 this was perl v5.42.0, dated 2025-07-25, and the header up top said Text::T...okenizer(3) — go tokenize something cute, ily, bye 🦋✨