Plucen...enizer(3pm)

💖 Plucene::Analysis::CharTokenizer — the vibe 💖

okay bestie, pull up a chair, we're looking at Plucene::Analysis::CharTokenizer, part of the Plucene::Analysis family, living over in the User Contributed Perl Documentation corner of your 3pm man pages 🫶. She's a base class, she's abstract, she's iconic, and she is not here to do the work herself. Let's get into it ✨

---

🌸 NAME 🌸

Plucene::Analysis::CharTokenizer — the base class for character tokenisers, no cap 👑

---

✨ SYNOPSIS ✨

# isa Plucene::Analysis::Tokenizer

my $next = $chartokenizer->next;

that's it, that's the whole vibe. she inherits, she hands you the next thing, periodt 💅

---

🦋 DESCRIPTION 🦋

So like… this is an abstract base class for simple, character-oriented tokenizers. Abstract means you don't just instantiate her and expect a whole personality — she's the blueprint, the main character energy that subclasses build on 🌈. She tokenizes, but character by character, and she keeps it simple.

---

🎀 METHODS 🎀

💜 token_re

This one should be defined in subclasses. Like, she's leaving it up to you, bestie — that's your homework, not hers 🙈

💜 normalize

This will normalise the character before it is added to the token. Little glow-up for every char on the way in ✨

💜 next

my $next = $chartokenizer->next;

This will return the next token in the string, or undef at the end of the string. So when you hit undef, that's the end, that's the finale, the credits are rolling 😭

---

fr fr: this page is short because she's a base class — the actual lore lives in her subclasses. Don't come for her, she's doing her job 🫶

---

and that's a wrap on the Plucene::Analysis::CharTokenizer(3pm) era, written by the User Contributed Perl Documentation girlies, from perl v5.36.0, dated 2022-12-04 💖 bye bestie, go tokenize something 🎀✨