Plucen...enizer(3pm)

💖 Plucene::Analysis::Tokenizer 💖

okay bestie, pull up a chair — this is a page from the User Contributed Perl Documentation, and it's all about Plucene::Analysis::Tokenizer 🎀

---

✨ NAME ✨

Plucene::Analysis::Tokenizer — it's the base class for tokenizers, no cap 💅

---

🌸 SYNOPSIS 🌸

my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
        ->new({ reader => $reader });

---

🦋 DESCRIPTION 🦋

So, real talk: this is an abstract base class for tokenizers. Like, it's the blueprint, not the actual thing you'd instantiate directly. You don't build a house from the idea of a house, you know? 🏠✨

And here's the vibe: a Tokenizer is a TokenStream whose input is a Reader. That's literally the whole relationship — tokenizer goes in, stream comes out, reader feeds it. It's giving pipeline energy 🔥

---

👑 METHODS 👑

💜 new

my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
        ->new({ reader => $reader });

This will create a new tokenizer. Slay. 🌟

💜 next

This one must be defined in a subclass. Not optional, bestie — it's the whole "you gotta do your part" situation 😭 The base class is not gonna do it for you.

💜 close

Close the input reader. That's it, that's the method. Say bye to your reader 🫶

---

🌈 quick reminder: new is fine to call, but next is on you to write in your subclass. The base class will absolutely not cover for you.

---

sign-off time 🎀 This lil page lives in the perl v5.36.0 docs and got dated 2022-12-04 — so go forth, subclass responsibly, and remember: a Tokenizer is a TokenStream whose input is a Reader, periodt 💅💖✨