Lucy::...enizer(3pm)

💖 NAME 💖

Lucy::Analysis::RegexTokenizer — split a string into tokens. That's it, that's the whole job, and bestie it slays at it. ✨

Oh and hey, welcome to the User Contributed Perl Documentation page for Lucy::...enizer(3pm) — yes the header truncates the name all dramatically, it's giving mystery, it's giving unhinged filename energy. 💅

---

🌸 SYNOPSIS 🌸

Okay so here's the whole vibe, copy-paste ready:

    my $whitespace_tokenizer
        = Lucy::Analysis::RegexTokenizer->new( pattern => '\S+' );

    # or...
    my $word_char_tokenizer
        = Lucy::Analysis::RegexTokenizer->new( pattern => '\w+' );

    # or...
    my $apostrophising_tokenizer = Lucy::Analysis::RegexTokenizer->new;

    # Then... once you have a tokenizer, put it into a PolyAnalyzer:
    my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
        analyzers => [ $word_char_tokenizer, $normalizer, $stemmer ], );

Three ways to make one, zero gatekeeping. Pick your fighter, then hand it off to a PolyAnalyzer like the main character it is. 👑

---

🦋 DESCRIPTION 🦋

So generically speaking, "tokenizing" is just the process of breaking up a string into an array of "tokens". No cap. Like, the string "three blind mice" might get tokenized into "three", "blind", "mice". Boom. That's the whole concept. 🎀

Lucy::Analysis::RegexTokenizer decides where it should break up the text based on a regular expression compiled from a supplied pattern matching one token. It's lowkey genius.

Say our source string is…

    "Eats, Shoots and Leaves."

…then a "whitespace tokenizer" with a pattern of \\S+ produces…

    Eats,
    Shoots
    and
    Leaves.

…while a "word character tokenizer" with a pattern of \\w+ produces…

    Eats
    Shoots
    and
    Leaves

…and the difference is literally that the word character tokenizer skips over punctuation as well as whitespace when determining token boundaries. The comma? Gone. The period? Evaporated. Not a single crumb left. 😭✨

---

🎀 CONSTRUCTORS 🎀

✨ new ✨

    my $word_char_tokenizer = Lucy::Analysis::RegexTokenizer->new(
        pattern => '\w+',    # required
    );

Create a new RegexTokenizer. Fresh out the box, glowing, ready to split strings. 💅

• pattern — A string specifying a Perl-syntax regular expression which should match one token. The default value is "\w+(?:[\x{2019}']\w+)*", which matches "it's" as well as "it" and "O'Henry's" as well as "Henry". The way it handles apostrophes?? Iconic. It just gets you. 🫶

---

💜 METHODS 💜

⭐ transform ⭐

    my $inversion = $regex_tokenizer->transform($inversion);

Take a single Inversion as input and returns an Inversion, either the same one (presumably transformed in some way), or a new one. Either way you're getting an Inversion back, that's the deal. 🤝

• inversion — An inversion.

---

🌈 INHERITANCE 🌈

Lucy::Analysis::RegexTokenizer isa Lucy::Analysis::Analyzer isa Clownfish::Obj.

It's a whole family tree bestie, the ancestry ate. 👑

---

Heads up: the synopsis above is doing that thing where it references $normalizer and $stemmer like you already know them — those are just other analyzers you'd bring along, the docs are trusting you here. 🙈

---

okay and that's the page!! 🌟 This was Lucy::...enizer(3pm) from the User Contributed Perl Documentation, and the footer says perl v5.42.0 dated 2025-07-25 — so yeah, freshly timestamped, go tokenize something gorgeous, mwah 💖🌸🦋💅