KinoSe...enizer(3pm)

💖 NAME 💖

okay bestie, meet KinoSearch1::Analysis::Tokenizer — it's the customizable tokenizing era, and it's giving main character energy ✨ (this one's a 3pm page from the User Contributed Perl Documentation stack, iykyk 🫶)

---

🌈 SYNOPSIS 🌈

    my $whitespace_tokenizer
        = KinoSearch1::Analysis::Tokenizer->new( token_re => qr/\S+/, );

    # or...
    my $word_char_tokenizer
        = KinoSearch1::Analysis::Tokenizer->new( token_re => qr/\w+/, );

    # or...
    my $apostrophising_tokenizer = KinoSearch1::Analysis::Tokenizer->new;

    # then... once you have a tokenizer, put it into a PolyAnalyzer
    my $polyanalyzer = KinoSearch1::Analysis::PolyAnalyzer->new(
        analyzers => [ $lc_normalizer, $word_char_tokenizer, $stemmer ], );

like. look at her. three whole vibes and then she slides into a PolyAnalyzer 💅 iconic behavior, no notes 🎀

---

🦋 DESCRIPTION 🦋

so generically speaking (as one does), "tokenizing" is literally just the process of breaking up a string into an array of "tokens". that's it. that's the whole job. she chops. she divides. she slays 🔥

    # before:
    my $string = "three blind mice";

    # after:
    @tokens = qw( three blind mice );

KinoSearch1::Analysis::Tokenizer decides where it should break up the text based on the value of token_re. she's not guessing, bestie, she's going off the regex you handed her 👑

    # before:
    my $string = "Eats, Shoots and Leaves.";

    # tokenized by $whitespace_tokenizer
    @tokens = qw( Eats, Shoots and Leaves. );

    # tokenized by $word_char_tokenizer
    @tokens = qw( Eats Shoots and Leaves   );

see how the punctuation just... stays or goes depending on which tokenizer you picked? the way that one single regex changes everything 😭 it's giving customization.

---

✨ METHODS ✨

🎀 new

    # match "O'Henry" as well as "Henry" and "it's" as well as "it"
    my $token_re = qr/
            \b        # start with a word boundary
            \w+       # Match word chars.
            (?:       # Group, but don't capture...
               '\w+   # ... an apostrophe plus word chars.
            )?        # Matching the apostrophe group is optional.
            \b        # end with a word boundary
        /xsm;
    my $tokenizer = KinoSearch1::Analysis::Tokenizer->new(
        token_re => $token_re, # default: what you see above
    );

okay so this is the Constructor 👑 she takes one hash style parameter. one. singular. unbothered. lowkey minimal 💜

• token_re — has to be a pre-compiled regular expression that matches one token. that's the whole requirement, fr fr. and if you don't pass one? the default is literally the regex you see right up there in the example, the one that catches "O'Henry" as well as "Henry" and "it's" as well as "it" 🙈

ngl bestie, don't slack on the "pre-compiled" part. it must be a pre-compiled regular expression matching one token — that's the rule, and you're gonna respect it 💅

---

🌸 COPYRIGHT 🌸

Copyright 2005-2010 Marvin Humphrey 🫶 (yes, the whole era, we love a long-running legend)

---

💀 LICENSE, DISCLAIMER, BUGS, etc. 💀

See KinoSearch1 version 1.01. that's where the lore lives, go read it 🦋

---

and that's the whole page, bestie!! ✨🫶 this absolute gem of a doc is from perl v5.42.3, dated 2026-08-20, under the KinoSe...enizer(3pm) name — go tokenize something iconic, you got this 💖🎀🌈