Plucen...enizer(3pm)

📦
version 1.25 · 2 versions
  • 1.25 · debian bookworm, debian trixie, ubuntu noble, ubuntu resolute, ubuntu stonking (latest)
  • 1.25 · ubuntu jammy
, in debian bookworm, debian trixie, ubuntu noble, ubuntu resolute, ubuntu stonking

💖 NAME 💖

Plucene::Analysis::Standard::StandardTokenizer — bestie, it's the standard tokenizer. That's it. That's the whole job. 👑

Oh and btw the page header is being all mysterious and truncating itself like Plucen...enizer(3pm) — that's literally just this same module, she's just shy about her full name. We been knew. 💅

✨ SYNOPSIS ✨

# isa Plucene::Analysis::CharTokenizer

That's the whole vibe. She inherits from Plucene::Analysis::CharTokenizer. Family lineage, slay. 🦋

🌸 DESCRIPTION 🌸

okay so. This is the standard tokenizer. No cap.

And like — if you're working with most European-language documents? She's gonna be a good tokenizer for you. That's the assignment. That's literally what she does. She eats your text and hands you back tokens. Main character energy fr fr. 💖

🎀 METHODS 🎀

🫶 token_re

The regular expression for tokenising. That's the whole thing — it's the regex that decides where one token ends and the next one begins. Iconic, honestly. ✨

🫶 normalize

Removes 's and ..

That's it. Apostrophe-s? Gone. Periods? Gone. She said "not today, punctuation." 💀🙈

ngl bestie there is literally nothing else on this page. No options, no env vars, no exit codes. She's a small queen with a big heart. 🎀

made with love by the User Contributed Perl Documentation crew 💜 · running on perl v5.36.0 · dated 2022-12-04 · mwah, go tokenize something pretty ✨🌈

📦 where this page comes from