Plucen...enizer(3pm)

📦
version 1.25 · 2 versions
  • 1.25 · debian bookworm, debian trixie, ubuntu noble, ubuntu resolute, ubuntu stonking (latest)
  • 1.25 · ubuntu jammy
, in debian bookworm, debian trixie, ubuntu noble, ubuntu resolute, ubuntu stonking

✨ NAME ✨

Plucene::Analysis::LetterTokenizer — Letter tokenizer 🦋

okay so bestie, welcome to the Plucen...enizer(3pm) manual page! It's User Contributed Perl Documentation, and it's all about that tokenizer life. 💖

🎀 SYNOPSIS 🎀

# isa Plucene::Analysis::CharTokenizer

ngl the synopsis is literally just that comment, no cap. It's giving isa Plucene::Analysis::CharTokenizer energy. 👑

💅 DESCRIPTION 💅

This is the letter tokenizer class, and its main character energy is: divides text at non-letters. Periodt. ✨

⚠️ Note bestie: this does a decent job for most European languages, but it does a terrible job for some Asian languages, where words are not separated by spaces. Not me being dramatic, it's just the vibe. 😭

mwah 💋 that's the whole page! It came from perl v5.36.0, dated 2022-12-04, and yes it's still the Plucen...enizer(3pm) page. bye bestie, stay iconic! 👑🌈

📦 where this page comes from