Plucen...:Token(3pm)

💖 NAME 💖

Plucene::Analysis::Token — it's a term in a field, bestie. That's the whole vibe. 👑

---

✨ SYNOPSIS ✨

Okay so... this section is literally empty in the original. Like, nothing. A blank canvas. An empty group chat. 🫶 So there's nothing to show you here — the real tea is all down in METHODS, keep scrolling babes.

---

🌸 DESCRIPTION 🌸

Okay so picture this: a Token is one single occurence of a term from the text of a field. One. Singular. Main character energy, no cap. 💅

And what is this little icon actually made of? Three things, periodt:

• the term's text itself
• the start and end offset of the term in the text of the field
• a type string

Now the start and end offsets? They're not just decorative, they're giving ✨purpose✨. They let applications re-associate a token with its source text. Like, for example:

• to display highlighted query terms in a document browser 🦋
• or to show matching text fragments in a KWIC (KeyWord In Context) display
• etc. (yes the original says etc., we love an open-ended queen)

Now the type — she's an interned string, assigned by a lexical analyzer (a.k.a. a tokenizer, iykyk), and she names the lexical or syntactic class that the token belongs to. For example? An end of sentence marker token might be implemented with type "eos". Cute right?

The default token type is "word". Remember that. It's important. 🎀

---

💅 METHODS 💅

🌈 new

my $token = Plucene::Analysis::Token->new({
        type  => $type,
        text  => $text,
        start => $start,
        end   => $end });

This will create a new Plucene::Analysis::Token object. That's it. That's the method. Go forth and tokenize, bestie 🫶✨

---

Okay and for the lil sign-off: this whole page is part of the User Contributed Perl Documentation, and it was written for perl v5.36.0, dated 2022-12-04. Love that for us. Mwah. 💖🎀🦋