Lucy::...::Token(3pm)

💖 Lucy::Analysis::Token — the manual, but make it bestie-coded 💖

okay so this is the page for Lucy::...::Token(3pm) from the User Contributed Perl Documentation, and honestly? it's a bop. it's short, it's foundational, it's lowkey the main character of Apache Lucy's analysis stack. let's get into it 🫶

---

✨ NAME ✨

Lucy::Analysis::Token — the Unit of text. like, the unit. the atom. the little baby building block. 🎀

---

✨ SYNOPSIS ✨

here's the vibe, bestie:

my $token = Lucy::Analysis::Token->new(
    text         => 'blind',
    start_offset => 8,
    end_offset   => 13,
);

$token->set_text('mice');

look at her. built different. 👑

---

✨ DESCRIPTION ✨

so Token is the fundamental unit that all of Apache Lucy's Analyzer subclasses work with. it's giving "everything is tokens, the tokens are everything." 💅

every single Token has 5 attributes: text, start_offset, end_offset, boost, and pos_inc. five. not four, not six. slay.

text is a Unicode string encoded as UTF-8. that's it, that's the attribute.

start_offset is where the token text starts, measured in Unicode code points from the top of the stored field, and end_offset marks the matching closing boundary. these two are doing the most, because they keep locating the Token inside a bigger context even if the Token's text attribute gets changed — by stemming, for example. no cap.

here's the iconic example: the Token for “beating” inside the text “beating a dead horse” starts its life with a start_offset of 0 and an end_offset of 7. after stemming, the text becomes “beat”, but the start_offset is still 0 and the end_offset is still 7. the way that lets “beating” get highlighted correctly after a search matches “beat”? obsessed. 🌸

boost is a per-token weight. reach for it when you want to give a particular token more or less importance — like you might for emboldened text inside an HTML document, for example. (⚠️ note: the field this token belongs to must be spec'd to use a posting of type RichPosting.) so, important caveat, don't come for me later.

pos_inc is the POSition INCrement, measured in Tokens. it defaults to 1, and it's a whole advanced tool for manipulating phrase matching, iykyk.

ordinarily, Tokens get consecutive position numbers: 0, 1, and 2 for "three blind mice". but — plot twist — if you set the position increment for “blind” to, say, 1000, then the three tokens end up assigned to positions 0, 1, and 1001, and they will no longer produce a phrase match for the query "three blind mice". bye phrase match 💀

---

✨ CONSTRUCTORS ✨

🌈 new

my $token = Lucy::Analysis::Token->new(
    text         => $text,          # required
    start_offset => $start_offset,  # required
    end_offset   => $end_offset,    # required
    boost        => 1.0,            # optional
    pos_inc      => 1,              # optional
);

• text — a string. 🍓
• start_offset — start offset into the original document, in Unicode code points.
• start_offset — end offset into the original document, in Unicode code points. (yes bestie, it says start_offset again in the original — i am simply reporting the lore, not editing it 🙈)
• boost — per-token weight.
• pos_inc — position increment for phrase matching.

---

✨ METHODS ✨

💜 get_text

my $text = $token->get_text;

get the token's text. she's right there. 💅

💜 set_text

$token->set_text($text);

set the token's text. glow up in one line. ✨

💜 get_start_offset

my $int = $token->get_start_offset();

💜 get_end_offset

my $int = $token->get_end_offset();

💜 get_boost

my $float = $token->get_boost();

💜 get_pos_inc

my $int = $token->get_pos_inc();

💜 get_len

my $int = $token->get_len();

---

✨ INHERITANCE ✨

Lucy::Analysis::Token isa Clownfish::Obj. it's giving family tree, it's giving lineage, it's giving class hierarchy royalty 👑

---

and that's the whole page, bestie!! no cap, fr fr. 💖

this manual page is from perl v5.42.0, dated 2025-07-25, and it lives under Lucy::...::Token(3pm) in the User Contributed Perl Documentation — so go forth, make tokens, and slay 🎀🦋💅