KinoSe...:Token(3pm)

✨ NAME ✨

KinoSearch1::Analysis::Token — the itty-bitty unit of text, bestie 💖

(and yeah, this is from the User Contributed Perl Documentation for KinoSearch1::Analysis::Token(3pm) — that's the whole title, we're just vibing with it) 💅

---

🦋 SYNOPSIS 🦋

# private class - no public API

That's it. That's the whole thing. She's mysterious, she's exclusive, she's not for you 🙈

---

🎀 PRIVATE CLASS 🎀

Okay so real talk: you literally cannot instantiate a Token object at the Perl level. Not a thing. Not allowed. She's not taking walk-ins 🚫

BUT — and this is the plot twist — you can still affect individual Tokens inside a TokenBatch by way of TokenBatch's (experimental) API. So like, you can't date her directly, but you can absolutely influence the friend group 👑

---

🌸 DESCRIPTION 🌸

Token is the fundamental unit used by KinoSearch1's Analyzer subclasses. The main character of the text-processing world, honestly 👑 Every single Token has 4 attributes: text, start_offset, end_offset, and pos_inc (which stands for position increment, iykyk).

The text of a token is a string. Simple. Elegant. Slay.

A Token's start_offset and end_offset locate it within a larger text — and here's the part that ate: this holds true even if the Token's text attribute gets modified. By stemming, for instance. 🍓

So picture this: the Token for "beating" in the text "beating a dead horse" begins life with a start_offset of 0 and an end_offset of 7. Then stemming happens, the text becomes "beat", but the end_offset is still 7. She kept receipts. The glow-up doesn't erase the origin story 💖

Now, the position increment 👀 — it defaults to 1 — and it's an advanced tool for manipulating phrase matching. Like, this is the graduate-level syllabus, ngl.

Ordinarily, Tokens get assigned consecutive position numbers: 0, 1, and 2 for "three blind mice". Cute, orderly, fine.

However!!! If you set the position increment for "blind" to, say, 1000, then the three tokens will end up assigned to positions 0, 1, and 1001 — and will no longer produce a phrase match for the query '"three blind mice"'. 💀

Periodt. That's the whole trick. You broke the phrase match on purpose and you meant to do it. Main character energy fr fr.

---

💜 COPYRIGHT 💜

Copyright 2006-2010 Marvin Humphrey 🫶

---

🌈 LICENSE, DISCLAIMER, BUGS, etc. 🌈

See KinoSearch1 version 1.01. 🧚

---

okay that's the whole page, you made it, I'm so proud of you 😭💖

this was rendered from perl v5.42.3, dated 2026-08-20 — shoutout to the toolchain, she's doing her best 🌸

bye bestie, go tokenize something iconic ✨👑💅