Lucy::...torial(3pm)

💖 NAME 💖

Lucy::Docs::Tutorial::AnalysisTutorial — like, how do you even choose an Analyzer, and then how do you use it?? bestie we're about to find out ✨

(this whole thing lives in the Lucy::...torial(3pm) manual, for the record 🎀)

---

🌸 DESCRIPTION 🌸

okay so first mission: go swap out the EasyAnalyzer in our Schema for a StandardTokenizer. just. try it. be brave 💅

my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
my $type = Lucy::Plan::FullTextType->new(
    analyzer => $tokenizer,
);

Now search for "senate", "Senate", and "Senator" — do it before AND after you make the change and re-index. the difference is gonna gag you fr fr 😭

Under EasyAnalyzer, all three searches give you identical results. cute, consistent, predictable. but under StandardTokenizer? searches become case-sensitive, and the result sets for "Senate" and "Senator" are distinct. not the same thing anymore, not even close 👑

✨ EasyAnalyzer ✨

so what's actually happening here? EasyAnalyzer is doing WAY more aggressive processing than StandardTokenizer. like, it's not just tokenizing — it's also:

• converting all text to lower case, so searches are case-insensitive (yes ma'am, lowercase everything 💖)
• running a "stemming" algorithm to squish related words down to a common stem ("senat", in this case) 🦋

EasyAnalyzer is actually multiple Analyzers wrapped up in one single package. in this case it's a three-in-one situation, because specifying an EasyAnalyzer with "language => 'en'" is equivalent to this snippet creating a PolyAnalyzer:

my $tokenizer    = Lucy::Analysis::StandardTokenizer->new;
my $normalizer   = Lucy::Analysis::Normalizer->new;
my $stemmer      = Lucy::Analysis::SnowballStemmer->new( language => 'en' );
my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
    analyzers => [ $tokenizer, $normalizer, $stemmer ],
);

and you can add or subtract Analyzers from there if you feel like it, no cap. try adding a fourth Analyzer — a SnowballStopFilter for suppressing "stopwords" like "the", "if", and "maybe" 🙈

my $stopfilter = Lucy::Analysis::SnowballStopFilter->new(
    language => 'en',
);
my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
    analyzers => [ $tokenizer, $normalizer, $stopfilter, $stemmer ],
);

also — try removing the SnowballStemmer. just for the plot 👀

my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
    analyzers => [ $tokenizer, $normalizer ],
);

ngl, the original choice of a stock English EasyAnalyzer probably still yields the best results for this document collection. but you get the idea, bestie: sometimes you just want a different Analyzer. it's giving ✨customization✨

💜 When the best Analyzer is no Analyzer 💜

sometimes you don't want an Analyzer at all. that was true for our "url" field because we didn't need it to be searchable — but it's also true for certain types of searchable fields. for instance, "category" fields are often set up to match exactly or not at all, and so are fields like "last_name" (because you may not want to conflate results for "Humphrey" and "Humphries" — those are two different people, okay?? 😤).

to specify that there should be no analysis performed at all, use StringType:

my $type = Lucy::Plan::StringType->new;
$schema->spec_field( name => 'category', type => $type );

lowkey the vibe here is: sometimes the best glow-up is no glow-up 💅

🌈 Highlighting up next 🌈

in our next tutorial chapter, HighlighterTutorial, we'll add highlighted excerpts from the "content" field to our search results. main character energy incoming, you're not ready 🔥

---

and that's the tea, bestie 🫶 this page came to you from perl v5.42.0, dated 2025-07-25 — go forth and analyze, periodt 💖✨