Lingua...opWords(3pm)

📦
version 0.12 · 5 versions
, in ubuntu noble, ubuntu resolute, ubuntu stonking

💖 NAME 💖

Lingua::StopWords — stop words for several languages, bestie. That's it, that's the whole vibe. (It's a Perl module, from the User Contributed Perl Documentation corner of the 3pm man pages, and its name is literally Lingua::StopWords, no secret lore.)

✨ SYNOPSIS ✨

use Lingua::StopWords qw( getStopWords );
my $stopwords = getStopWords('en');

my @words = qw( i am the walrus goo goo g'joob );

# prints "walrus goo goo g'joob"
print join ' ', grep { !$stopwords->{$_} } @words;

okay the way this little snippet just filters the Beatles like that?? walrus goo goo g'joob ate and left no crumbs fr fr 🦭✨

🌸 DESCRIPTION 🌸

So in keyword search, it's super common practice to suppress a whole collection of "stopwords" — the "the", "and", "maybe" type words. They show up in a huge number of documents and honestly tell you nothing important about any document that contains them. Iconic of them to be everywhere and mean nothing, honestly main character energy with zero plot 💅

This module hands you those "stoplists" in several languages. That's the whole job. No cap.

🦋 Supported Languages 🦋

|-----------------------------------------------------------|
| Language   | ISO code | default encoding | also available |
|-----------------------------------------------------------|
| Danish     | da       | ISO-8859-1       | UTF-8          |
| Dutch      | nl       | ISO-8859-1       | UTF-8          |
| English    | en       | ISO-8859-1       | UTF-8          |
| Finnish    | fi       | ISO-8859-1       | UTF-8          |
| French     | fr       | ISO-8859-1       | UTF-8          |
| German     | de       | ISO-8859-1       | UTF-8          |
| Hungarian  | hu       | ISO-8859-2       | UTF-8          |
| Indonesian | id       | ISO-8859-1       | UTF-8          |
| Italian    | it       | ISO-8859-1       | UTF-8          |
| Norwegian  | no       | ISO-8859-1       | UTF-8          |
| Portuguese | pt       | ISO-8859-1       | UTF-8          |
| Romanian   | ro       | ISO-8859-2       | UTF-8          |
| Spanish    | es       | ISO-8859-1       | UTF-8          |
| Swedish    | sv       | ISO-8859-1       | UTF-8          |
| Russian    | ru       | KOI8-R           | UTF-8          |
|-----------------------------------------------------------|

look at that roster 😭 fifteen languages and every single one of them has a UTF-8 glow-up available. Russian said KOI8-R for default, which is a choice, and we respect it 🇷🇺

👑 FUNCTIONS 👑

🎀 getStopWords 🎀

my $stoplist      = getStopWords('en');
my $utf8_stoplist = getStopWords('en', 'UTF-8');

This is the one function, and she delivers. She retrieves a stoplist in the form of a hashref — where the keys are all the stopwords and the values are all 1. Every single one. 1. Like a group chat where everyone just replies "1" 🫶

$stoplist = {
    and => 1,
    if  => 1,
    # ...
};

so getStopWords() is expecting 1-2 arguments, bestie, and here's the breakdown:

  • the first argument — required, no exceptions 💀 — is an ISO code representing a supported language.
  • if the ISO code cannot be found, getStopWords returns undef. Just undef. Nothing. The void. Don't be that girl, use a real code.

and then:

  • the second argument should be 'UTF-8' if you want the stopwords encoded in UTF-8.
⚠️ and now the important part: the UTF-8 flag will be turned on, so make sure you understand all the implications of that. Not "some". ALL. Do your research before you flip that flag, queen 🙈

🧚 INSTALLATION 🧚

To install this module type the following:

perl Build.PL
./Build
./Build test
./Build install

four commands. that's it. build, build, test, install. periodt ✨

⭐ SEE ALSO ⭐

The stoplists supplied by this module were created as part of the Snowball project (see <http://snowball.tartarus.org>, Lingua::Stem::Snowball).

Also! Lingua::EN::StopWords provides a different stoplist for English. So if you're like "hmm, the English one isn't giving what I need," there's another option out there for you, iykyk 💜

🌈 SOURCE REPOSITORY 🌈

<https://github.com/wollmers/Lingua-StopWords>

go look, go star it, go read the code fr fr 🔥

💜 AUTHOR 💜

Maintained by Helmut Wollmersdorfer <helmut@wollmersdorfer.at> and Marvin Humphrey <marvin at rectangular dot com>. Original author Fabien Potencier, <fabpot at cpan dot org>.

three legends keeping this thing alive, we love to see it 👑

Copyright 2021 Helmut Wollmersdorfer Copyright 2004-2008 Fabien Potencier, Marvin Humphrey

💅 LICENSE 💅

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.3 or, at your option, any later version of Perl 5 you may have available.

and that's a wrap on the Lingua...opWords(3pm) page — rendered for you by perl v5.36.0, dated 2022-10-13, from the User Contributed Perl Documentation crew 🫶✨ bye bestie, go filter some stopwords 💖

📦 where this page comes from