Lingua::Sentence(3pm)

💖 Lingua::Sentence — the Perl era of sentence splitting 💖

okay bestie, gather round, because this is the Lingua::Sentence manual page (yes, the 3pm one, user-contributed Perl documentation, we love her) and she is so much more chill than she looks. 🎀

---

✨ NAME ✨

Lingua::Sentence — a Perl extension for breaking text paragraphs into sentences. periodt. 💅

---

🌸 SYNOPSIS 🌸

        use Lingua::Sentence;

        my $splitter = Lingua::Sentence->new("en");

        my $text = 'This is a paragraph. It contains several sentences. "But why," you ask?';

        print $splitter->split($text);

like… that's it. that's the whole vibe. three lines and a dream. 🦋

---

🫶 DESCRIPTION 🫶

So this module lets you split text paragraphs into sentences. It's built on scripts that Philipp Koehn and Josh Schroeder made for processing the Europarl corpus (<http://www.statmt.org/europarl/>), so it's giving academically sourced, not just vibes. 👑

It uses punctuation and capitalization clues to chop paragraphs into a newline-separated string with one sentence per line. For example, this:

        This is a paragraph. It contains several sentences. "But why," you ask?

goes to:

        This is a paragraph.
        It contains several sentences.
        "But why," you ask?

literally chef's kiss. no cap. ✨

Languages currently supported by the module are:

• Catalan
• Czech
• Dutch
• English
• French
• German
• Greek
• Hungarian
• Icelandic
• Italian
• Latvian
• Polish
• Portuguese
• Russian
• Spanish
• Slovak
• Slovenian
• Swedish

🎀 Nonbreaking Prefixes Files 🎀

ok so, nonbreaking prefixes are loosely defined as any word ending in a period that does NOT indicate an end of sentence marker. the basic English example is Mr. and Ms. — like, Mr. is not the end of anything, bestie, it's just a guy. 🙈

The sentence splitter module uses the nonbreaking prefix files included in this distribution.

Wanna add a file for other languages? Follow the naming convention nonbreaking_prefix.?? and use the two-letter language code you intend to use when creating a Lingua::Sentence object. 💖

The sentence splitter module will first look for a file for the language it is processing, and fall back to English if a file for that language is not found.

For the splitter, normally a period followed by an uppercase word results in a sentence split. But if the word preceeding the period is a nonbreaking prefix, this line break is not inserted. 👑

A special case of prefixes, NUMERIC_ONLY, is included for special cases where the prefix should be handled ONLY when before numbers. For example, in "Article No. 24 states this." the No. is a nonbreaking prefix. However, in "No. It is not true." No functions as a word. the duality. the range. 💅

See the example prefix files included in the distribution for more examples.

🏆 CREDITS 🏆

Thanks for the following individuals for supplying nonbreaking prefix files: Bas Rozema (Dutch), Hilário Leal Fontes (Portuguese), Jesús Giménez (Catalan & Spanish), Anne-Kathrin Schumann (Russian). icons, all of them. 🌈

🌟 EXPORT 🌟

• new($lang_id) — Instantiate an object to split sentences in language $lang_id. If the language is not supported, a splitter object for English will be instantiated. (English really said "I got you, babe." 😭)
• new($lang_id,$nonbreaking_prefix_file) — Instantiate an object to split sentences in language $lang_id and the nonbreaking prefix file $nonbreaking_prefix_file. If the file does not exist, a splitter object for English will be instantiated. same energy, different file. 🫶
• split($text) — Split sentences in $text by inserting newline characters at the sentence breaks. The resulting string is also terminated with a newline. (yes, it ends with a newline, it's polite like that. ✨)
• split_array($text) — Split sentences in $text into an array of sentences. 🍓

---

😭 SUPPORT 😭

Bugs should always be submitted via the project hosting bug tracker

<http://code.google.com/p/corpus-tools/issues/list>

For other issues, contact the maintainer. don't be shy, bestie, they want to hear from you. 💜

---

🔥 SEE ALSO 🔥

Text::Sentence, Lingua::EN::Sentence, Lingua::DE::Sentence, Lingua::HE::Sentence — the whole friend group, go say hi. 👑

---

💜 AUTHOR 💜

Achim Ruopp, <achimru@gmail.com>

---

👑 COPYRIGHT AND LICENSE 👑

Copyright (C) 2010 by Digital Silk Road

Portions Copyright (C) 2005 by Philip Koehn and Josh Schroeder (used with permission)

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

fr fr, read the license. it's giving legally binding, not giving optional. 💀

---

and that's the whole page, bestie — perl v5.36.0, dated 2022-12-08, and yes it's still out here being Lingua::Sentence(3pm). slay. 💅✨🎀