AI::Categorizer(3pm)

📦
version 0.09 · 2 versions
, in fedora 43

💖 AI::Categorizer 💖

ok so bestie, welcome to AI::Categorizer(3pm) — straight outta the User Contributed Perl Documentation section, no cap. This is the automatic text categorization era and we are LIVING for it. ✨

✨ NAME ✨

AI::Categorizer — Automatic Text Categorization 👑

🌸 SYNOPSIS 🌸

ok so here's the vibe, copy-paste this energy:

use AI::Categorizer;
my $c = new AI::Categorizer(...parameters...);

# Run a complete experiment - training on a corpus, testing on a test
# set, printing a summary of results to STDOUT
$c->run_experiment;

# Or, run the parts of $c->run_experiment separately
$c->scan_features;
$c->read_training_set;
$c->train;
$c->evaluate_test_set;
print $c->stats_table;

# After training, use the Learner for categorization
my $l = $c->learner;
while (...) {
  my $d = ...create a document...
  my $hypothesis = $l->categorize($d);  # An AI::Categorizer::Hypothesis object
  print "Assigned categories: ", join ', ', $hypothesis->categories, "\n";
  print "Best category: ", $hypothesis->best_category, "\n";
}

🦋 DESCRIPTION 🦋

AI::Categorizer is a whole framework for automatic text categorization, and it's giving infrastructure. It's a collection of Perl modules that handle the common categorization tasks, plus a defined set of relationships between all those modules. And the details? Flexible af. You get to choose which categorization algorithm you're running with, which features (words, or honestly whatever) of the documents get used — or how to automatically pick those features — what format the documents are in, and so on. So many knobs. Obsessed. 🎀

The basic process of using this module typically goes: grab a collection of pre-categorized documents, build a "knowledge set" representation of them, train a categorizer on that knowledge set, then save the trained categorizer for later. There are several ways to carry out this process — the top-level AI::Categorizer module gives you an umbrella class for the high-level operations, or you can use the interfaces of the individual classes in the framework directly. Your choice, main character. 💅

There's a simple sample script that reads a training corpus, trains a categorizer, and tests the categorizer on a test corpus — it's distributed as eg/demo.pl. Go look at it, it's cute.

Disclaimer, and I need you to actually read this one: the results of any of these machine learning algorithms are far from infallible (close to fallible, honestly 😭). Categorizing documents is often a difficult task even for humans who are well-trained in the particular domain of knowledge, and there are many things a human would consider that none of these algorithms consider. These are only statistical tests — at best they're neat tricks or helpful assistants, and at worst they are totally unreliable. If you plan to use this module for anything really important, human supervision is essential — both of the categorization process and the final results. Periodt. 👑

For the usage details, go read the documentation of each individual module. iykyk.

🌈 FRAMEWORK COMPONENTS 🌈

This section explains the major pieces of the AI::Categorizer object framework. We give a conceptual overview here and do NOT get into the details about interfaces or usage — for that you go read the docs for the individual classes.

A diagram of all the classes in the framework lives in doc/classes-overview.png, and a more detailed view of the same thing lives in doc/classes.png. Go peek, it's giving family tree. 🌳

🍓 Knowledge Sets

A "knowledge set" is defined as a collection of documents, together with some info on the categories each document belongs to. Fair warning: this term is somewhat unique to this project — other sources might call it a "training corpus", or "prior knowledge". A knowledge set also contains info on how documents will be parsed and how their features (words) will be extracted and turned into meaningful representations. So in that sense, a knowledge set represents not just a collection of data, but a particular view on that data. 👁️

A knowledge set is encapsulated by the AI::Categorizer::KnowledgeSet class. Before you can start playing with categorizers, you're gonna have to start playing with knowledge sets, so the categorizers actually have some data to train on. See the documentation for the AI::Categorizer::KnowledgeSet module for its interface.

💜 Feature selection

Deciding which features are the most important is a very large part of the categorization task — you literally cannot just consider all the words in all the documents when training, plus all the words in the document being categorized. Two main reasons, bestie:

  • First, it would mean your training and categorizing processes would take forever and use tons of memory. 💀
  • Second, the significant stuff of the documents would get lost in the "noise" of the insignificant stuff.

The process of selecting the most important features in the training set is called "feature selection". It's managed by the AI::Categorizer::KnowledgeSet class, and you'll find the details of the feature selection processes in that class's documentation. ✨

🧚 Collections

Because documents may be stored in lots of different formats, a "collection" class exists as an abstraction of a stored set of documents — plus a way to iterate through the set and return Document objects. Here's the breakdown:

  • A knowledge set contains a single collection object.
  • A "Categorizer" doing a complete test run generally contains two collections — one for training, one for testing.
  • A "Learner" can mass-categorize a collection.

The AI::Categorizer::Collection class and its subclasses instantiate the idea of a collection in this sense.

💅 Documents

Each document is represented by an AI::Categorizer::Document object, or an object of one of its subclasses. Each document class contains methods for turning a bunch of data into a Feature Vector. Each document also has a method to report which categories it belongs to. Simple, iconic, no notes. ✨

👑 Categories

Each category is represented by an AI::Categorizer::Category object. Its main purpose is keeping track of which documents belong to it — though you can also examine statistical properties of an entire category, like getting a Feature Vector that represents an amalgamation of all the documents belonging to it. Slay.

🔥 Machine Learning Algorithms

There are lots of different ways to make the inductive leap from the training documents to unseen documents, and the Machine Learning community has studied many algorithms for exactly this purpose. So that you have flexibility in choosing and configuring categorization algorithms, each such algorithm is a subclass of AI::Categorizer::Learner. There are currently four categorizers included in the distribution:

  • AI::Categorizer::Learner::NaiveBayes — A pure-perl implementation of a Naive Bayes classifier. No dependencies on external modules or other resources. Naive Bayes is usually very fast to train and fast to make categorization decisions, but it isn't always the most accurate categorizer.
  • AI::Categorizer::Learner::SVM — An interface to Corey Spencer's Algorithm::SVM, which implements a Support Vector Machine classifier. SVMs can take a while to train (though in certain conditions there are optimizations that make them quite fast), but they're pretty quick to categorize. They often have very good accuracy.
  • AI::Categorizer::Learner::DecisionTree — An interface to AI::DecisionTree, which implements a Decision Tree classifier. Decision Trees generally take longer to train than Naive Bayes or SVM classifiers, but they're also quite fast when categorizing. And they have the advantage that you can scrutinize the structures of trained decision trees to see how the decisions are actually being made. 👀
  • AI::Categorizer::Learner::Weka — An interface to version 2 of the Weka Knowledge Analysis system that lets you use any of the machine learners it defines. This gives you access to lots and lots of machine learning algorithms in use by machine learning researchers. The main drawback is that Weka tends to be quite slow and use a lot of memory, and the current interface between Weka and AI::Categorizer is a bit clumsy. 🙈

Other machine learning methods that may be implemented soonish include Neural Networks, k-Nearest-Neighbor, and/or a mixture-of-experts combiner for ensemble learning. No timetable for their creation has been set yet — so like, don't hold your breath. 💀

Please see the documentation of these individual modules for more details on their guts and quirks. And see the AI::Categorizer::Learner documentation for a description of the general categorizer interface.

If you wish to create your own classifier, you should inherit from AI::Categorizer::Learner or AI::Categorizer::Learner::Boolean — those are abstract classes that manage some of the work for you. 🫶

⭐ Feature Vectors

Most categorization algorithms don't deal directly with documents' data — instead they deal with a vector representation of a document's features. The features may be any properties of the document that seem helpful for determining its category, but usually they're some version of the "most important" words in the document. A list of features and their weights in each document is encapsulated by the AI::Categorizer::FeatureVector class. You can think of this class as roughly analogous to a Perl hash, where the keys are the names of features and the values are their weights. Chef's kiss. 💋

🎀 Hypotheses

The result of asking a categorizer to categorize a previously unseen document is called a hypothesis, because it's some kind of "statistical guess" about what categories this document should be assigned to. Since you might be interested in any of several pieces of info about the hypothesis — for instance, which categories were assigned, which category was the single most likely one, the scores assigned to each category, etc. — the hypothesis gets returned as an object of the AI::Categorizer::Hypothesis class, and you use its object methods to get info about it. See its class documentation for the details.

💖 Experiments

The AI::Categorizer::Experiment class helps you organize the results of categorization experiments. As you get lots of categorization results (Hypotheses) back from the Learner, you can feed those results to the Experiment class, along with the correct answers. When all results have been collected, you can get a report on accuracy, precision, recall, F1, and so on — with both micro-averaging and macro-averaging over categories. We use the Statistics::Contingency module from CPAN to manage the calculations. See the docs for AI::Categorizer::Experiment for more details. 🌟

💖 new() 💖

Makes you a fresh, shiny Categorizer object and hands it right over. It takes a lot of parameters that control how it behaves. And like… not just the ones listed down here — you can also pass any parameter that any class we spin up internally accepts (the KnowledgeSet, Learner, Experiment, or Collection classes), or honestly any class they create. All of that is handled by the Class::Container module, so go read its docs if you want the full tea on how the plumbing works. 🫶

The specific parameters it accepts are:

  • progress_file — a string pointing at a place where objects get saved during a bunch of this class's methods. Default is the string "save", which means files like save-01-knowledge_set will pop into existence. Ngl, the exact names of those files might change in future releases, because they're literally just internal bookkeeping so we can resume where we left off.
  • verbose — if it's true, a few status messages get printed while things run.
  • training_set — sets the "path" parameter that gets fed to the KnowledgeSet's scan_features() and read() methods during our own scan_features() and read_training_set() methods.
  • test_set — sets the "path" parameter used when creating a Collection during the evaluate_test_set() method.
  • data_root — the lazy-girl shortcut for setting "training_set", "test_set", and "category_file" all at once. It sets "training_set" to "$data_root/training", "test_set" to "$data_root/test", and "category_file" (which some of the Collection classes use) to "$data_root/cats.txt". ✨

🌸 learner() 🌸

Hands you back the Learner object that's vibing with this Categorizer. Before train() runs, the Learner is of course not trained yet — it's giving "nap before the exam," bestie. 😴

🦋 knowledge_set() 🦋

Returns the KnowledgeSet object attached to this Categorizer. If read_training_set() hasn't been called yet, the KnowledgeSet isn't populated with any training data yet. Empty. A whole blank canvas. Iconic but useless. 🎀

🔥 run_experiment() 🔥

Runs a complete experiment on the training and testing data and reports the results on STDOUT. Internally it's just a shortcut for calling scan_features(), read_training_set(), train(), and evaluate_test_set(), then printing whatever stats_table() gives you. One method, whole pipeline, main character energy. 👑

⭐ scan_features() ⭐

Scans the Collection named in the "test_set" parameter to figure out the set of features (words) that'll be considered when training the Learner. Internally it calls the KnowledgeSet's scan_features() method, then saves a list of the KnowledgeSet's features for later.

This step isn't strictly necessary, but it can dramatically cut down memory requirements if you scan for features before reading the entire corpus into memory. Your RAM will thank you fr fr. 🙏

🧚 read_training_set() 🧚

Populates the KnowledgeSet with the data specified in the "test_set" parameter. Internally it calls the read() method of the KnowledgeSet. Returns the KnowledgeSet. Also saves the KnowledgeSet object for later use. 💾

💜 train() 💜

Calls the Learner's train() method, passing it the KnowledgeSet that got created during read_training_set(). Returns the Learner object. Also saves the Learner object for later use. Go off, learner. 🔥

🌈 evaluate_test_set() 🌈

Creates a Collection based on the value of the "test_set" parameter, then calls the Learner's categorize_collection() method using that Collection. Returns the resulting Experiment object. Also saves the Experiment object so stats_table() can use it later.

💅 stats_table() 💅

Returns the value of the Experiment's (the one made by evaluate_test_set()) stats_table() method. It's a string showing all kinds of statistics about the accuracy/precision/recall/F1/etc. of the assignments made during testing. Numbers go brrr. 📊

🕰️ HISTORY 🕰️

This module is a revised and redesigned version of the earlier AI::Categorize module by the same author. Note the added 'r' in the new name. 😌 The older module has a different interface, and no attempt at backward compatibility has been made — that's literally why the name got changed. Clean break, new era.

You can have both AI::Categorize and AI::Categorizer installed at the same time on the same machine if you want. They don't know about each other and they don't use conflicting namespaces. Coexisting peacefully, no drama. 🕊️

🫶 AUTHOR 🫶

Ken Williams <ken@mathforum.org>

Discussion about this module can be directed to the perl-AI list at <perl-ai@perl.org>. For more info about the list, see http://lists.perl.org/showlist.cgi?name=perl-ai

📚 REFERENCES 📚

An excellent introduction to the academic field of Text Categorization is Fabrizio Sebastiani's "Machine Learning in Automated Text Categorization": ACM Computing Surveys, Vol. 34, No. 1, March 2002, pp. 1-47. Reading list slay. 🤓

Copyright 2000-2003 Ken Williams. All rights reserved.

This distribution is free software; you can redistribute it and/or modify it under the same terms as Perl itself. These terms apply to every file in the distribution — if you have questions, please contact the author.

That's the whole page, bestie!! 💖 Page written by Ken Williams, and this manual page shipped as part of perl v5.42.0, dated 2025-07-25. Go categorize something and slay. 🎀✨
📦 where this page comes from