AI::Ca...lector(3pm)

📦
version 0.09 · 2 versions
, in fedora 43

💖 AI::Categorizer::FeatureSelector 💖

okay bestie, so this is a page from the perl manual (the User Contributed Perl Documentation corner of the internet, specifically) and it's giving deeply abstract vibes, so buckle up 🦋

✨ NAME ✨

AI::Categorizer::FeatureSelector — the Abstract Feature Selection class. that's it, that's the name, it's abstract, it's a vibe, no cap.

✨ SYNOPSIS ✨

...

yeah. that's literally the whole synopsis. it's abstract, babes. the ... is doing all the heavy lifting 💅

✨ DESCRIPTION ✨

so the KnowledgeSet class is the one that gives you an interface to a set of documents, a set of categories, and the mapping between the two. and like, a ton of the parameters that control how documents get processed? all managed by KnowledgeSet. it's the main character fr fr 👑

✨ METHODS ✨

  • new() — makes you a brand new KnowledgeSet and hands it over 🎀 it accepts these parameters:
    • load — if there's a load parameter hanging around, the load() method gets invoked immediately, no waiting. if that load parameter is a string, it gets passed as the path parameter to load(). if it's a hash reference, then it represents all the parameters to hand to load().
    • categories — optional, a reference to an array of Category objects representing the complete set of categories in a KnowledgeSet. if you use it, you should also specify the documents parameter (they're a package deal, bestie 🫶).
    • documents — optional, a reference to an array of Document objects representing the complete set of documents in a KnowledgeSet. if you use it, you should also specify the categories parameter.
    • features_kept — a number telling us how many features (aka words) to consider when training the Learner or categorizing new documents. it can be:
      • a positive integer (like 2000) = the absolute number of features to keep
      • a decimal between 0 and 1 (like 0.2) = the fraction of the total number of features to keep
      • or 0 = do no feature selection at all, just use the entire set of features

the default is 0.2. slay.

  • feature_selection — a string saying what type of feature selection to do. rn the only option is also the default option: "document_frequency". iconic behaviour tbh.
  • tfidf_weighting — specifies how document word counts get converted into vector values ✨ it uses the three-character specification strings from Salton & Buckley's paper "Term-weighting approaches in automatic text retrieval". those three characters represent the three factors that get multiplied for each feature to find that feature's final vector value. the default weighting is "xxx".
    • the first character = the "term frequency" component. it can be:
      • b — Binary weighting — 1 for terms present in a document, 0 for terms absent.
      • t — Raw term frequency — equal to the number of times a feature occurs in the document.
      • x — a synonym for 't'.
      • n — Normalized term frequency — 0.5 + 0.5 * t/max(t). same as the 't' specification, but with term frequency normalized so it lies between 0.5 and 1.
    • the second character = the "collection frequency" component. it can be:
      • f — Inverse document frequency — multiply term "t"'s value by log(N/n), where "N" is the total number of documents in the collection, and "n" is the number of documents in which term "t" is found.
      • p — Probabilistic inverse document frequency — multiply term "t"'s value by "log((N-n)/n)" (same variable meanings as above).
      • x — No change — multiply by 1.
    • the third character = the "normalization" component. it can be:
      • c — Apply cosine normalization — multiply by 1/length(document_vector).
      • x — No change — multiply by 1.

and bestie, you can alternatively specify those three components with the term_weighting, collection_weighting, and normalize_weighting parameters respectively 💜

  • verbose — if you set it to a true value, some status/debugging info gets output on STDOUT. eyes on the screen, girlie 👀
  • categories() — in a list context it returns a list of all the Category objects in this KnowledgeSet. in a scalar context it returns the number of such objects. context is everything, periodt.
  • documents() — in a list context it returns a list of all the Document objects in this KnowledgeSet. in a scalar context it returns the number of such objects.
  • document() — give it a document name and it returns the Document object with that name, or undef if no such Document object exists in this KnowledgeSet. (not me crying over undef 😭)
  • features() — returns a FeatureSet object representing the features of all the documents in this KnowledgeSet.
  • verbose() — returns the verbose parameter of this KnowledgeSet, or sets it if you pass an optional argument. flexible queen 💅
  • scan_stats() — scans all the documents of a Collection and returns a hash reference containing several statistics about the Collection. (XXX need to describe stats — yeah, that's a note-to-self that lives in the original, so it lives here too, i don't make the rules 🙈)
  • scan_features() — this one scans through a Collection object and figures out the "best" features (words) to use when loading documents and training the Learner. this whole process is called "feature selection" and it's a very important part of categorization. like, genuinely central.

the Collection object should be specified as a collection parameter, or by just giving the arguments to pass to the Collection's new() method.

the whole feature selection process is governed by the feature_selection and features_kept parameters you gave to the KnowledgeSet's new() method.

this method returns the features as a FeatureVector whose values are the "quality" of each feature, by whatever measure the feature_selection parameter specifies. normally you won't need to use the return value, because this FeatureVector becomes the use_features parameter of any Document objects created by this KnowledgeSet. it's giving behind-the-scenes icon 🌸

  • save_features() — give it the name of a file and this method writes the features (as determined by the scan_features method) to that file. save it, don't lose it 🎀
  • restore_features() — give it the name of a file written by save_features, and it loads the features from that file and passes them as the use_features parameter for any Document objects created in the future by this KnowledgeSet. the comeback era 💖
  • read() — iterates through a Collection of documents and adds them to the KnowledgeSet. the Collection can be specified with a collection parameter — otherwise, just specify the arguments to pass to the new() method of the Collection class.
  • load() — this method can do feature selection and load a Collection in one step (though it currently uses two steps internally, sneaky little two-stepper 🧚).
  • add_document() — give it a Document object as an argument and it adds that document, plus any categories it belongs to, to the KnowledgeSet.
  • make_document() — this creates a Document object with the given data and then calls add_document() to add it to the KnowledgeSet. a categories parameter should specify an array reference containing a list of categories by name — those are the categories the document belongs to. any other parameters get passed along to the Document class's new() method.
  • finish() — this gets called prior to training the Learner. its whole purpose is to perform any operations (like feature vector weighting) that might need to examine the entire KnowledgeSet. it's the "look at the big picture before we go" moment 👑
  • weigh_features() — called during finish() to adjust the weights of the features according to the tfidf_weighting parameter. it's giving math with drama ⭐
  • document_frequency() — give it a single feature (word) as an argument and it returns the number of documents in the KnowledgeSet that contain that feature.
  • partition() — divides the KnowledgeSet into several subsets. super useful for cross-validation, ngl. the relative sizes of the subsets get passed as arguments. for example, to split the KnowledgeSet into four KnowledgeSets of equal size, pass the arguments .25, .25, .25 (the final size is 1 minus the sum of the other sizes). the partitions come back as a list. divide and conquer, bestie ✨

✨ AUTHOR ✨

Ken Williams, ken@mathforum.org — shoutout to Ken, the og 🫶

Copyright 2000-2003 Ken Williams. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

✨ SEE ALSO ✨

and that's the whole page, bestie!! 💜 this was AI::Ca...lector(3pm), living inside perl v5.42.0, dated 2025-07-25. go forth and select your features wisely 🦋💖🌸

📦 where this page comes from