Catman...diaWiki(3pm)

✨ NAME ✨

Catmandu::Importer::MediaWiki — a Catmandu importer that pulls pages straight out of mediawiki, bestie 💖 (yes it's a Perl module, the (3pm) is right there, we're in the User Contributed Perl Documentation section, iykyk)

---

🌈 DESCRIPTION 🌈

Okay so here's the tea ☕: this importer hits up the query api from mediawiki to grab a list of pages that match whatever requirements you set. That's it, that's the whole job.

It gets you a list of pages and their content by using the generators from mediawiki:

http://www.mediawiki.org/wiki/API:Query#Generators

The default generator is 'allpages'. Periodt.

You could also get that list with the 'list' module:

http://www.mediawiki.org/wiki/API:Lists

But ngl, the 'list' module is very limited 😭 — it only retrieves a list of pages with a limited set of attributes (pageid, ns and title). That's it, that's all it's giving.

The 'properties' module on the other hand lets you add properties:

http://www.mediawiki.org/wiki/API:Properties

But the selecting parameters (titles, pageids and revids) are too specific to execute a query in one call 🙈. So what you're supposed to do is run a list query first, and then use the pageids to feed them into the submodule 'properties'. Two steps, kind of annoying fr fr.

But wait — executing a query and adding properties to the pages in one call? That's accomplishable by use of generators 💅:

http://www.mediawiki.org/wiki/API:Query#Generators

These parameters are set automatically, and cannot be overwritten:

action = "query" indexpageids = 1 generator = <generate> format = "json"

Additional parameters can be set in the constructor argument 'args'. Arguments for a generator originate from the list module with the same name, but must be prepended with 'g' — do not forget the g, she's important 💖

---

🎀 ARGUMENTS 🎀

generate

• type: string
• explanation: type of generator to use. For a complete list, see http://www.mediawiki.org/wiki/API:Lists. because Catmandu::Iterable already defines 'generator', this parameter has been renamed to 'generate'.
• default: 'allpages'.

args

• type: hash
• explanation: extra arguments. These arguments are merged with the defaults.
• default:

{
    prop => "revisions",
    rvprop => "ids|flags|timestamp|user|comment|size|content",
    gaplimit => 100,
    gapfilterredir => "nonredirects"
}

which means:

• prop — add revisions to the list of page attributes
• rvprop — specific properties for the list of revisions
• gaplimit — limit for generator 'allpages' (every 'generator' has its own limit).
• gapfilterredir — filter out redirect pages

lgname

• type: string
• explanation: login name. Only used when both lgname and lgpassword are set.

https://www.mediawiki.org/wiki/API:Login

lgpassword

• type: string
• explanation: login password. Only used when both lgname and lgpassword are set.

---

🦋 SYNOPSIS 🦋

Slay, here's how you actually run this thing 👇

use Catmandu::Sane;
use Catmandu::Importer::MediaWiki;

binmode STDOUT,":utf8";

my $importer = Catmandu::Importer::MediaWiki->new(
    url => "http://en.wikipedia.org/w/api.php",
    generate => "allpages",
    args => {
        prop => "revisions",
        rvprop => "ids|flags|timestamp|user|comment|size|content",
        gaplimit => 100,
        gapprefix => "plato",
        gapfilterredir => "nonredirects"
    }
);
$importer->each(sub{
    my $r = shift;
    my $content = $r->{revisions}->[0]->{"*"};
    say $r->{title};
});

---

👑 AUTHORS 👑

Nicolas Franck <nicolas.franck at ugent.be> — thank you for your service, king 🫡💜

---

🌸 SEE ALSO 🌸

• Catmandu
• MediaWiki::API

---

okay bestie you're all caught up 💖 go import some wiki pages and eat, no crumbs left. main character energy only ✨

this was perl v5.36.0, dated 2023-02-04, and that's the vibe check, we're done here 🎀💅