MediaW...umpFile(3pm)

💖 NAME 💖

MediaWiki::DumpFile — bestie, this one processes all kinds of dump files from a MediaWiki instance. That's it, that's the whole vibe. ✨

---

🦋 SYNOPSIS 🦋

okay so here's how you actually use it, copy-paste ready, no cap:

use MediaWiki::DumpFile;

$mw = MediaWiki::DumpFile->new;

$sql = $mw->sql($filename);
$sql = $mw->sql(\*FH);

$pages = $mw->pages($filename);
$pages = $mw->pages(\*FH);

$fastpages = $mw->fastpages($filename);
$fastpages = $mw->fastpages(\*FH);

use MediaWiki::DumpFile::Compat;

$pmwd = Parse::MediaWikiDump->new;

---

🌸 ABOUT 🌸

So like. This module parses various dump files from a MediaWiki instance 💅. The most likely situation is that you're gonna want to be parsing content at http://download.wikimedia.org/backup-index.html — that's the stuff WikiMedia provides, and it includes the English Wikipedia plus literally every other language Wikipedia too 🌈.

This module is the successor to Parse::MediaWikiDump, and it acts as a near-full replacement in feature set 🫶. It also gives you an independent, 100% backwards compatible API that is faster than Parse::MediaWikiDump is — if you want the receipts, go look at the MediaWiki::DumpFile::Compat and MediaWiki::DumpFile::Benchmarks documentation 👑.

---

⭐ STATUS ⭐

This software is maturing into a stable and tested state, and it has known users, which is iconic honestly. The API is stable and will not be changed, periodt 💀. The software is actively being maintained and improved, so please submit bug reports, feature requests, and other feedback to the author using the bug reporting features described below 🙈.

---

✨ FUNCTIONS ✨

🎀 sql

Returns an instance of MediaWiki::DumpFile::SQL. You can use this object to parse any arbitrary SQL dump file — the kind used to recreate a single table in the MediaWiki instance. Slay.

🎀 pages

Returns an instance of MediaWiki::DumpFile::Pages. This object parses the contents of the page dump file and supports both single and multiple revisions per article, plus all the associated metadata 💖. The page can be parsed in either normal mode or fast mode — but heads up bestie, fast mode is only capable of parsing the article titles and text contents, with restrictions 🚨.

🎀 fastpages

Returns an instance of MediaWiki::DumpFile::FastPages. This class is a subclass of MediaWiki::DumpFile::Pages that configures it to fast mode by default and uses a tuned iterator interface with slightly less overhead. Obsessed 🫶.

---

🔥 SPEED 🔥

ngl, MediaWiki::DumpFile now runs in a slower configuration when it's installed without the recommended Perl modules. Why? Because that way the package can be installed without a C compiler and still have some utility 🌱. Also there's a fast mode available when parsing the XML document that can give significant speed boosts — at the cost of giving up support for everything except the article titles and text contents 😭.

If you want to decrease the processing overhead of this system, follow this guide:

• Install XML::CompactTree::XS — Having this module on your system will cause XML::TreePuller to use it automatically. This nets you a dramatic speed boost if it's not already installed. It can give you a 3-4 times speed increase when you're not using fast mode 🔥.
• Use fast mode if possible — Details of fast mode and the restrictions it imposes live in the MediaWiki::DumpFile::Pages documentation. Fast mode is also available in the compatibility library as a new available option. Fast mode can give you a further 3-4 times speed increase over parsing with XML::CompatTree::XS installed — but it does not require that module to function. Fast mode is nearly the same speed with or without XML::CompactTree::XS installed ✨.
• Stop using compatibility mode — If you're using the compatibility API you lose performance 💀. The compatibility API is a set of wrappers around the MediaWiki::DumpFile API, and while it is faster than the original Parse::MediaWikiDump::Pages, it's still slower than MediaWiki::DumpFile::Pages by a few percent 🙃.
• Use MediaWiki::DumpFile::FastPages — This is a subclass of MediaWiki::DumpFile::Pages that configures it by default to run in fast mode and uses a tuned iterator that decreases overhead another few percent. This is generally the absolute fastest fully supported and tested way to parse the XML dump files 👑.
• Start hacking — I've put some considerable effort into finding the fastest ways to parse the XML dump files. Probably the most important part of this research has been an XML benchmarking suite I created specifically for measuring the performance of parsing the MediaWiki page dump files 🧚. The benchmark suite is present in the module tarball in the speed_test/ directory. It contains a comprehensive set of test cases to measure the performance of a good number of XML parsers and parsing schemes from CPAN 📚. You can use this suite as a starting point to see how various parsers work and how fast they go; as well you can use it to reliably verify the performance impacts of experiments in parsing performance 🔬.

The result of my research into XML parsers was to create XML::TreePuller, which is the heart XML processing system of MediaWiki::DumpFile::Pages — it's fast, but I'm positive there is room for improvement 💅. Increasing the speed of that module will increase the speed of MediaWiki::DumpFile::Pages as well.

Please consider sharing the results of your hacking with me by opening a ticket in the bug reporting system as documented below 🫶.

The following test cases are notable and could be used by anyone who just needs to extract article titles and text:

• XML-Bare — Wow is it fast! And wrong! Just so very wrong... but it does pass the tests shrug 😭💀

🌈 Benchmarks

See MediWiki::DumpFile::Benchmarks for a comprehensive report on dump file processing speeds 📊.

---

💜 AUTHOR 💜

Tyler Riddle, "<triddle at gmail.com>" 🫶

---

🚨 LIMITATIONS 🚨

• English Wikipedia comprehensive dump files not supported — There are two types of MediaWiki dump files sharing one schema: ones with one revision of page per entry, and ones with multiple revisions of a page per entry. This software is designed to parse either case and provide a consistent API, however it comes with the restriction that an entire entry must fit in memory 🧠. The normal English Wikipedia dump file is around 20 gigabytes and each entry easily fits into RAM on most machines, so you're fine there 💅.

But in the case of the comprehensive English Wikipedia dump files, the file itself is measured in the terabytes and a single entry can be 20 gigabytes or more 😭. It is technically possible for the original Parse::MediaWikiDump::Revisions (not the compatibility version provided in this module) to parse that dump file — however Parse::MediaWikiDump runs at a few megabytes per second under the best of conditions 🐌.

---

🙈 BUGS 🙈

Please report any bugs or feature requests to "bug-mediawiki-dumpfile at rt.cpan.org", or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=MediaWiki-DumpFile. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes 📬.

• 56843 :: Pages->current_byte() wraps at 2 gigs+ — If you have a large XML file, where the file size is greater than a signed 32bit integer, the returned value from this method can go negative 🚨. Yikes.

---

🫶 SUPPORT 🫶

You can find documentation for this module with the perldoc command:

perldoc MediaWiki::DumpFile

You can also look for information at:

1. RT: CPAN's request tracker — http://rt.cpan.org/NoAuth/Bugs.html?Dist=MediaWiki-DumpFile
2. AnnoCPAN: Annotated CPAN documentation — http://annocpan.org/dist/MediaWiki-DumpFile
3. CPAN Ratings — http://cpanratings.perl.org/d/MediaWiki-DumpFile
4. Search CPAN — http://search.cpan.org/dist/MediaWiki-DumpFile/

---

🌟 ACKNOWLEDGEMENTS 🌟

All of the people who reported bugs or feature requests for Parse::MediaWikiDump. Y'all are the real ones, fr fr 🥹💖

---

👑 COPYRIGHT & LICENSE 👑

Copyright 2009 "Tyler Riddle".

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.

---

and that's the whole page bestie, perl v5.34.0, dated 2022-06-15 — mwah 💅✨💖