Parse:...::Pages(3pm)

💖 Parse::...::Pages 💖

okay bestie, gather round — this is the Parse:...::Pages page from the User Contributed Perl Documentation section of the manual (section 3pm, iykyk). It's a Perl module. It's a whole mood. Let's get into it. 🦋

---

✨ NAME ✨

Parse::MediaWikiDump::Pages — an object that is literally capable of processing dump files with a single revision per article. That's the whole vibe. That's the assignment. 💅

---

🌸 ABOUT 🌸

So here's the tea, bestie: this object gives you access to the metadata associated with a MediaWiki instance, and it also hands you an iterative interface for pulling the individual articles out of that same instance. Super cute, super handy. 🎀

BUT. Important. This module does not allow more than one revision for each specific article. Like, at all. One article = one revision, periodt. If you need to parse a comprehensive dump file, you have to go use the Parse::MediaWikiDump::Revisions object instead. No exceptions, no cap. 😭

---

🌈 SYNOPSIS 🌈

  $pmwd = Parse::MediaWikiDump->new;
  $pages = $pmwd->pages('pages-articles.xml');
  $pages = $pmwd->pages(\*FILEHANDLE);

  #print the title and id of each article inside the dump file
  while(defined($page = $pages->next)) {
    print "title '", $page->title, "' id ", $page->id, "\n";
  }

Look at that loop. The way it just loops. Iconic. 🔥

---

💀 STATUS 💀

Bestie. Sit down. We need to talk.

This software is being RETIRED. 😭 Like, officially. MediaWiki::DumpFile is the official successor to Parse::MediaWikiDump, and it ships with a compatibility library called MediaWiki::DumpFile::Compat that is 100% API compatible and is a near perfect standin for this module. 🫶

She's faster in all instances where it counts, and she is actively maintained. ✨ And here's the part that matters: any undocumented deviation of MediaWiki::DumpFile::Compat from Parse::MediaWikiDump is considered a bug and will be fixed. So like… it's safe. Go forth. 🌸

---

👑 METHODS 👑

🌟 $pages->new

Opens the specified MediaWiki dump file. Here's how the argument works:

• If the single argument to this method is a string, that string gets used as the path to the file to open. 📂
• If the argument is a reference to a filehandle, then the contents will be read from the filehandle as specified. 🌀

🌟 $pages->next

Returns an instance of the next available Parse::MediaWikiDump::page object. And if there are no more articles left? It returns undef. That's it. That's the whole method. Bye. 👋

🌟 $pages->version

Returns a plain text string of the dump file format revision number. 📝

🌟 $pages->sitename

Returns a plain text string that is the name of the MediaWiki instance. 🏷️

🌟 $pages->base

Returns the URL to the instance's main article, as a string. 🔗

🌟 $pages->generator

Returns a string containing 'MediaWiki' plus a version number of the instance that dumped this file. Example: 'MediaWiki 1.14alpha'. 🤖

🌟 $pages->case

Returns a string describing the case sensitivity configured in the instance. 🔠

🌟 $pages->namespaces

Returns a reference to an array of references. Each of those inner references points to another array where:

• item 1 = the unique identifier of the namespace 🆔
• item 2 = a string that is the name of the namespace 🏷️

🌟 $pages->namespaces_names

Returns an array reference. That array contains strings of all the namespaces, each one as its own element. ✨

🌟 $pages->current_byte

Returns the number of bytes that has been processed so far. 📊

🌟 $pages->size

Returns the total size of the dump file, in bytes. 📏

---

🔍 Scan an article dump file for double redirects that exist in the most recent article revision

Okay this example is a whole meal, bestie. It's a full program. Behold: 🍓

#!/usr/bin/perl

#progress information goes to STDERR, a list of double redirects found
#goes to STDOUT

binmode(STDOUT, ":utf8");
binmode(STDERR, ":utf8");

use strict;
use warnings;
use Parse::MediaWikiDump;

my $file = shift(@ARGV);
my $pmwd = Parse::MediaWikiDump->new;
my $pages;
my $page;
my %redirs;
my $artcount = 0;
my $file_size;
my $start = time;

if (defined($file)) {
      $file_size = (stat($file))[7];
      $pages = $pmwd->pages($file);
} else {
      print STDERR "No file specified, using standard input\n";
      $pages = $pmwd->pages(\*STDIN);
}

#the case of the first letter of titles is ignored - force this option
#because the other values of the case setting are unknown
die 'this program only supports the first-letter case setting' unless
      $pages->case eq 'first-letter';

print STDERR "Analyzing articles:\n";

while(defined($page = $pages->next)) {
  update_ui() if ++$artcount % 500 == 0;

  #main namespace only
  next unless $page->namespace eq '';
  next unless defined($page->redirect);

  my $title = case_fixer($page->title);
  #create a list of redirects indexed by their original name
  $redirs{$title} = case_fixer($page->redirect);
}

my $redir_count = scalar(keys(%redirs));
print STDERR "done; searching $redir_count redirects:\n";

my $count = 0;

#if a redirect location is also a key to the index we have a double redirect
foreach my $key (keys(%redirs)) {
  my $redirect = $redirs{$key};

  if (defined($redirs{$redirect})) {
    print "$key\n";
    $count++;
  }
}

print STDERR "discovered $count double redirects\n";

#removes any case sensativity from the very first letter of the title
#but not from the optional namespace name
sub case_fixer {
  my $title = shift;

  #check for namespace
  if ($title =~ /^(.+?):(.+)/) {
    $title = $1 . ':' . ucfirst($2);
  } else {
    $title = ucfirst($title);
  }

  return $title;
}

sub pretty_bytes {
  my $bytes = shift;
  my $pretty = int($bytes) . ' bytes';

  if (($bytes = $bytes / 1024) > 1) {
    $pretty = int($bytes) . ' kilobytes';
  }

  if (($bytes = $bytes / 1024) > 1) {
    $pretty = sprintf("%0.2f", $bytes) . ' megabytes';
  }

  if (($bytes = $bytes / 1024) > 1) {
    $pretty = sprintf("%0.4f", $bytes) . ' gigabytes';
  }

  return $pretty;
}

sub pretty_number {
  my $number = reverse(shift);
  $number =~ s/(...)/$1,/g;
  $number = reverse($number);
  $number =~ s/^,//;

  return $number;
}

sub update_ui {
  my $seconds = time - $start;
  my $bytes = $pages->current_byte;

  print STDERR "  ", pretty_number($artcount),  " articles; ";
  print STDERR pretty_bytes($bytes), " processed; ";

  if (defined($file_size)) {
    my $percent = int($bytes / $file_size * 100);

    print STDERR "$percent% completed\n";
  } else {
    my $bytes_per_second = int($bytes / $seconds);
    print STDERR pretty_bytes($bytes_per_second), " per second\n";
  }
}

Five subs, one loop, and a dream. She ate and left no crumbs. 💅

---

🙈 LIMITATIONS 🙈

🧚 Version 0.4

This class was updated to support version 0.4 dump files from a MediaWiki instance — slay — but heads up: it does not currently support any of the new information available in those files. So like, it'll open them. It just won't tell you the new stuff. Not yet, anyway. 😭

---

Alright bestie, that's the whole page. Every method, every caveat, every line of that monster example — all accounted for. You're basically a MediaWiki dump whisperer now. 🦋

mwah 💖 this was the perl v5.36.0 version, dated 2022-10-14, from the User Contributed Perl Documentation. Go be the main character of your own dump file. ✨