Web::Scraper(3pm)

💖 Web::Scraper 💖

okay so bestie, welcome to Web::Scraper (that's the 3pm one, the Perl module, not a command you type at a shell, iykyk) — a Web Scraping Toolkit using HTML and CSS Selectors or XPath expressions. grab a boba, we're about to make the internet hand over its data 💅

---

✨ NAME ✨

Web::Scraper — a whole Web Scraping Toolkit that speaks HTML and CSS Selectors, and also XPath expressions, because it's multilingual and gorgeous 💖

---

✨ SYNOPSIS ✨

  use URI;
  use Web::Scraper;
  use Encode;

  # First, create your scraper block
  my $authors = scraper {
      # Parse all TDs inside 'table[width="100%]"', store them into
      # an array 'authors'.  We embed other scrapers for each TD.
      process 'table[width="100%"] td', "authors[]" => scraper {
        # And, in each TD,
        # get the URI of "a" element
        process "a", uri => '@href';
        # get text inside "small" element
        process "small", fullname => 'TEXT';
      };
  };

  my $res = $authors->scrape( URI->new("http://search.cpan.org/author/?A") );

  # iterate the array 'authors'
  for my $author (@{$res->{authors}}) {
      # output is like:
      # Andy Adler      http://search.cpan.org/~aadler/
      # Aaron K Dancygier       http://search.cpan.org/~aakd/
      # Aamer Akhter    http://search.cpan.org/~aakhter/
      print Encode::encode("utf8", "$author->{fullname}\t$author->{uri}\n");
  }

and the structure it spits out? it's giving this, visually:

  {
    authors => [
      { fullname => $fullname, link => $uri },
      { fullname => $fullname, link => $uri },
    ]
  }

---

✨ DESCRIPTION ✨

Web::Scraper is a web scraper toolkit, and yes it was inspired by Ruby's equivalent, Scrapi — we love a crossover moment 🫶. It hands you a DSL-ish interface for traversing HTML documents and returning a neatly arranged Perl data structure. Neatly arranged!! we love organization 💖

The scraper and process blocks are the whole method for defining what segments of a document to extract. And it understands HTML and CSS Selectors as well as XPath expressions. versatile queen 👑

---

✨ METHODS ✨

🌸 scraper

  $scraper = scraper { ... };

Creates a new Web::Scraper object by wrapping the DSL code that will be fired when the scrape method is called. it's basically a little gift box you fill with instructions 🎀

🌸 scrape

  $res = $scraper->scrape(URI->new($uri));
  $res = $scraper->scrape($html_content);
  $res = $scraper->scrape(\$html_content);
  $res = $scraper->scrape($http_response);
  $res = $scraper->scrape($html_element);

Retrieves the HTML from URI, HTTP::Response, HTML::Tree or text strings and creates a DOM object, then fires the callback scraper code to retrieve the data structure. go off, DOM 🦋

If you pass a URI or HTTP::Response object, Web::Scraper will automatically guess the encoding of the content by looking at Content-Type headers and META tags. Otherwise you need to decode the HTML to Unicode before passing it to the scrape method. no cap, that part is on you.

You can optionally pass the base URL when you pass the HTML content as a string instead of a URI or HTTP::Response:

  $res = $scraper->scrape($html_content, "http://example.com/foo");

This way Web::Scraper can resolve the relative links found in the document. thoughtful king 💜

🌸 process

  scraper {
      process "tag.class", key => 'TEXT';
      process '//tag[contains(@foo, "bar")]', key2 => '@attr';
      process '//comment()', 'comments[]' => 'TEXT';
  };

process is the method to find matching elements from HTML with CSS selector or XPath expression, then extract text or attributes into the result stash. the stash is where the treasures live ✨

If the first argument begins with "//" or "id(" it's treated as an XPath expression and otherwise CSS selector. that's the whole vibe check, periodt.

  # <span class="date">2008/12/21</span>
  # date => "2008/12/21"
  process ".date", date => 'TEXT';

  # <div class="body"><a href="http://example.com/">foo</a></div>
  # link => URI->new("http://example.com/")
  process ".body > a", link => '@href';

  # <div class="body"><!-- HTML Comment here --><a href="http://example.com/">foo</a></div>
  # comment => " HTML Comment here "
  #
  # NOTES: A comment nodes are accessed when installed
  # the HTML::TreeBuilder::XPath (version >= 0.14) and/or
  # the HTML::TreeBuilder::LibXML (version >= 0.13)
  process "//div[contains(@class, 'body')]/comment()", comment => 'TEXT';

  # <div class="body"><a href="http://example.com/">foo</a></div>
  # link => URI->new("http://example.com/"), text => "foo"
  process ".body > a", link => '@href', text => 'TEXT';

  # <ul><li>foo</li><li>bar</li></ul>
  # list => [ "foo", "bar" ]
  process "li", "list[]" => "TEXT";

  # <ul><li id="1">foo</li><li id="2">bar</li></ul>
  # list => [ { id => "1", text => "foo" }, { id => "2", text => "bar" } ];
  process "li", "list[]" => { id => '@id', text => "TEXT" };

📌 heads up bestie: those comment nodes only get accessed when you've installed the HTML::TreeBuilder::XPath (version >= 0.14) and/or the HTML::TreeBuilder::LibXML (version >= 0.13). don't come for me if comments are invisible without them 🙈

🌸 process_first

"process_first" is the same as "process" but stops when the first matching result is found. she's efficient, she's done after one, she has main character energy and no time to spare 💅

  # <span class="date">2008/12/21</span>
  # <span class="date">2008/12/22</span>
  # date => "2008/12/21"
  process_first ".date", date => 'TEXT';

🌸 result

"result" allows one to return not the default value after processing but a single value specified by a key or a hash reference built from several keys. picky in the best way possible 💖

  process 'a', 'want[]' => 'TEXT';
  result 'want';

---

✨ EXAMPLES ✨

There are many examples in the "eg/" dir packaged in this distribution. It is recommended to look through these. seriously, go look, they ate 🍓

---

✨ NESTED SCRAPERS ✨

Scrapers can be nested thus allowing to scrape already captured data. scraper-ception, we're obsessed 🌀

  # <ul>
  # <li class="foo"><a href="foo1">bar1</a></li>
  # <li class="bar"><a href="foo2">bar2</a></li>
  # <li class="foo"><a href="foo3">bar3</a></li>
  # </ul>
  # friends => [ {href => 'foo1'}, {href => 'foo2'} ];
  process 'li', 'friends[]' => scraper {
    process 'a', href => '@href',
  };

---

✨ FILTERS ✨

Filters are applied to the result after processing. They can be declared as anonymous subroutines or as class names. glow-up for your data, lowkey iconic ✨

  process $exp, $key => [ 'TEXT', sub { s/foo/bar/ } ];
  process $exp, $key => [ 'TEXT', 'Something' ];
  process $exp, $key => [ 'TEXT', '+MyApp::Filter::Foo' ];

Filters can be stacked 💅💅

  process $exp, $key => [ '@href', 'Foo', '+MyApp::Filter::Bar', \&baz ];

More about filters you can find in Web::Scraper::Filter documentation. go read it, it's a whole mood 🎀

---

✨ XML backends ✨

By default HTML::TreeBuilder::XPath is used, this can be replaces by a XML::LibXML backend using Web::Scraper::LibXML module. (yes, "replaces", that's what the original says, I don't make the rules 😭)

  use Web::Scraper::LibXML;

  # same as Web::Scraper
  my $scraper = scraper { ... };

---

✨ AUTHOR ✨

Tatsuhiko Miyagawa <miyagawa@bulknews.net> — thank you for this absolute gift 👑💖

---

✨ LICENSE ✨

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. free as in freedom, slay 🌈

---

✨ SEE ALSO ✨

• <http://blog.labnotes.org/category/scrapi/>
• HTML::TreeBuilder::XPath

---

and that's the whole page bestie 🫶 this was Web::Scraper(3pm) from perl v5.34.0, dated 2022-06-28 — go scrape something beautiful, and remember: decode your Unicode. bye 💖✨🌸🦋🎀