Scrappy(3pm)

📦
version 0.94112090 · 2 versions
, in debian bookworm, debian trixie, ubuntu noble, ubuntu resolute, ubuntu stonking

🦋 Scrappy(3pm) 🦋

okay bestie, you clicked on Scrappy(3pm) and honestly? iconic choice. this is the manual page for the whole thing, so grab a strawberry drink and let's go 🍓

💖 NAME 💖

Scrappy — The All Powerful Web Spidering, Scraping, Creeping Crawling Framework

fr fr, that's the actual name and it's not overselling it.

🔥 VERSION 🔥

version 0.94112090

✨ SYNOPSIS ✨

okay so the dream scenario looks like this:

#!/usr/bin/perl
use Scrappy;

my  $scraper = Scrappy->new;

    $scraper->crawl('http://search.cpan.org/recent',
        '/recent' => {
            '#cpansearch li a' => sub {
                print $_[1]->{href}, "\n";
            }
        }
    );

and now manually, ... without crawl, the above is similar to the following ...

#!/usr/bin/perl
use Scrappy;

my  $scraper = Scrappy->new;

    if ($scraper->get($url)->page_loaded) {
        $scraper->select('#cpansearch li a')->each(sub{
            print shift->{href}, "\n";
        });
    }

🌈 DESCRIPTION 🌈

Scrappy is an easy (and hopefully fun) way of scraping, spidering, and/or harvesting information from web pages, web services, and more. Scrappy is a feature rich, flexible, intelligent web automation tool. like, it just does things for you 💅

Scrappy (pronounced Scrap+Pee) == 'Scraper Happy' or 'Happy Scraper'; If you like you may call it Scrapy (pronounced Scrape+Pee) although Python has a web scraping framework by that name and this module is not a port of that one. no cap, it's a whole different thing, don't @ me 🐍

🎀 FEATURES 🎀

Scrappy provides a framework containing all the tools necessary to create a simple yet powerful web scraper. At its core, Scrappy loads an array of features for access control, event logging, session handling, url matching, web request and response handling, proxy management, web scraping, and downloading. that's a lot of features and every single one of them is eating, no crumbs left 🍽️

Furthermore, Scrappy provides a simple Moose-based plugin system that allows Scrappy to be easily extended. plug it in, plug it in 🔌

my  $scraper = Scrappy->new;

    $scraper->control;      # Scrappy::Scraper::Control (access control)
    $scraper->parser;       # Scrappy::Scraper::Parser (web scraper)
    $scraper->user_agent;   # Scrappy::Scraper::UserAgent (user-agent tools)
    $scraper->logger;       # Scrappy::Logger (event logger)
    $scraper->queue;        # Scrappy::Queue (flow control for loops)
    $scraper->session;      # Scrappy::Session (session management)

Please see the METHODS section for a more in-depth look at all Scrappy functionality. (that's the next part bestie, we're getting there 👀)

💜 ATTRIBUTES 💜

The following is a list of object attributes available with every Scrappy instance, attributes always return an instance of the class they represent. so like, you ask for it, you get the actual object, periodt.

🫶 content

The content attribute holds the HTTP::Response object of the current request. Returns undef if no page has been successfully fetched. so if it's empty, that's why — nothing landed yet 😭

my  $scraper = Scrappy->new;
    $scraper->content;

🫶 control

The control attribute holds the Scrappy::Scraper::Control object which is used the provide access conrtol to the scraper. it's literally the bouncer at the club door of your crawl 🚪

my  $scraper = Scrappy->new;
    $scraper->control;

    ... $scraper->control->restrict('google.com');
    ... $scraper->control->allow('cpan.org');
    ... if $scraper->control->is_allowed($url);

🫶 debug

The debug attribute holds a boolean which controls whether event logs are captured.

my  $scraper = Scrappy->new;
    $scraper->debug(1);

🫶 logger

The logger attribute holds the Scrappy::Logger object which is used to provide event logging capabilities to the scraper. it's the diary of your scraper, and it remembers everything 📓

my  $scraper = Scrappy->new;
    $scraper->logger;

🫶 parser

The parser attribute holds the Scrappy::Scraper::Parser object which is used to scrape html data from the specified source material. the main character of the whole scraping situation tbh 👑

my  $scraper = Scrappy->new;
    $scraper->parser;

🫶 plugins

The plugins attribute holds the Scrappy::Plugin object which is an interface used to load plugins.

my  $scraper = Scrappy->new;
    $scraper->plugins;

🫶 queue

The queue attribute holds the Scrappy::Queue object which is used to provide flow-control for the standard loop approach to crawling. line up, everyone, no cutting 🧍‍♀️🧍🧍‍♂️

my  $scraper = Scrappy->new;
    $scraper->queue;

🫶 session

The session attribute holds the Scrappy::Session object which is used to provide session support and persistent data across executions.

my  $scraper = Scrappy->new;
    $scraper->session;

🫶 user_agent

The user_agent attribute holds the Scrappy::Scraper::UserAgent object which is used to set and manipulate the user-agent header of the scraper. it's your little disguise, your fit for the internet 🧥

my  $scraper = Scrappy->new;
    $scraper->user_agent;

🫶 worker

The worker attribute holds the WWW::Mechanize object which is used navigate web pages and provide request and response header information. the one actually doing the walking, we just cheer them on 🫡

my  $scraper = Scrappy->new;
    $scraper->worker;

🚀 METHODS 🚀

🌸 back

The back method is the equivalent of hitting the "back" button in a browser, it returns the previous page (response) and returns that URL, it will not backtrack beyond the first request. so it's giving browser history but with boundaries — one step back, that's it, no time travel 🕰️

my  $scraper = Scrappy->new;

    $scraper->get(...);
    ...
    $scraper->get(...);
    ...
my $last_url = $scraper->back;

🌸 cookies

The cookies method returns an HTTP::Cookie object. Note! Cookies can be made persistent by enabling session-support. Session-support is enable by simply specifying a file to be used. like, one file and suddenly your cookies survive between runs. obsessed 🍪

my  $scraper = Scrappy->new;

    $scraper->session->write('session.yml'); # enable session support
    $scraper->get(...);
my  $cookies = $scraper->cookies;

🌸 crawl

The crawl method is very useful when it is desired to crawl an entire website or at-least partially, it automates the tasks of creating a queue, fetching and parsing html pages, and establishing simple flow-control. See the SYNOPSIS for a simplified example, ... the following is a more complex example. this is the method that does the most, lowkey the overachiever of the group 💅

my  $scrappy = Scrappy->new;

    $scrappy->crawl('http://search.cpan.org/recent',
        '/recent' => {
            '#cpansearch li a' => sub {
                my ($self, $item) = @_;
                # follow all recent modules from search.cpan.org
                $self->queue->add($item->{href});
            }
        },
        '/~:author/:name-:version/' => {
            'body' => sub {
                my ($self, $item, $args) = @_;

                my $reviews = $self
                ->select('.box table tr')->focus(3)->select('td.cell small a')
                ->data->[0]->{text};

                $reviews = $reviews =~ /\d+ Reviews/ ?
                    $reviews : '0 reviews';

                print "found $args->{name} version $args->{version} ".
                    "[$reviews] by $args->{author}\n";
            }
        }
    );

🌸 domain

The domain method returns the domain host of the current page. Local pages, e.g. file:///this/that/the_other will return undef. so local files get nothing back, they're off the grid 🛸

my  $scraper = Scrappy->new;

    $scraper->get('http://www.google.com');
    print $scraper->domain; # print www.google.com

🌸 download

The download method is passed a URL, a Download Directory Path and a optionally a File Path, then it will follow the link and store the response contents into the specified file without leaving the current page. Basically it downloads the contents of the request (especially when the request pushes a file download). If a File Path is not specified, Scrappy will attempt to name the file automatically resorting to a random 6-charater string only if all else fails, then returns to the originating page. so it goes and gets the file, then comes right back like nothing happened 🫶

my  $scaper = Scrappy->new;
my  $requested_url = '...';

    $scraper->download($requested_url, '/tmp');

    # supply your own file name
    $scraper->download($requested_url, '/tmp', 'somefile.txt');

🌸 dumper

The dumper method is a convenience feature that passes the passed-in objects to Data::Dumper which in turn returns a stringified representation of that object/data-structure. perfect for when you're like "what IS this thing actually" and you need to see it 💀

my  $scaper = Scrappy->new;
my  $requested_url = '...';

    $scraper->get($requested_url);

my  $data = $scraper->select('//a[@href]')->data;

# print out the scraped data
print $scraper->dumper($data);

🌸 form

The form method is used to submit a form on the current page.

my  $scraper = Scrappy->new;

    $scraper->form(fields => {
        username => 'mrmagoo',
        password => 'foobarbaz'
    });

    # or more specifically, for pages with multiple forms

    $scraper->form(form_name => 'login_form', fields => {
        username => 'mrmagoo',
        password => 'foobarbaz'
    });

    $scraper->form(form_number => 1, fields => {
        username => 'mrmagoo',
        password => 'foobarbaz'
    });

🌸 get

The get method takes a URL or URI object, fetches a web page and returns the Scrappy object.

my  $scraper = Scrappy->new;

if ($scraper->get($new_url)->page_loaded) {
    ...
}

# $self->content has the HTTP::Response object

🌸 log

The log method logs an event with the event logger.

my  $scraper = Scrappy->new;

    $scraper->debug(1); # unnecessary, on by default
    $scraper->logger->verbose(1); # more detailed log

    $scraper->log('error', 'Somthing bad happened');

    ...

    $scraper->log('info', 'Somthing happened');
    $scraper->log('warn', 'Somthing strange happened');
    $scraper->log('coolness', 'Somthing cool happened');
Note! Event logs are always recorded but never automatically written to a file unless explicitly told to do so using the following:
    $scraper->logger->write('log.yml');

🌸 page_content_type

The page_content_type method returns the content_type of the current page.

my  $scraper = Scrappy->new;
    $scraper->get('http://www.google.com/');
    print $scraper->page_content_type; # prints text/html

🌸 page_data

The page_data method returns the HTML content of the current page, additionally this method when passed a string with HTML markup, updates the content of the current page with that data and returns the modified content. so it reads and writes, a true multi-hyphenate ✨

my  $scraper = Scrappy->new;
    $scraper->get(...);
my  $html = $scraper->page_data;

🌸 page_ishtml

The page_ishtml method returns true/false based on whether our content is HTML, according to the HTTP headers.

my $scraper = Scrappy->new;

    $scraper->get($requested_url);
    if ($scraper->is_html) {
        ...
    }

🌸 page_loaded

The page_loaded method returns true/false based on whether the last request was successful.

my $scraper = Scrappy->new;
           $scraper->get($requested_url);
           if ($scraper->page_loaded) {
               ...
           }

✨ page_match ✨

Okay so page_match is giving route-matching realness. It checks the URL you pass in (or if you leave it empty, it checks the current page's URL) against the URL pattern (aka the route) you defined. If it's a match, it hands you back the parameters of that match, basically the same way a modern web app framework handles URL routes. Slay. 💅

           my $url = 'http://somesite.com/tags/awesomeness';

           ...

           my $scraper = Scrappy->new;

           # match against the current page
           my $this = $scraper->page_match('/tags/:tag');
           if ($this) {
               print $this->{'tag'};
               # ... prints awesomeness
           }

           .. or ..

           # match against a passed url
           my $this = $scraper->page_match('/tags/:tag', $url, {
               host => 'somesite.com'
           });

           if ($this) {
               print "This is the ", $this->{tag}, " page";
               # ... prints this is the awesomeness page
           }

✨ page_reload ✨

Bestie, page_reload is literally the refresh button in your browser. It just repeats the current request. No cap. 🔄

           my  $scraper = Scrappy->new;

               $scraper->get(...);
               ...
               $scraper->reload;

✨ page_status ✨

page_status returns the 3-digit HTTP status code of the response. Periodt. 📟

           my  $scraper = Scrappy->new;
               $scraper->get(...);

               if ($scraper->page_status == 200) {
                   ...
               }

✨ page_text ✨

page_text gives you a text representation of the last page, with all the HTML markup stripped out. It's giving clean, plain vibes. 🧼

           my  $scraper = Scrappy->new;
               $scraper->get(...);

           my  $text = $scraper->page_text;

✨ page_title ✨

page_title returns the content of the title tag if the current page is HTML. If it's not HTML, it returns undef. So if there's no title, don't come for it. 🙈

           my  $scraper = Scrappy->new;
               $scraper->get('http://www.google.com/');

           my  $title = $scraper->page_title;
               print $title; # print Google

✨ pause ✨

pause sets breaks between your requests to simulate human interaction. Like, be a real one, not a bot. 🤖➡️👩

           my  $scraper = Scrappy->new;
               $scraper->pause(20);

               $scraper->get($request_1);
               $scraper->get($request_2);
               $scraper->get($request_3);

In that example, there will be a 20-second break between each request made — get, post, request, etc., You can also specify a range so pause picks randomly:

               $scraper->pause(5,20);

               $scraper->get($request_1);
               $scraper->get($request_2);

               # reset/turn it off
               $scraper->pause(0);

               print "I slept for ", ($scraper->pause), " seconds";
Note! The download method is exempt from any automatic pausing. So it's not like the other girls. 💅

✨ plugin ✨

plugin lets you load a plugin. Using the appropriate case is recommended but not necessary. See Scrappy::Plugin for more info.

           my $scraper = Scrappy->new;

           $scraper->plugin('foo_bar');    # will load Scrappy::Plugin::FooBar
           $scraper->plugin('foo-bar');    # will load Scrappy::Plugin::Foo::Bar
           $scraper->plugin('Foo::Bar');   # will load Scrappy::Plugin::Foo::Bar

           # more pratically
           $scraper->plugin('whois', 'spammer_check');

           ... somewhere in code

           my $var = $scraper->plugin_method();

           # example using core plugin Scrappy::Plugin::RandomProxy

           my  $s = Scrappy->new;

               $s->plugin('random_proxy');
               $s->use_random_proxy;

               $s->get(...);

✨ post ✨

post takes a URL, a hashref of key/value pairs, and optionally an array of key/value pairs, then posts that data to the specified URL and returns an HTTP::Response object. It's giving HTTP realness. 📮

           my $scraper = Scrappy->new;

           $scraper->post($requested_url, {
               input_a => 'value_a',
               input_b => 'value_b'
           });

           # w/additional headers
           my %headers = ('Content-Type' => 'multipart/form-data');
           $scraper->post($requested_url, {
               input_a => 'value_a',
               input_b => 'value_b'
           },  %headers);
Note! The most common post headers for content-type are application/x-www-form-urlencoded and multipart/form-data.

✨ proxy ✨

proxy sets the proxy for the next request to be tunneled through. Sneaky link energy. 🕵️‍♀️

           my $scraper = Scrappy->new;

           $scraper->proxy('http', 'http://proxy1.example.com:8000/');
           $scraper->get($requested_url);

           $scraper->proxy('http', 'ftp', 'http://proxy2.example.com:8000/');
           $scraper->get($requested_url);

           # best practice when using proxies

           use Tiny::Try;

           my $proxie = Scrappy->new;

           $proxie->proxy('http', 'http://proxy.example.com:8000/');

           try {
               $proxie->get($requested_url);
           } catch {
               die "Proxy failed\n";
           };
Note! When using a proxy to perform requests, be aware that if they fail your program will die unless you wrap your code in an eval statement or use a try/catch mechanism. In the example above we use Tiny::Try to trap any errors that might occur when using proxy.

✨ request_denied ✨

request_denied is a simple shortcut to figure out if the page you requested actually loaded or got redirected. It's super useful on systems that require authentication and redirect you if you're not authorized. This function returns a boolean: 1 if the current page doesn't match the requested page.

           my $scraper = Scrappy->new;
           $scraper->get($url_to_dashboard);

           if ($scraper->request_denied) {
               # do login, again
           }
           else {
               # resume ...
           }

✨ response ✨

response returns the HTTP::Repsonse object of the current page.

           my  $scraper = Scrappy->new;
               $scraper->get(...);
           my  $res = $scraper->response;

✨ select ✨

select takes XPATH or CSS selectors and returns a Scrappy::Scraper::Parser object containing the matching elements. It's giving DOM manipulation, but make it fashion. 👑

           my $scraper = Scrappy->new;

           # return a list of links
           my $list = $scraper->select('#profile li a')->data; # see Scrappy::Scraper::Parser

           foreach my $link (@{$list}) {
               print $link->{href}, "\n";
           }

           # Zoom in on specific chunks of html code using the following ...
           my $list = $scraper
           ->select('#container table tr') # select all rows
           ->focus(4) # focus on the 5th row
           ->select('div div')->data;

           # The code above selects the div > div inside of the 5th tr in #container table
           # Access attributes html, text and other attributes as follows...

           $element = $scraper->select('table')->data->[0];
           $element->{html}; # HTML representation of the table
           $element->{text}; # Table stripped of all HTML
           $element->{cellpadding}; # cellpadding
           $element->{height}; # ...

✨ stash ✨

stash sets a stash (shared) variable or returns a reference to the entire stash object. It's like a little purse for your data. 👜

           my  $scraper = Scrappy->new;
               $scraper->stash(age => 31);

               print 'stash access works'
                   if $scraper->stash('age') == $scraper->stash->{age};

           my  @array = (1..20);
               $scraper->stash(integers => [@array]);

✨ store ✨

store saves the contents of the current page into the specified file. If the content-type does not begin with 'text', the content is saved as binary data. So if it's not text, it's giving binary realness. 💾

           my  $scraper = Scrappy->new;

               $scraper->get($requested_url);
               $scraper->store('/tmp/foo.html');

✨ url ✨

url returns the complete URL for the current page. So you always know where you at. 📍

           my  $scraper = Scrappy->new;
               $scraper->get('http://www.google.com/');
               print $scraper->url; # prints http://www.google.com/

✨ AUTHOR ✨

Al Newkirk <awncorp@cpan.org> — the icon, the legend, the moment. 👑

This software is copyright (c) 2010 by awncorp.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. So go off, bestie, but respect the license. 📜

That's the whole vibe! This page is from Scrappy(3pm), Perl v5.34.0, dated 2022-06-17. Stay scrappy! 💖✨

📦 where this page comes from