Scrapp...Control(3pm)

💖 NAME 💖

okay so this is the manual page for Scrapp...Control(3pm) — aka Scrappy::Scraper::Control, the Scrappy HTTP Request Constraints System. she decides who's allowed in and who gets left on read 🚪✨

---

🌈 VERSION 🌈

version 0.94112090 — yes bestie, that's the whole number, she's got range 💅

---

✨ SYNOPSIS ✨

here's the vibe, copy-paste ready:

#!/usr/bin/perl
use Scrappy::Scraper::Control;

my  $control = Scrappy::Scraper::Control->new;

    $control->allow('http://search.cpan.org');
    $control->allow('http://search.cpan.org', if => {
            content_type => ['text/html', 'application/x-tar']
        }
    );

    $control->restrict('http://www.cpan.org');

    if ($control->is_allowed('http://search.cpan.org/')) {
        ...
    }

    # constraints will only be checked if the is_allowed method is
    # passed a HTTP::Response object.

lowkey important: those constraints? they only get checked if is_allowed gets handed an HTTP::Response object. no object, no vibe check, iykyk 👑

---

🎀 DESCRIPTION 🎀

Scrappy::Scraper::Control hands out HTTP request access control for the Scrappy framework. think of it as the bouncer at the club but the club is your scraper and the bouncer is literally a Perl module 🪩

🦋 ATTRIBUTES 🦋

every single Scrappy::Scraper::Control instance comes with these object attributes. collect them all fr fr:

allowed

the allowed attribute holds a hashref of allowed domain/constraints 🫶

my  $control = Scrappy::Scraper::Control->new;
    $control->allowed;

    e.g.

    {
        'www.foobar.com' => {
            methods => [qw/GET POST PUSH PUT DELETE/],
            content_type => ['text/html']
        }
    }

restricted

the restricted attribute holds a hashref of restricted domain/constraints 🙈

my  $control = Scrappy::Scraper::Control->new;
    $control->restricted;

    e.g.

    {
        'www.foobar.com' => {
            methods => [qw/GET POST PUSH PUT DELETE/]
        }
    }

---

💜 METHODS 💜

allow

say yes to a domain, optionally with conditions attached (she's picky, respectfully):

my  $control = Scrappy::Scraper::Control->new;
    $control->allow('http://www.perl.org');
    $control->allow('http://search.cpan.org', if => {
            content_type => ['text/html', 'application/x-tar']
        }
    );

restrict

say nope to a domain, also optionally with conditions. boundaries are iconic ✋✨

my  $control = Scrappy::Scraper::Control->new;
    $control->restrict('http://www.perl.org');
    $control->restrict('http://search.cpan.org', if => {
            content_type => ['text/html', 'application/x-tar']
        }
    );

is_allowed

the actual vibe check 👀 returns whether a URL is allowed, and remember — constraints only get evaluated when you pass in an HTTP::Response object.

my  $control = Scrappy::Scraper::Control->new;
    $control->allow('http://search.cpan.org');
    $control->restrict('http://www.perl.org');

    if (! $control->is_allowed('http://perl.org')) {
        die 'Cant get to Perl.org';
    }

not me using this to block http://perl.org in the example… the audacity 😭

---

👑 AUTHOR 👑

Al Newkirk <awncorp@cpan.org> — the icon, the legend, the main character 🌟

---

🌸 COPYRIGHT AND LICENSE 🌸

This software is copyright (c) 2010 by awncorp.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. 🫶

---

aaaand that's the page bestie 💖 this was perl v5.34.0, dated 2022-06-17, straight out of User Contributed Perl Documentation. go forth and control your scraper, periodt 💅🎀