Scrappy::Queue(3pm)

💖 Scrappy::Queue — the manual page, bestie edition 💖

okay so this is Scrappy::Queue(3pm), straight out of the User Contributed Perl Documentation section, and it's literally about making your HTTP requests chill out and queue up nicely. Yes queen, flow control. Periodt.

---

✨ NAME ✨

Scrappy::Queue — the Scrappy HTTP Request Flow-Control System 🌈

🎀 VERSION 🎀

version 0.94112090 (iconic numbers, no notes)

🦋 SYNOPSIS 🦋

Like, here's the whole vibe in one go:

#!/usr/bin/perl
use Scrappy::Queue;

my  $queue = Scrappy::Queue->new;

    $queue->add($url);

    while (my $url = $queue->next) {
        ... $queue->add(...);
    }

See how she's adding while she's looping? That's the whole point, bestie. The queue feeds itself. Main character energy fr fr. 👑

🌸 DESCRIPTION 🌸

Scrappy::Queue gives you a whole system for saving URLs into a recordset/queue, and then iterating over them using the Scrappy framework. That's it, that's the gig. Store URLs, walk through them, no cap. 💅

---

💅 METHODS 💅

🍓 list

The list method returns the list of URLs that are in the queue. Heads up: this comes back in list context, so assign it to an array or don't bother.

my  $queue = Scrappy::Queue->new;

...

my  @list = $queue->list;

🍓 add

The add method adds new URLs to the queue. And listen — duplicate URLs will be ignored. She does not do repeats, she does not do reruns. ✨

my  $queue = Scrappy::Queue->new;
    $queue->add($url);

🍓 clear

The clear method completely empties the queue and resets the cursor (that's the loop position, keep up 🫶). Full factory reset, but for your queue.

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    $queue->clear;

🍓 reset

The reset method resets the cursor (the loop position). It does not empty the queue — just rewinds where you're standing. Different method, different job, don't mix them up bestie. 💜

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    while (my $url = $queue->next) {
        $queue->reset if ...; # beware the infinate loop
    }

    $queue->reset;

⚠️ REAL TALK: look at that comment. # beware the infinate loop. If you reset the cursor inside the while loop, you're gonna spin forever. The manpage said it (typo and all) and I'm saying it louder: beware. 💀

🍓 current

The current method returns the URL sitting at the current loop position. Where you at right now? This tells you. 📍

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    while (my $url = $queue->next) {
        last if ...;
    }

    print 'great' if $url eq $queue->current;

🍓 next

The next method moves the cursor to the next loop position and returns the URL. This is the one doing the heavy lifting in basically every example on this page. She's the moment. 🌟

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    while (my $url = $queue->next) {
        ...
    }

🍓 previous

The previous method moves the cursor to the previous loop position and returns the URL. Yes, you can go back. Time travel, but make it URLs. ⏪

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    while (my $url = $queue->next) {
        ...
    }

    print $queue->previous;

🍓 first

The first method moves the cursor to the first loop position and returns the URL. Back to the beginning, bestie. 🏁

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    print $queue->first;

🍓 last

The last method moves the cursor to the last loop position and returns the URL. Jump straight to the end, no walking required. 🏁✨

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    print $queue->last;

🍓 index

The index method moves the cursor to the specified loop position and returns the URL. And here's the tea: the loop position is a standard array index position. So 0 is the first one, 1 is the second one, etc. You know the drill. 🔢

my  $queue = Scrappy::Queue->new;

    $queue->add(...);
    $queue->add(...);
    $queue->add(...);

    print $queue->index(1);

🍓 cursor

The cursor method returns the current loop position. Just the position, not the URL. Pure coordinates. 🧭

my  $queue = Scrappy::Queue->new;
    print $queue->cursor;

---

👑 AUTHOR 👑

Al Newkirk <awncorp@cpan.org> — thank you for your service, king 🫡💖

📜 COPYRIGHT AND LICENSE 📜

This software is copyright (c) 2010 by awncorp.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. Slay, honestly. 🌈

---

and that's the whole page bestie!! 💖 This gorgeous little document ships as perl v5.34.0, dated 2022-06-17, and it lives in the Scrappy::Queue(3pm) slot of your friendly neighbourhood User Contributed Perl Documentation. Go queue some URLs. You got this. 🦋✨🎀