HTTP::Async(3pm)

📦
version 0.33 · 3 versions
, in ubuntu noble, ubuntu resolute, ubuntu stonking

🦋 HTTP::Async — the HTTP::Async(3pm) bestie guide 🦋

User Contributed Perl Documentation · the Perl module that does a million HTTP things at once and never blocks you, periodt

✨ NAME ✨

HTTP::Async — process multiple HTTP requests in parallel without blocking. That's it, that's the whole vibe. 💅

🌸 SYNOPSIS 🌸

okay so step one, you make an object and you dump some requests in it:

use HTTP::Async;
my $async = HTTP::Async->new;

# create some requests and add them to the queue.
$async->add( HTTP::Request->new( GET => 'http://www.perl.org/'         ) );
$async->add( HTTP::Request->new( GET => 'http://www.ecclestoad.co.uk/' ) );

and then you've got OPTIONS bestie. EITHER you deal with the responses as they roll back in:

while ( my $response = $async->wait_for_next_response ) {
    # Do some processing with $response
}

OR you go do something else entirely whenever there's no response ready:

while ( $async->not_empty ) {
    if ( my $response = $async->next_response ) {
        # deal with $response
    } else {
        # do something else
    }
}

OR — and honestly? iconic behaviour — you just let the async object fetch stuff in the background while you live your life, and deal with the responses at the very end:

# Do some long code...
for ( 1 .. 100 ) {
  some_function();
  $async->poke;            # lets it check for incoming data.
}

while ( my $response = $async->wait_for_next_response ) {
    # Do some processing with $response
}

💖 DESCRIPTION 💖

ngl, using the conventional LWP::UserAgent is fast and easy and we love her, but she does have some drawbacks — your code execution blocks until the request is done, and you can only process one request at a time. HTTP::Async said "not on my watch" and set out to fix exactly those two things.

It hands you an 'Async' object. You add requests to it, and you scoop the requests back off as they finish. All the actual sending and receiving is abstracted away from you, king. The moment you add a request it gets transmitted — and if there are already too many requests in progress right then, they just get queued up. There is no concept of starting or stopping — it runs continuously. It's giving always-on.

And while it's waiting around for data it hands control right back to whatever code called it, so you can keep doing your own processing while it's pulling data off the network. All of this with zero forking and zero threading — no cap, it's actually done using select lists. 🧚

⭐ Default settings ⭐

There are a bunch of default settings that should be suitable for most uses. However, in some circumstances you might wish to change these. (Valid, get specific.)

            slots:  20
          timeout:  180 (seconds)
 max_request_time:  300 (seconds)
    max_redirect:   7
    poll_interval:  0.05 (seconds)
       proxy_host:  ''
       proxy_port:  ''
       local_addr:  ''
       local_port:  ''
      ssl_options: {}
       cookie_jar:  undef
        peer_addr:  ''

If defined, cookie_jar is expected to be similar to HTTP::Cookies, with extract_cookies and add_cookie_header methods.

The option max_redirects has been renamed to max_redirect to be consistent with LWP::UserAgent, although max_redirects still works. (Legacy support, we respect it. 👑)

🎀 METHODS 🎀

🌈 new

my $async = HTTP::Async->new( %args );

Creates a new HTTP::Async object and sets it up. Variations from the default can be set by passing them in as %args.

$old_value = $async->slots;
$new_value = $async->slots( $new_value );

Get/setters for the $async object's config settings. timeout is for inactivity and is in seconds.

slots is the maximum number of parallel requests to make.

🌈 add

my @ids      = $async->add(@requests);
my $first_id = $async->add(@requests);

Adds requests to the queues. Each request gets a unique integer id (unique for this $async) that you can use to track the requests if you need to. Call it in list context and you get an array of ids back; call it in scalar context and you get the id of the first request added. Slay.

🌈 add_with_opts

my $id = $async->add_with_opts( $request, \%opts );

This method lets you add a single request to the queue with options that differ from the defaults. Say you want a longer timeout, or a specific proxy — this is the girl for that job. Returns the id of the request.

Warning bestie: the method croaks when you pass it an invalid option. It will not let that slide.

🌈 poke

$async->poke;

At fairly frequent intervals some housekeeping needs to be performed — stuff like reading received data and starting new requests. Calling poke lets the object do exactly that and then return quickly. Usually you won't need this, because most other methods do it for you.

You should use poke if your code is spending time elsewhere (ie. not using the async object), so the object can keep the data flowing over the network. If you don't use it, the buffers may fill up and completed responses will not be replaced with new requests. 💀 So like, poke your object. It needs attention.

🌈 next_response

my $response          = $async->next_response;
my ( $response, $id ) = $async->next_response;

Returns the next response (as a HTTP::Response object) that is waiting, or returns undef if there is none. In list context it returns a (response, id) pair, or an empty list if there's none. It does not wait for a response, so it comes back very quickly. Zoom.

🌈 wait_for_next_response

my $response          = $async->wait_for_next_response( 3.5 );
my ( $response, $id ) = $async->wait_for_next_response( 3.5 );

Same energy as next_response but it only returns if there is a next response, or once the time in seconds you passed in has elapsed. If you don't give it a time then it blocks. Whilst waiting it checks the queues every poll_interval seconds. The times can be fractional seconds. Yes, 3.5 is legal. 🕐

🌈 to_send_count

my $pending = $async->to_send_count;

Returns the number of items which have been added but have not yet started being processed.

🌈 to_return_count

my $completed = $async->to_return_count;

Returns the number of items which have completed transferring, and are waiting to be returned by next_response().

🌈 in_progress_count

my $running = $async->in_progress_count;

Returns the number of items which are currently being processed asynchronously. Main character energy, all of them, simultaneously.

🌈 total_count

my $total = $async->total_count;

Returns the sum of the to_send_count, in_progress_count and to_return_count.

This should be the total number of items which have been added that have not yet been returned by next_response().

🌈 info

print $async->info;

Prints a line describing what the current state is. Perfect for when you're like "girl what is happening in there".

🌈 remove

$async->remove($id);
my $success = $async->remove($id);

Removes the item with the given id no matter which state it is currently in. Returns true if an item is removed, and false otherwise.

🌈 remove_all

$async->remove_all;
my $success = $async->remove_all;

Removes all items no matter what states they are currently in. Returns true if any items are removed, and false otherwise.

🌈 empty, not_empty

while ( $async->not_empty ) { ...; }
while (1) { ...; last if $async->empty; }

Returns true or false depending on whether there are request or responses still on the object.

🌈 DESTROY

The destroy method croaks if an object is destroyed but is not empty. This is to help with debugging. (It's not being mean, it's being honest. 💜)

🔥 SEE ALSO 🔥

  • HTTP::Async::Polite — a polite form of this module. Slows the scraping down by domain so that the remote server is not overloaded. Manners! We love to see it.

💀 GOTCHAS 💀

The responses may not come back in the same order as the requests were made. Just so you know.

For https requests to work, you must have Net::HTTPS::NB installed. No that module, no https. Non-negotiable.

🫶 THANKS 🫶

Shoutouts to everyone who made this module what it is, fr fr:

  • Egor Egorov contributed patches for proxies, catching connections that die before headers sent and more.
  • Tomohiro Ikebe from livedoor.jp submitted patches (and a test) to properly handle 304 responses.
  • Naveed Massjouni for adding the https handling code.
  • Alex Balhatchet for adding the https + proxy handling code, and for making the tests run ok in parallel.
  • Josef Toman for fixing two bugs, one related to header handling and another related to producing an absolute URL correctly.
  • Github user 'c00ler-' for adding LocalAddr and LocalPort support.
  • rt.cpan.org user 'Florian (fschlich)' for typo in documentation.
  • Heikki Vatiainen for the ssl_options support patch.
  • Daniel Lintott of the Debian Perl Group for pointing out a test failure when using a very recent version of HTTP::Server::Simple to implement t/TestServer.pm

🐛 BUGS AND REPO 🐛

Please submit all bugs, patches etc on github

<https://github.com/evdb/HTTP-Async>

👑 AUTHOR 👑

Edmund von der Burg "<evdb@ecclestoad.co.uk>".

<http://www.ecclestoad.co.uk/>

Copyright (c) 2006, Edmund von der Burg "<evdb@ecclestoad.co.uk>". All rights reserved.

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

🙈 DISCLAIMER OF WARRANTY 🙈

okay so this part is in ALL CAPS because it's the legal bit — it means what it says, and I'm keeping every word:

BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENCE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

and that's the whole page bestie 🫶 this one's from perl v5.36.0, dated 2022-10-15, and it's HTTP::Async(3pm) — go forth and fetch in parallel, you absolute icon ✨💖

📦 where this page comes from