lwpcook(3pm)

✨ lwpcook — the libwww-perl cookbook ✨

okay bestie, pull up a chair 💖 this is lwpcook, part of the User Contributed Perl Documentation situation, and it's literally a whole cookbook of recipes for the libwww-perl library. no cap.

---

🌸 DESCRIPTION 🌸

This doc is a lil collection of examples showing how you'd typically use the libwww-perl library. Real talk: if you want the deep lore on any single module, go read that module's own docs, this is the vibes-level tour 🫶

Every example here is a runnable program, fr fr. And in most cases you can just pipe the program text straight into perl and test it. iconic behaviour honestly.

---

💅 GET 💅

Using this library to just grab documents off the net is so easy. The LWP::Simple module gives you the get() function, which returns the document at whatever URL you hand it:

use LWP::Simple;
$doc = get 'http://search.cpan.org/dist/libwww-perl/';

or, if you're in your perl one-liner era, use getprint():

perl -MLWP::Simple -e 'getprint "http://search.cpan.org/dist/libwww-perl/"'

or — hear me out — how about fetching the latest perl by running this:

perl -MLWP::Simple -e '
  getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
           "perl.tar.gz"'

First tho? You probably wanna find a CPAN site closer to you by running something like:

perl -MLWP::Simple -e 'getprint "http://www.cpan.org/SITES.html"'

okay enough of the simple stuff, we're levelling up 🧚 The LWP object oriented interface gives you way more control over the request you send to the server. With this interface you have full control over the headers you send and how you wanna handle the response that comes back.

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("$0/0.1 " . $ua->agent);
# $ua->agent("Mozilla/8.0") # pretend we are very capable browser

$req = HTTP::Request->new(
   GET => 'http://search.cpan.org/dist/libwww-perl/');
$req->header('Accept' => 'text/html');

# send request
$res = $ua->request($req);

# check the outcome
if ($res->is_success) {
   print $res->decoded_content;
}
else {
   print "Error: " . $res->status_line . "\n";
}

Alsooo the lwp-request program (alias GET) that ships with the library can fetch documents from WWW servers too. two paths, same destination, we love options 💜

---

🎀 HEAD 🎀

If all you wanna know is whether a document is there (i.e. the URL is valid), try running code like this:

use LWP::Simple;

if (head($url)) {
   # ok document exists
}

ngl, head() is doing more than you think — it really returns a list of meta-information about the document. The first three values of that returned list are the document type, the size of the document, and the age of the document. ✨

Want more control over the request, or access to all the header values that came back? Then you need the object oriented interface described up in GET. Just s/GET/HEAD/g. that's it, that's the trick 💅

---

🔥 POST 🔥

okay so there's no simple procedural interface for posting data to a WWW server. You must use the object oriented interface for this one, bestie. The most common POST operation is hitting a WWW form application:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(
    POST => 'https://rt.cpan.org/Public/Dist/Display.html');
$req->content_type('application/x-www-form-urlencoded');
$req->content('Status=Active&Name=libwww-perl');

my $res = $ua->request($req);
print $res->as_string;

Lazy people (respectfully, me) use the HTTP::Request::Common module to set up a suitable POST request message. It handles all the escaping issues and has a suitable default for the content_type:

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
$ua = LWP::UserAgent->new;

my $req = POST 'https://rt.cpan.org/Public/Dist/Display.html',
              [ Status => 'Active', Name => 'libwww-perl' ];

print $ua->request($req)->as_string;

And the lwp-request program (alias POST) that comes with the library can post data too 🫶

---

🌈 PROXIES 🌈

Some sites use proxies to go through fire wall machines, or just as cache to improve performance. Proxies can also be used for accessing resources through protocols that libwww-perl doesn't support directly — or supports badly :-) — so yeah, they're the moment.

You should initialize your proxy setting before you start sending requests:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->env_proxy; # initialize from environment variables
# or
$ua->proxy(ftp  => 'http://proxy.myorg.com');
$ua->proxy(wais => 'http://proxy.myorg.com');
$ua->no_proxy(qw(no se fi));

my $req = HTTP::Request->new(GET => 'wais://xxx.com/');
print $ua->request($req)->as_string;

Heads up: the LWP::Simple interface will call env_proxy() for you automatically. And applications that use the $ua->env_proxy() method will normally not use the $ua->proxy() and $ua->no_proxy() methods. pick a lane basically 💀

Some proxies also require that you send a username/password before they let requests through. You should be able to add the required header with something like this:

use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');

$req = HTTP::Request->new('GET',"http://www.perl.com");

$res = $ua->request($req);
print $res->decoded_content if $res->is_success;

Replace "proxy.myorg.com", "username" and "password" with something suitable for your site. don't be out here shipping placeholder creds, bestie 😭

---

👑 ACCESS TO PROTECTED DOCUMENTS 👑

Documents protected by basic authorization can easily be accessed like this:

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => 'http://www.linpro.no/secret/');
$req->authorization_basic('aas', 'mypassword');
print $ua->request($req)->as_string;

The other alternative is to provide a subclass of LWP::UserAgent that overrides the get_basic_credentials() method. Study the lwp-request program for an example of this. go read the source, it's giving main character energy ✨

---

🍓 COOKIES 🍓

Some sites like to play games with cookies. By default LWP ignores cookies provided by the servers it visits. LWP will collect cookies and respond to cookie requests if you set up a cookie jar. LWP doesn't provide a cookie jar itself, but if you install HTTP::CookieJar::LWP, it can be used like this:

use LWP::UserAgent;
use HTTP::CookieJar::LWP;

$ua = LWP::UserAgent->new(
    cookie_jar => HTTP::CookieJar::LWP->new,
);

# and then send requests just as you used to do
$res = $ua->request(HTTP::Request->new(GET => "http://no.yahoo.com/"));
print $res->status_line, "\n";

cookie jar acquired 🍪 the vibes are immaculate.

---

🔒 HTTPS 🔒

URLs with the https scheme are accessed in exactly the same way as with the http scheme — provided that an SSL interface module for LWP has been properly installed (peep the README.SSL file in the libwww-perl distribution for more details, iykyk). If no SSL interface is installed for LWP to use, then you'll get "501 Protocol scheme 'https' is not supported" errors when accessing such URLs. 💀

Here's an example of fetching and printing a WWW page using SSL:

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => 'https://www.helsinki.fi/');
my $res = $ua->request($req);
if ($res->is_success) {
    print $res->as_string;
}
else {
    print "Failed: ", $res->status_line, "\n";
}

---

🦋 MIRRORING 🦋

If you wanna mirror documents from a WWW server, then try running code similar to this at regular intervals:

use LWP::Simple;

%mirrors = (
   'http://www.sn.no/'                       => 'sn.html',
   'http://www.perl.com/'                    => 'perl.html',
   'http://search.cpan.org/distlibwww-perl/' => 'lwp.html',
   'gopher://gopher.sn.no/'                  => 'gopher.html',
);

while (($url, $localfile) = each(%mirrors)) {
   mirror($url, $localfile);
}

Or, as a perl one-liner:

perl -MLWP::Simple -e 'mirror("http://www.perl.com/", "perl.html")';

Key detail, bestie: the document will not be transferred unless it has been updated. efficiency queen 💅

---

🧚 LARGE DOCUMENTS 🧚

If the document you wanna fetch is too large to be kept in memory, then you have two alternatives. Option one: you can instruct the library to write the document content to a file (the second $ua->request() argument is a file name):

use LWP::UserAgent;
$ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(GET =>
    'http://www.cpan.org/CPAN/authors/id/O/OA/OALDERS/libwww-perl-6.26.tar.gz');
$res = $ua->request($req, "libwww-perl.tar.gz");
if ($res->is_success) {
   print "ok\n";
}
else {
   print $res->status_line, "\n";
}

Option two: you can process the document as it arrives (the second $ua->request() argument is a code reference):

use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$URL = 'ftp://ftp.isc.org/pub/rfc/rfc-index.txt';

my $expected_length;
my $bytes_received = 0;
my $res =
   $ua->request(HTTP::Request->new(GET => $URL),
             sub {
                 my($chunk, $res) = @_;
                 $bytes_received += length($chunk);
                 unless (defined $expected_length) {
                    $expected_length = $res->content_length || 0;
                 }
                 if ($expected_length) {
                      printf STDERR "%d%% - ",
                                100 * $bytes_received / $expected_length;
                 }
                 print STDERR "$bytes_received bytes received\n";

                 # XXX Should really do something with the chunk itself
                 # print $chunk;
             });
 print $res->status_line, "\n";

streaming that data chunk by chunk, no memory anxiety, we love to see it ✨

---

💖 COPYRIGHT 💖

Copyright 1996-2001, Gisle Aas

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

---

and that's the whole cookbook, bestie 🫶 this was lwpcook(3pm) from perl v5.34.0, dated 2026-05-22, out here living its best life in the User Contributed Perl Documentation. go forth and fetch responsibly 💅🎀