lwptut(3pm)

📦
version 6.61 · 3 versions
, in ubuntu jammy

lwptut 💖 — an LWP tutorial, bestie edition

okay so this is lwptut(3pm) from the User Contributed Perl Documentation, and it's literally the glow-up guide to LWP. grab a snack, we're going on a journey 🦋

✨ NAME ✨

lwptut — An LWP Tutorial 🌈

🌸 DESCRIPTION 🌸

LWP (short for "Library for WWW in Perl") is a very popular group of Perl modules for accessing data on the Web. like, everyone uses it. 💅

So here's the tea: like most Perl module-distributions, every component module of LWP ships with its own docs that are a complete reference to its interface. cute, right? but bestie, there are so many modules in LWP that it's genuinely hard to know where to even start looking for info on how to do the simplest, most common lil things. 😭

fr fr, really introducing you to LWP would take a whole book — and plot twist, that book exists, it's called Perl & LWP. 📚 But this article? it's just here to give you a taste of how you can go about some common tasks with LWP. snack-sized. obsessed.

🎀 Getting documents with LWP::Simple 🎀

if you literally just want to get what's at a particular URL, the simplest way is LWP::Simple's functions. no notes. ✨

In a Perl program, you can call its get($url) function. it'll try getting that URL's content. if it works, it returns the content; but if there's some error, it returns undef. that's the whole vibe check. 🙈

my $url = 'http://www.npr.org/programs/fa/?todayDate=current';
  # Just an example: the URL for the most recent /Fresh Air/ show

use LWP::Simple;
my $content = get $url;
die "Couldn't get $url" unless defined $content;

# Then go do things with $content, like this:

if($content =~ m/jazz/i) {
  print "They're talking about jazz today on Fresh Air!\n";
}
else {
  print "Fresh Air is apparently jazzless today.\n";
}

The handiest variant on get is getprint, which is useful in Perl one-liners. if it can get the page whose URL you provide, it sends it to STDOUT; otherwise it complains to STDERR. iconic behaviour honestly.

% perl -MLWP::Simple -e "getprint 'http://www.cpan.org/RECENT'"

That's the URL of a plain text file that lists new files in CPAN in the past two weeks. and you can easily make it part of a tidy lil shell command, like this one that mails you the list of new Acme:: modules:

% perl -MLWP::Simple -e "getprint 'http://www.cpan.org/RECENT'"  \
   | grep "/by-module/Acme" | mail -s "New Acme modules! Joy!" $USER

There are other useful functions in LWP::Simple too, bestie — including one function for running a HEAD request on a URL (useful for checking links, or getting the last-revised time of a URL), and two functions for saving/mirroring a URL to a local file. See the LWP::Simple documentation for the full details, or chapter 2 of Perl & LWP for more examples. 💖

🦋 The Basics of the LWP Class Model 🦋

LWP::Simple's functions are handy for simple cases, but lowkey its functions don't support cookies or authorization, don't support setting header lines in the HTTP request, and generally don't support reading header lines in the HTTP response (notably the full HTTP error message, in case of an error). so if you want all those features, you're gonna have to use the full LWP class model. main character energy only. 👑

While LWP consists of dozens of classes, the main two that you have to understand are LWP::UserAgent and HTTP::Response. LWP::UserAgent is a class for "virtual browsers" which you use for performing requests, and HTTP::Response is a class for the responses (or error messages) that you get back from those requests. match made in heaven fr.

The basic idiom is $response = $browser->get($url), or more fully illustrated:

# Early in your program:

use LWP 5.64; # Loads all important LWP classes, and makes
              #  sure your version is reasonably recent.

my $browser = LWP::UserAgent->new;

...

# Then later, whenever you need to make a get request:
my $url = 'http://www.npr.org/programs/fa/?todayDate=current';

my $response = $browser->get( $url );
die "Can't get $url -- ", $response->status_line
 unless $response->is_success;

die "Hey, I was expecting HTML, not ", $response->content_type
 unless $response->content_type eq 'text/html';
   # or whatever content-type you're equipped to deal with

# Otherwise, process the content somehow:

if($response->decoded_content =~ m/jazz/i) {
  print "They're talking about jazz today on Fresh Air!\n";
}
else {
  print "Fresh Air is apparently jazzless today.\n";
}

There are two objects involved: $browser, which holds an object of class LWP::UserAgent, and then the $response object, which is of class HTTP::Response. You really need only one browser object per program; but every time you make a request, you get back a new HTTP::Response object, which will have some interesting attributes:

  • A status code indicating success or failure (which you can test with $response->is_success).
  • An HTTP status line that is hopefully informative if there's failure (which you can see with $response->status_line, returning something like 404 Not Found).
  • A MIME content-type like text/html, image/gif, application/xml, etc., which you can see with $response->content_type
  • The actual content of the response, in $response->decoded_content. if the response is HTML, that's where the HTML source will be; if it's a GIF, then $response->decoded_content will be the binary GIF data. 🖼️
  • And dozens of other convenient and more specific methods that are documented in the docs for HTTP::Response, and its superclasses HTTP::Message and HTTP::Headers.

💅 Adding Other HTTP Request Headers 💅

The most commonly used syntax for requests is $response = $browser->get($url), but in truth, you can add extra HTTP header lines to the request by adding a list of key-value pairs after the URL, like so:

$response = $browser->get( $url, $key1, $value1, $key2, $value2, ... );

For example, here's how to send some commonly used headers, in case you're dealing with a site that would otherwise reject your request (rude of them, honestly):

my @ns_headers = (
 'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
 'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',
 'Accept-Charset' => 'iso-8859-1,*,utf-8',
 'Accept-Language' => 'en-US',
);

...

$response = $browser->get($url, @ns_headers);

If you weren't reusing that array, you could just go ahead and do this:

$response = $browser->get($url,
 'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
 'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',
 'Accept-Charset' => 'iso-8859-1,*,utf-8',
 'Accept-Language' => 'en-US',
);

If you were only ever changing the 'User-Agent' line, you could just change the $browser object's default line from libwww-perl/5.65 (or the like) to whatever you like, using the LWP::UserAgent agent method:

$browser->agent('Mozilla/4.76 [en] (Win98; U)');

🍓 Enabling Cookies 🍓

A default LWP::UserAgent object acts like a browser with its cookies support turned off. There are various ways of turning it on, by setting its cookie_jar attribute. A "cookie jar" is an object representing a little database of all the HTTP cookies that a browser knows about. It can correspond to a file on disk or an in-memory object that starts out empty, and whose collection of cookies will disappear once the program is finished running. 🍪✨

To give a browser an in-memory empty cookie jar, you set its cookie_jar attribute like so:

use HTTP::CookieJar::LWP;
$browser->cookie_jar( HTTP::CookieJar::LWP->new );

To save a cookie jar to disk, see dump_cookies in HTTP::CookieJar. To load cookies from disk into a jar, see load_cookies in HTTP::CookieJar.

🌈 Posting Form Data 🌈

Many HTML forms send data to their server using an HTTP POST request, which you can send with this syntax:

$response = $browser->post( $url,
  [
    formkey1 => value1,
    formkey2 => value2,
    ...
  ],
);

Or if you need to send HTTP headers:

$response = $browser->post( $url,
  [
    formkey1 => value1,
    formkey2 => value2,
    ...
  ],
  headerkey1 => value1,
  headerkey2 => value2,
);

For example, the following program makes a search request to AltaVista (by sending some form data via an HTTP POST request), and extracts from the HTML the report of the number of matches:

use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;

my $word = 'tarragon';

my $url = 'http://search.yahoo.com/yhs/search';
my $response = $browser->post( $url,
  [ 'q' => $word,  # the Altavista query string
    'fr' => 'altavista', 'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',
  ]
);
die "$url error: ", $response->status_line
 unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type
 unless $response->content_is_html;

if( $response->decoded_content =~ m{([0-9,]+)(?:<.*?>)? results for} ) {
  # The substring will be like "996,000</strong> results for"
  print "$word: $1\n";
}
else {
  print "Couldn't find the match-string in the response\n";
}

🧚 Sending GET Form Data 🧚

Some HTML forms convey their form data not by sending the data in an HTTP POST request, but by making a normal GET request with the data stuck on the end of the URL. like, sneaky but valid. 👀

For example, if you went to www.imdb.com and ran a search on "Blade Runner", the URL you'd see in your browser window would be:

http://www.imdb.com/find?s=all&q=Blade+Runner

To run the same search with LWP, you'd use this idiom, which involves the URI class:

use URI;
my $url = URI->new( 'http://www.imdb.com/find' );
  # makes an object representing the URL

$url->query_form(  # And here the form data pairs:
  'q' => 'Blade Runner',
  's' => 'all',
);

my $response = $browser->get($url);

See chapter 5 of Perl & LWP for a longer discussion of HTML forms and of form data, and chapters 6 through 9 for a longer discussion of extracting data from HTML. 📖

⭐ Absolutizing URLs ⭐

The URI class that we just mentioned above provides all sorts of methods for accessing and modifying parts of URLs (such as asking sort of URL it is with $url->scheme, and asking what host it refers to with $url->host, and so on, as described in the docs for the URI class). However, the methods of most immediate interest are the query_form method seen above, and now the new_abs method for taking a probably-relative URL string (like ../foo.html) and getting back an absolute URL (like http://www.perl.com/stuff/foo.html), as shown here:

use URI;
$abs = URI->new_abs($maybe_relative, $base);

For example, consider this program that matches URLs in the HTML list of new modules in CPAN:

use strict;
use warnings;
use LWP;
my $browser = LWP::UserAgent->new;
         my $url = 'http://www.cpan.org/RECENT.html';
         my $response = $browser->get($url);
         die "Can't get $url -- ", $response->status_line
          unless $response->is_success;

         my $html = $response->decoded_content;
         while( $html =~ m/<A HREF=\"(.*?)\"/g ) {
           print "$1\n";
         }

okay so when you actually run that, the output starts out like this:

         MIRRORING.FROM
         RECENT
         RECENT.html
         authors/00whois.html
         authors/01mailrc.txt.gz
         authors/id/A/AA/AASSAD/CHECKSUMS
         ...

but bestie. those are not absolute URLs 😭 like they're relative, they're giving "I live somewhere but I'm not telling you where". so if you actually want the absolute versions, you grab the URI module's new_abs method and swap your while loop for this:

         while( $html =~ m/<A HREF=\"(.*?)\"/g ) {
           print URI->new_abs( $1, $response->base ) ,"\n";
         }

(and that $response->base method — it comes from HTTP::Message — is literally just there to hand you back the URL you should use for resolving relative URLs. it's usually exactly the same as the URL you requested. no cap. 🫶)

and then the program goes off and emits these gorgeously absolute URLs:

         http://www.cpan.org/MIRRORING.FROM
         http://www.cpan.org/RECENT
         http://www.cpan.org/RECENT.html
         http://www.cpan.org/authors/00whois.html
         http://www.cpan.org/authors/01mailrc.txt.gz
         http://www.cpan.org/authors/id/A/AA/AASSAD/CHECKSUMS
         ...

see chapter 4 of Perl & LWP for a longer discussion of URI objects. it's a whole thing, it's giving lore.

and ok, real talk — using a regexp to match hrefs is a bit simplistic, ngl. for more robust programs you're probably gonna want an HTML-parsing module, like HTML::LinkExtor or HTML::TokeParser, or honestly even maybe HTML::TreeBuilder. slay 💅

✨ Other Browser Attributes ✨

LWP::UserAgent objects have so many attributes for controlling how they work. here are a few notable ones, bestie:

  • $browser->timeout(15); — this sets the browser object to give up on any request that doesn't answer within 15 seconds. ghosted, basically. ⏰
  • $browser->protocols_allowed( [ 'http', 'gopher'] ); — this makes the browser object refuse to speak any protocol other than HTTP and gopher. if it tries to access any other kind of URL (like an ftp: or mailto: or news: URL), it won't even attempt to connect — instead it immediately hands you back an error code 500, with a message like Access to 'ftp' URIs has been disabled. no ftp for you 🚫
  • use LWP::ConnCache; $browser->conn_cache(LWP::ConnCache->new()); — this tells the browser object to try using the HTTP/1.1 Keep-Alive feature, which speeds up requests by reusing the same socket connection for multiple requests to the same server. efficiency queen 👑
  • $browser->agent( 'SomeName/1.23 (more info here maybe)' ) — this changes how the browser object identifies itself in the default User-Agent line of its HTTP requests. by default it sends libwww-perl/versionnumber, like libwww-perl/5.65. you can change that to something more descriptive, like this:
  $browser->agent( 'SomeName/3.14 (contact@robotplexus.int)' );

or if need be, you can go full incognito in disguise, like this:

  $browser->agent( 'Mozilla/4.0 (compatible; MSIE 5.12; Mac_PowerPC)' );

the way that disguise ATE 😭✨

  • push @{ $ua->requests_redirectable }, 'POST'; — this tells the browser to obey redirection responses to POST requests (like most modern interactive browsers do), even though the HTTP RFC says that should not normally be done. lowkey chaotic, but do you 🙈

for more options and more info, see the full documentation for LWP::UserAgent.

🎀 Writing Polite Robots 🎀

if you wanna make sure your LWP-based program respects robots.txt files and doesn't spam requests too fast — because being rude to servers is not the vibe — you can use the LWP::RobotUA class instead of the LWP::UserAgent class.

LWP::RobotUA class is just like LWP::UserAgent, and you use it like so:

         use LWP::RobotUA;
         my $browser = LWP::RobotUA->new('YourSuperBot/1.34', 'you@yoursite.com');
           # Your bot's name and your email address

         my $response = $browser->get($url);

but HTTP::RobotUA adds these features, bestie:

  • if the robots.txt on $url's server forbids you from accessing $url, then the $browser object (assuming it's of class LWP::RobotUA) won't actually request it — instead it gives you back (in $response) a 403 error with a message Forbidden by robots.txt. so like, if you have this line:
  die "$url -- ", $response->status_line, "\nAborted"
   unless $response->is_success;

then the program would die with an error message like this:

  http://whatever.site.int/pith/x.html -- 403 Forbidden by robots.txt
  Aborted at whateverprogram.pl line 1234

💀 rip

  • if this $browser object sees that the last time it talked to $url's server was too recently, then it will pause (via sleep) to avoid making too many requests too often. how long it pauses for is, by default, one minute — but you can control it with the $browser->delay( minutes ) attribute.

for example, this code:

  $browser->delay( 7/60 );

...means that this browser will pause when it needs to avoid talking to any given server more than once every 7 seconds. 7/60 of a minute, math girlies rise 🧮✨

for more options and more info, see the full documentation for LWP::RobotUA.

🌈 Using Proxies 🌈

sometimes you're gonna want to (or you're gonna have to) use proxies for accessing certain sites and/or using certain protocols. this is most commonly the case when your LWP program is running (or could be running) on a machine that's behind a firewall. the firewall is the bouncer and the proxy is your plus-one 💅

to make a browser object use proxies that are defined in the usual environment variables (HTTP_PROXY, etc.), just call the env_proxy on a user-agent object before you go making any requests on it. specifically:

         use LWP::UserAgent;
         my $browser = LWP::UserAgent->new;

         # And before you go making any requests:
         $browser->env_proxy;

for more info on proxy parameters, see the LWP::UserAgent documentation — specifically the proxy, env_proxy, and no_proxy methods.

🔐 HTTP Authentication 🔐

lots of web sites restrict access to documents by using HTTP Authentication. and ngl this isn't just any old "enter your password" situation — it's a specific mechanism where the HTTP server sends the browser an HTTP code that basically says "That document is part of a protected realm, and you can access it only if you re-request it and add some special authorization headers to your request". it's giving VIP section 🎟️

for example, the Unicode.org admins stop email-harvesting bots from harvesting the contents of their mailing list archives by protecting them with HTTP Authentication, and then publicly stating the username and password (at http://www.unicode.org/mail-arch/) — namely username unicode-ml and password unicode. iconic of them honestly 🫶

for example, consider this URL, which is part of the protected area of the web site:

         http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html

if you access that with a browser, you'll get a prompt like Enter username and password for 'Unicode-MailList-Archives' at server 'www.unicode.org'.

in LWP, if you just request that URL, like this:

         use LWP;
         my $browser = LWP::UserAgent->new;

         my $url =
          'http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html';
         my $response = $browser->get($url);

         die "Error: ", $response->header('WWW-Authenticate') || 'Error accessing',
           #  ('WWW-Authenticate' is the realm-name)
           "\n ", $response->status_line, "\n at $url\n Aborting"
          unless $response->is_success;

then you'll get this error:

         Error: Basic realm="Unicode-MailList-Archives"
          401 Authorization Required
          at http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html
          Aborting at auth1.pl line 9.  [or wherever]

...because the $browser doesn't know the username and password for that realm (Unicode-MailList-Archives) at that host (www.unicode.org). it's locked out of the group chat 😭

the simplest way to let the browser know about this is to use the credentials method, to let it know about a username and password that it can try using for that realm at that host. the syntax is:

         $browser->credentials(
           'servername:portnumber',
           'realm-name',
          'username' => 'password'
         );

in most cases, the port number is 80, the default TCP/IP port for HTTP; and you usually call the credentials method before you make any requests. for example:

         $browser->credentials(
           'reports.mybazouki.com:80',
           'web_server_usage_reports',
           'plinky' => 'banjo123'
         );

so if we add the following to the program above, right after the $browser = LWP::UserAgent->new; line...

         $browser->credentials(  # add this to our $browser 's "key ring"
           'www.unicode.org:80',
           'Unicode-MailList-Archives',
           'unicode-ml' => 'unicode'
         );

...then when we run it, the request succeeds, instead of causing the die to be called. glow up ✨ the "key ring" comment is doing so much for me, it's literally a key ring 🔑💖

🦋 Accessing HTTPS URLs 🦋

when you access an HTTPS URL, it'll work for you just like an HTTP URL would — if your LWP installation has HTTPS support (via an appropriate Secure Sockets Layer library). for example:

         use LWP;
         my $url = 'https://www.paypal.com/';   # Yes, HTTPS!
         my $browser = LWP::UserAgent->new;
         my $response = $browser->get($url);
         die "Error at $url\n ", $response->status_line, "\n Aborting"
          unless $response->is_success;
         print "Whee, it worked!  I got that ",
          $response->content_type, " document!\n";

if your LWP installation doesn't have HTTPS support set up, then the response will be unsuccessful, and you'll get this error message:

         Error at https://www.paypal.com/
          501 Protocol scheme 'https' is not supported
          Aborting at paypal.pl line 7.   [or whatever program and line]

if your LWP installation does have HTTPS support installed, then the response should be successful, and you should be able to consult $response just like with any normal HTTP response. 👑

for info about installing HTTPS support for your LWP installation, see the helpful README.SSL file that comes in the libwww-perl distribution. read the README bestie, it's there for a reason 🎀

🍓 Getting Large Documents 🍓

when you're requesting a large (or at least potentially large) document, a problem with the normal way of using the request methods (like $response = $browser->get($url)) is that the response object in memory will have to hold the whole entire document — in memory. if the response is a thirty megabyte file, this is likely to be quite an imposition on this process's memory usage. like, your RAM is gonna be fighting for its life 😭

a notable alternative is to have LWP save the content to a file on disk, instead of saving it up in memory. this is the syntax to use:

         $response = $ua->get($url,
                                ':content_file' => $filespec,
                             );

for example,

         $response = $ua->get('http://search.cpan.org/',
                                ':content_file' => '/tmp/sco.html'
                             );

straight to disk, no middleman, we love a direct route 💾✨

okay so when you use this :content_file option, your $response is gonna have all the normal header lines, but $response->content is gonna be empty — like, ghosted, no content bestie 🫥. And if something goes wrong writing to the content file (say, permission denied, or the filesystem is just full 😭), that error does NOT show up through the is_success method. Nope! It gets reported through the Client-Aborted or X-Died response headers instead:

if ($response->header('Client-Aborted') eq 'die') {
  # handle error ...
}
⚠️ also heads up: this :content_file option isn't supported under older versions of LWP. So if you think your program might end up running on systems with older versions, you should lowkey consider adding use LWP 5.66; to check the LWP version first. periodt.

And if you do need to stay compatible with older LWP versions, use this syntax instead — it does the exact same thing:

use HTTP::Request::Common;
$response = $ua->request( GET($url), $filespec );

🌈 SEE ALSO 🌈

reminder bestie: this article is literally just the most rudimentary introduction to LWP 💅. To actually learn more about LWP and all the LWP-related tasks, you really must read from the following:

  • LWP::Simple — simple functions for getting/heading/mirroring URLs
  • LWP — overview of the libwww-perl modules
  • LWP::UserAgent — the class for objects that represent "virtual browsers"
  • HTTP::Response — the class for objects that represent the response to a LWP response, as in $response = $browser->get(...)
  • HTTP::Message and HTTP::Headers — classes that provide more methods to HTTP::Response.
  • URI — class for objects that represent absolute or relative URLs
  • URI::Escape — functions for URL-escaping and URL-unescaping strings (like turning this & that to and from this%20%26%20that).
  • HTML::Entities — functions for HTML-escaping and HTML-unescaping strings (like turning C. & E. Brontë to and from C. &amp; E. Bront&euml;)
  • HTML::TokeParser and HTML::TreeBuilder — classes for parsing HTML
  • HTML::LinkExtor — class for finding links in HTML documents
  • The book Perl & LWP by Sean M. Burke. O'Reilly & Associates, 2002. ISBN: 0-596-00178-9, <http://oreilly.com/catalog/perllwp/>. The whole book is also available free online: <http://lwp.interglacial.com>.
📚 the way that reading list ate, honestly. no cap, go read the book fr fr.

Copyright 2002, Sean M. Burke. You can redistribute this document and/or modify it, but only under the same terms as Perl itself.

💖 AUTHOR 💖

Sean M. Burke "sburke@cpan.org"

and that's a wrap on the whole tutorial bestie!! 🎀✨ this lil gem was served up by perl v5.34.0, all fresh and dated 2026-05-22, living at lwptut(3pm) 💅 go forth and fetch those URLs, you absolute icon, you 🦋💖

📦 where this page comes from