LWP::UserAgent(3)

💖 LWP::UserAgent(3) 💖

okay bestie, so this is from the User Contributed Perl Documentation section — it's the manual page for LWP::UserAgent(3), and honestly? it's kind of the main character of the whole libwww-perl situation. let's get into it 🫶

✨ NAME ✨

LWP::UserAgent — the web user agent class. that's it, that's the vibe. she's a class, she does web requests, she slays 💅

🦋 SYNOPSIS 🦋

the basic starter pack:

use strict;
use warnings;

use LWP::UserAgent ();

my $ua = LWP::UserAgent->new(timeout => 10);
$ua->env_proxy;

my $response = $ua->get('http://example.com');

if ($response->is_success) {
    print $response->decoded_content;
}
else {
    die $response->status_line;
}

and then the extra layers of security version (note the cookie_jar and protocols_allowed, bestie — those matter):

use strict;
use warnings;

use HTTP::CookieJar::LWP ();
use LWP::UserAgent       ();

my $jar = HTTP::CookieJar::LWP->new;
my $ua  = LWP::UserAgent->new(
    cookie_jar        => $jar,
    protocols_allowed => ['http', 'https'],
    timeout           => 10,
);

$ua->env_proxy;

my $response = $ua->get('http://example.com');

if ($response->is_success) {
    print $response->decoded_content;
}
else {
    die $response->status_line;
}

look at her go 😭 iconic behaviour, no notes.

🌸 DESCRIPTION 🌸

so LWP::UserAgent is a class implementing a web user agent — like, an actual object you can dispatch web requests with. that's the whole point of her.

in normal use the application creates an LWP::UserAgent object, then configures it with values for timeouts, proxies, name, etc. then it creates an instance of HTTP::Request for the request that needs to be performed. that request gets passed to one of the request methods on the UserAgent, which dispatches it using the relevant protocol, and hands you back an HTTP::Response object. simple as that ✨

there are convenience methods for sending the most common request types: get in LWP::UserAgent, head in LWP::UserAgent, post in LWP::UserAgent, put in LWP::UserAgent and delete in LWP::UserAgent. when you use these methods, the creation of the request object is hidden from you — exactly like the synopsis up there shows. so convenient, honestly 🎀

the basic approach of the library is to use HTTP-style communication for all protocol schemes. which means you'll be constructing HTTP::Request objects and receiving HTTP::Response objects even for non-HTTP resources like gopher and ftp 👑. and to make things even more HTTP-flavoured, gopher menus and file directories get converted to HTML documents. cute!

🧚 CONSTRUCTOR METHODS 🧚

the following constructor methods are available:

💜 clone

my $ua2 = $ua->clone;

returns a copy of the LWP::UserAgent object.

CAVEAT: please be aware that the clone method does not copy or clone your cookie_jar attribute. due to the limited restrictions on what can be used for your cookie jar, there is no way to clone the attribute. the cookie_jar attribute will be undef in the new object instance. so like... the cookies do not come with you on this journey 💀

🍓 new

my $ua = LWP::UserAgent->new( %options )

this method constructs a new LWP::UserAgent object and returns it. key/value pair arguments may be provided to set up the initial state. the following options correspond to the attribute methods described below:

KEY                            DEFAULT
---------------------------    --------------------
agent                          "libwww-perl/#.###"
allow_credentialed_redirects   undef
allow_downgrade                undef
conn_cache                     undef
cookie_jar                     undef
cookie_jar_class               HTTP::Cookies
default_headers                HTTP::Headers->new
from                           undef
local_address                  undef
max_redirect                   7
max_size                       undef
no_proxy                       []
parse_head                     1
protocols_allowed              undef
protocols_forbidden            undef
proxy                          {}
requests_redirectable          ['GET', 'HEAD']
send_te                        1
show_progress                  undef
ssl_opts                       { verify_hostname => 1 }
timeout                        180

when following a 3xx redirect to a different origin (a different scheme, host, or port), LWP::UserAgent strips Authorization and Proxy-Authorization from the cloned request to avoid leaking caller-supplied credentials to the redirect target. set allow_credentialed_redirects to a true value to opt out and forward these headers across origins. security queen behaviour 🔥

a 3xx redirect that downgrades an https request to plain http is refused by default — the original response is returned with a Client-Warning header explaining the refusal. set allow_downgrade to a true value to opt in to following such redirects. yeah she's not letting you walk into that alleyway, periodt ✋

the following additional options are also accepted: if the env_proxy option is passed in with a true value, then proxy settings are read from environment variables (see env_proxy in LWP::UserAgent). if env_proxy isn't provided, the PERL_LWP_ENV_PROXY environment variable controls if env_proxy in LWP::UserAgent is called during initialization. if the keep_alive option value is defined and non-zero, then an LWP::ConnCache is set up (see conn_cache in LWP::UserAgent). the keep_alive value is passed on as the total_capacity for the connection cache. got all that? 🫶

proxy must be set as an arrayref of key/value pairs. no_proxy takes an arrayref of domains.

🌈 ATTRIBUTES 🌈

the settings of the configuration attributes modify the behaviour of the LWP::UserAgent when it dispatches requests. most of these can also be initialized by options passed to the constructor method.

the following attribute methods are provided. the attribute value is left unchanged if no argument is given. the return value from each method is the old attribute value. so it's a little time capsule of what she used to be 💫

⭐ agent

my $agent = $ua->agent;
$ua->agent('Checkbot/0.4 ');    # append the default to the end
$ua->agent('Mozilla/5.0');
$ua->agent("");                 # don't identify

get/set the product token that's used to identify the user agent on the network. the agent value is sent as the User-Agent header in the requests.

the default is a string of the form libwww-perl/#.###, where #.### is substituted with the version number of this library.

if the provided string ends with space, the default libwww-perl/#.### string is appended to it. sneaky little trick, love that for us.

the user agent string should be one or more simple product identifiers with an optional version number separated by the / character.

💅 allow_credentialed_redirects

my $allow = $ua->allow_credentialed_redirects;
$ua->allow_credentialed_redirects( 1 );

get/set whether caller-supplied Authorization and Proxy-Authorization headers are forwarded across cross-origin 3xx redirects (a different scheme, host, or port). defaults to a false value, meaning the headers are stripped on cross-origin redirects to avoid leaking credentials to the redirect target. same-origin redirects always retain these headers. so your secrets stay home unless you explicitly say otherwise 🎀

🦋 allow_downgrade

my $allow = $ua->allow_downgrade;
$ua->allow_downgrade( 1 );

get/set whether a 3xx redirect from an https request to a plain http URL is followed. defaults to a false value, meaning such redirects are refused; the original response is returned with a Client-Warning header. set to a true value to opt in to following the redirect. note that even when allow_downgrade is true, cross-origin credential stripping still applies (see allow_credentialed_redirects). she's gonna protect you even when you're being reckless, honestly 😭

💖 conn_cache

my $cache_obj = $ua->conn_cache;
$ua->conn_cache( $cache_obj );

get/set the LWP::ConnCache object to use. see LWP::ConnCache for details. go read it, iykyk.

my $jar = $ua->cookie_jar;
$ua->cookie_jar( $cookie_jar_obj );

get/set the cookie jar object to use. the only requirement is that the cookie jar object must implement the extract_cookies($response) and add_cookie_header($request) methods. those methods will then be invoked by the user agent as requests are sent and responses are received. normally this will be an HTTP::Cookies object or some subclass. you are, however, encouraged to use HTTP::CookieJar::LWP instead. see "BEST PRACTICES" for more information.

use HTTP::CookieJar::LWP ();

my $jar = HTTP::CookieJar::LWP->new;
my $ua = LWP::UserAgent->new( cookie_jar => $jar );

# or after object creation
$ua->cookie_jar( $cookie_jar );

the default is to have no cookie jar, i.e. never automatically add Cookie headers to the requests. crumbs-free lifestyle ✨

if $jar contains an unblessed hash reference, a new cookie jar object is created for you automatically. the object is of the class set with the cookie_jar_class constructor argument, which defaults to HTTP::Cookies.

$ua->cookie_jar({ file => "$ENV{HOME}/.cookies.txt" });

is really just a shortcut for:

require HTTP::Cookies;
$ua->cookie_jar(HTTP::Cookies->new(file => "$ENV{HOME}/.cookies.txt"));

as described above and in "BEST PRACTICES", you should set cookie_jar_class to HTTP::CookieJar::LWP to get a safer cookie jar.

my $ua = LWP::UserAgent->new( cookie_jar_class => 'HTTP::CookieJar::LWP' );
$ua->cookie_jar({}); # HTTP::CookieJar::LWP takes no args

these can also be combined into the constructor, so a jar is created at instantiation. chef's kiss, one-step setup 🔥

my $ua = LWP::UserAgent->new(
  cookie_jar_class => 'HTTP::CookieJar::LWP',
  cookie_jar       =>  {},
);

🔐 credentials

my $creds = $ua->credentials();
$ua->credentials( $netloc, $realm );
$ua->credentials( $netloc, $realm, $uname, $pass );
$ua->credentials("www.example.com:80", "Some Realm", "foo", "secret");

get/set the user name and password to be used for a realm. the $netloc is a string of the form <host>:<port>. the username and password will only be passed to this server — nobody else gets a crumb of that info, periodt 💅

🌸 default_header

$ua->default_header( $field );
$ua->default_header( $field => $value );
$ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());
$ua->default_header('Accept-Language' => "no, en");

this is just a shortcut for $ua->default_headers->header( $field => $value ). lazy girl math but make it efficient 🎀

🌟 default_headers

my $headers = $ua->default_headers;
$ua->default_headers( $headers_obj );

get/set the headers object that will provide default header values for any requests sent. by default this will be an empty HTTP::Headers object. a blank canvas, we love to see it ✨

💌 from

my $from = $ua->from;
$ua->from('foo@bar.com');

get/set the email address for the human user who controls the requesting user agent. the address should be machine-usable, as defined in RFC2822 <https://tools.ietf.org/html/rfc2822>. the from value is sent as the From header in the requests. put your email in, be a real one 🫶

So yeah — the default is to not send a "From" header. If you want to default literally any other header, peep "default_headers" in LWP::UserAgent — that's the more general interface that lets any header be defaulted. 🎀

🫶 `local_address`

my $address = $ua->local_address;
$ua->local_address( $address );

Gets/sets the local interface we bind to for network connections — think of it as picking which door your request walks out of. 🚪 You can specify the interface as a hostname or an IP address. This value gets passed as the LocalAddr argument to IO::Socket::INET. slay.

🔁 `max_redirect`

my $max = $ua->max_redirect;
$ua->max_redirect( $n );

Reads or sets the object's limit of how many times it will obey redirection responses in a given request cycle. Iconic little safety rail.

The default value is 7. Which means: if you call request in LWP::UserAgent and the response is a redirect elsewhere which is in turn a redirect, and so on seven times, then LWP gives up after that seventh request. Seven. Not eight. She counted. 💅

📏 `max_size`

my $size = $ua->max_size;
$ua->max_size( $bytes );

Get/set the size limit for response content. The default is undef, which literally means there is no limit — go off, download the whole internet, I guess. 🙈

If the returned response content is only partial, because the size limit was exceeded, then a Client-Aborted header will be added to the response. Heads up tho: the content might end up longer than max_size, as we abort once appending a chunk of data makes the length exceed the limit. And the Content-Length header, if present, will indicate the length of the full content and will normally not be the same as length($res->content). so don't @ me when the numbers don't match, that's just how she is 😭

🧠 `parse_head`

my $bool = $ua->parse_head;
$ua->parse_head( $boolean );

Get/set a value indicating whether we should initialize response headers from the <head> section of HTML documents. The default is true. ✨ Do not turn this off unless you know what you are doing. like fr fr, this is a certified you-will-regret-it toggle 💀

🚦 `protocols_allowed`

my $aref = $ua->protocols_allowed;      # get allowed protocols
$ua->protocols_allowed( \@protocols );  # allow ONLY these
$ua->protocols_allowed(undef);          # delete the list
$ua->protocols_allowed(['http',]);      # ONLY allow http

By default, an object has neither a protocols_allowed list, nor a protocols_forbidden in LWP::UserAgent list. Free spirit, no rules. 🦋

This reads (or sets) this user agent's list of protocols that the request methods will exclusively allow. The protocol names are case insensitive.

For example: $ua->protocols_allowed( [ 'http', 'https'] ); means that this user agent will allow only those protocols, and attempts to use this user agent to access URLs with any other schemes (like ftp://...) will result in a 500 error. not the vibe, but okay 💀

Note that having a protocols_allowed list causes any protocols_forbidden in LWP::UserAgent list to be ignored. the allow-list is the main character here, sorry not sorry 👑

🙅 `protocols_forbidden`

my $aref = $ua->protocols_forbidden;    # get the forbidden list
$ua->protocols_forbidden(\@protocols);  # do not allow these
$ua->protocols_forbidden(['http',]);    # All http reqs get a 500
$ua->protocols_forbidden(undef);        # delete the list

This reads (or sets) this user agent's list of protocols that the request method will not allow. The protocol names are case insensitive.

For example: $ua->protocols_forbidden( [ 'file', 'mailto'] ); means that this user agent will not allow those protocols, and attempts to use this user agent to access URLs with those schemes will result in a 500 error. blocked. banned. escorted out by security 🚫

🔀 `requests_redirectable`

my $aref = $ua->requests_redirectable;
$ua->requests_redirectable( \@requests );
$ua->requests_redirectable(['GET', 'HEAD',]); # the default

This reads or sets the object's list of request names that redirect_ok in LWP::UserAgent will allow redirection for. By default, this is ['GET', 'HEAD'], as per RFC 2616 (https://tools.ietf.org/html/rfc2616). Tea. ☕

Wanna sneak POST in there too? Consider:

push @{ $ua->requests_redirectable }, 'POST';

📨 `send_te`

my $bool = $ua->send_te;
$ua->send_te( $boolean );

If true, will send a TE header along with the request. The default is true. Set it to false to disable the TE header for systems who can't handle it. be considerate bestie 🫶

📊 `show_progress`

my $bool = $ua->show_progress;
$ua->show_progress( $boolean );

Get/set a value indicating whether a progress bar should be displayed on the terminal as requests are processed. The default is false. Turn it on if you want that little loading bar serotonin 🌈

🔒 `ssl_opts`

my @keys = $ua->ssl_opts;
my $val = $ua->ssl_opts( $key );
$ua->ssl_opts( $key => $value );

Get/set the options for SSL connections. Without argument, return the list of options keys currently set. With a single argument, return the current value for the given option. With 2 arguments, set the option value and return the old one. Setting an option to the value undef removes this option.

The options that LWP relates to are:

`"verify_hostname" => $bool`

When TRUE, LWP will for secure protocol schemes ensure it connects to servers that have a valid certificate matching the expected hostname. If FALSE, no checks are made and you can't be sure that you communicate with the expected peer. 😬 The no checks behaviour was the default for libwww-perl-5.837 and earlier releases.

This option is initialized from the PERL_LWP_SSL_VERIFY_HOSTNAME environment variable. If this environment variable isn't set; then verify_hostname defaults to 1.

Please note that that recently the overall effect of this option with regards to SSL handling has changed. As of version 6.11 of LWP::Protocol::https, which is an external module, SSL certificate verification was harmonized to behave in sync with IO::Socket::SSL. With this change, setting this option no longer disables all SSL certificate verification, only the hostname checks. To disable all verification, use the SSL_verify_mode option in the ssl_opts attribute. For example: $ua-"ssl_opts(SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE);

`"SSL_ca_file" => $path`

The path to a file containing Certificate Authority certificates. A default setting for this option is provided by checking the environment variables PERL_LWP_SSL_CA_FILE and HTTPS_CA_FILE in order.

`"SSL_ca_path" => $path`

The path to a directory containing files containing Certificate Authority certificates. A default setting for this option is provided by checking the environment variables PERL_LWP_SSL_CA_PATH and HTTPS_CA_DIR in order.

Other options can be set and are processed directly by the SSL Socket implementation in use. See IO::Socket::SSL or Net::SSL for details.

The libwww-perl core no longer bundles protocol plugins for SSL. You will need to install LWP::Protocol::https separately to enable support for processing https-URLs. 💅 like, go install it, she's not coming with you by default

⏱️ `timeout`

my $secs = $ua->timeout;
$ua->timeout( $secs );

Get/set the timeout value in seconds. The default value is 180 seconds, i.e. 3 minutes. three whole minutes of patience, she's generous ✨

The request is aborted if no activity on the connection to the server is observed for timeout seconds. Which means the time it takes for the complete transaction and the request in LWP::UserAgent method to actually return might be longer. yeah. math. 😭

When a request times out, a response object is still returned. The response will have a standard HTTP Status Code (500). This response will have the Client-Warning header set to the value of Internal response. See the get in LWP::UserAgent method description below for further details. she still shows up to the function even when she's late 💀

🕵️‍♀️ PROXY ATTRIBUTES ✨

The following methods set up when requests should be passed via a proxy server. proxy era. 🌐

🌍 `env_proxy`

$ua->env_proxy;

Load proxy settings from *_proxy environment variables. You might specify proxies like this (sh-syntax):

gopher_proxy=http://proxy.my.place/
wais_proxy=http://proxy.my.place/
no_proxy="localhost,example.com"
export gopher_proxy wais_proxy no_proxy

csh or tcsh users should use the setenv command to define these environment variables. different shell, different ritual 🐚

On systems with case insensitive environment variables there exists a name clash between the CGI environment variables and the HTTP_PROXY environment variable normally picked up by env_proxy. Because of this, HTTP_PROXY is not honored for CGI scripts. The CGI_HTTP_PROXY environment variable can be used instead. iykyk 🙈

🚧 `no_proxy`

$ua->no_proxy( @domains );
$ua->no_proxy('localhost', 'example.com');
$ua->no_proxy(); # clear the list

Do not proxy requests to the given domains, including subdomains. Calling no_proxy without any domains clears the list of domains. clean slate, no notes 🧼

🔗 `proxy`

$ua->proxy(\@schemes, $proxy_url)
$ua->proxy(['http', 'ftp'], 'http://proxy.sn.no:8001/');

# For a single scheme:
$ua->proxy($scheme, $proxy_url)
$ua->proxy('gopher', 'http://proxy.sn.no:8001/');

# To set multiple proxies at once:
$ua->proxy([
    ftp => 'http://ftp.example.com:8001/',
    [ 'http', 'https' ] => 'http://http.example.com:8001/',
]);

Set/retrieve proxy URL for a scheme. 👑

The first form specifies that the URL is to be used as a proxy for access methods listed in the list in the first method argument, i.e. http and ftp.

The second form shows a shorthand form for specifying proxy URL for a single access scheme.

The third form demonstrates setting multiple proxies at once. This is also the only form accepted by the constructor. remember that or she will simply not work for you 💀

✨ HANDLERS ✨

okay so handlers are little bits of code you inject at different phases while requests are being processed. they’re giving behind-the-scenes main character energy 💅. here are the methods that manage the active handlers:

💖 add_handler 💖

$ua->add_handler( $phase => \&cb, %matchspec )

Add a handler to be invoked in the given processing phase. For how to specify %matchspec, see "Matching" in HTTP::Config.

The possible values for $phase and the corresponding callback signatures are below. note: these handlers are documented in the order they will be run, which is:

request_preprepare
request_prepare
request_send
response_header
response_data
response_done
response_redirect

request_preprepare => sub { my($request, $ua, $handler) = @_; ... } — this handler is called before the request_prepare and other standard initialization of the request. it can be used to set up headers and attributes that the request_prepare handler depends on. proxy initialization should take place here; but in general, don’t register handlers for this phase.
request_prepare => sub { my($request, $ua, $handler) = @_; ... } — this handler is called before the request is sent and can modify the request any way it sees fit. for instance, it can add certain headers to specific requests. the method can assign a new request object to $_[0] to fully replace the request that is sent. the return value from the callback is ignored. if an exception is raised, it will abort the request and make the request method return a "400 Bad request" response.
request_send => sub { my($request, $ua, $handler) = @_; ... } — this handler gets a chance to handle requests before they’re sent to the protocol handlers. it should return an HTTP::Response object if it wishes to terminate the processing; otherwise it should return nothing. the response_header and response_data handlers will not be invoked for this response, but response_done will be.
response_header => sub { my($response, $ua, $handler) = @_; ... } — this handler is called right after the response headers have been received, but before any content data. it might set up handlers for data and might croak to abort the request. it might also set the $response->{default_add_content} value to control if any received data should be added to the response object directly. this will initially be false if the $ua->request() method was called with a $content_file or $content_cb argument; otherwise true.
response_data => sub { my($response, $ua, $handler, $data) = @_; ... } — this handler is called for each chunk of data received for the response. it might croak to abort the request. it needs to return a TRUE value to be called again for subsequent chunks for the same request.
response_done => sub { my($response, $ua, $handler) = @_; ... } — this handler is called after the response has been fully received, but before any redirect handling is attempted. it can be used to extract information or modify the response.
response_redirect => sub { my($response, $ua, $handler) = @_; ... } — this handler is called in $ua->request after response_done. if the handler returns an HTTP::Request object, we’ll start over with processing this request instead.

for all of these, $handler is a code reference to the handler that is currently being run.

🎀 get_my_handler 🎀

$ua->get_my_handler( $phase, %matchspec );
$ua->get_my_handler( $phase, %matchspec, $init );

Will retrieve the matching handler as a hash ref.

If $init is passed as a true value, create and add the handler if it’s not found. if $init is a subroutine reference, then it’s called with the created handler hash as argument. this sub might populate the hash with extra fields; especially the callback. if $init is a hash reference, merge the hashes.

🦋 handlers 🦋

$ua->handlers( $phase, $request )
$ua->handlers( $phase, $response )

Returns the handlers that apply to the given request or response at the given processing phase.

🌸 remove_handler 🌸

$ua->remove_handler( undef, %matchspec );
$ua->remove_handler( $phase, %matchspec );
$ua->remove_handler(); # REMOVE ALL HANDLERS IN ALL PHASES

Remove handlers that match the given %matchspec. if $phase is not provided, remove handlers from all phases.

⚠️ be careful, bestie: calling this function with a %matchspec that is not specific enough can remove handlers not owned by you. it’s probably better to use the "set_my_handler" in LWP::UserAgent method instead.

The removed handlers are returned.

💜 set_my_handler 💜

$ua->set_my_handler( $phase, $cb, %matchspec );
$ua->set_my_handler($phase, undef); # remove handler for phase

Set handlers private to the executing subroutine. works by defaulting an "owner" field to the %matchspec that holds the name of the called subroutine. you might pass an explicit "owner" to override this.

If $cb is passed as undef, remove the handler.

✨ REQUEST METHODS ✨

Okay bestie, this is the fun part — the actual doing stuff section 💅 These are all the methods that send requests out into the world through your user agent. Ready? Let's go 🦋

The following request methods are provided:

🌸 delete

my $res = $ua->delete( $url );
my $res = $ua->delete( $url, $field_name => $value, ... );

This one fires off a DELETE request at the $url you give it. Extra headers and content options work exactly like they do in the get in LWP::UserAgent method — same energy, different verb ✨

Under the hood it uses the DELETE() function from HTTP::Request::Common to build the request, so if you wanna know how to pass form content and other fancy stuff, go peek at HTTP::Request::Common 🫶

🎀 get

my $res = $ua->get( $url );
my $res = $ua->get( $url , $field_name => $value, ... );

The classic, the icon, the main character 👑 This dispatches a GET request on the given $url. You can pass extra arguments to set up the request's headers — just give them as separate name/value pairs. What comes back is a response object, and HTTP::Response has the whole rundown on what interface it gives you 💖

Now here's the part you need to know: even when LWP can't connect to the server in the URL, or when other protocol-handler failures happen, you still get a response object back. These internal responses use the standard HTTP status codes, which means — plot twist 😭 — you literally can't tell them apart from real server responses just by testing the status code. Internal error responses that LWP generates will have the Client-Warning header set to the value Internal response. So if you need to distinguish those internal ones from what a remote server actually sent you, you gotta test that header value, periodt 🫡

Field names that start with : are special, bestie. They don't set up request headers — instead they decide how the response content gets treated. Here are the special field names you can use:

':content_file'   => $filename # or $filehandle
':content_cb'     => \&callback
':read_size_hint' => $bytes

If you hand over a $filename or $filehandle with the :content_file option, the response content gets saved there instead of into the response object. Fun fact: the $filehandle can also be an object with an open file descriptor, like a File::Temp object 🧚 If you pass a callback with the :content_cb option instead, that function gets called for each chunk of response content as it arrives from the server. And if you give neither? Then the response content just piles up in the response object itself — which honestly might not be it for very large response bodies 💀

You can only specify one of :content_file or :content_cb — pick a lane. Also important: the content of unsuccessful responses will always accumulate in the response object itself, no matter what :content_file or :content_cb options you passed in. And note that errors writing to the content file (say, permission denied or the filesystem being full 😬) get reported via the Client-Aborted or X-Died response headers — not the is_success method. Don't get caught out by that one.

The :read_size_hint option gets passed along to the protocol module, which will try to read data from the server in chunks of that size. Smaller :read_size_hint value = a higher number of callback invocations, just so you know ⭐

The callback function gets called with 3 arguments: a chunk of data, a reference to the response object, and a reference to the protocol object. And yes, the callback can abort the request by invoking die(). The exception message will show up as the X-Died header field in the response returned by the $ua->get() method 🙈

🦋 head

my $res = $ua->head( $url );
my $res = $ua->head( $url , $field_name => $value, ... );

This dispatches a HEAD request on the given $url. Other than that, it works just like the get in LWP::UserAgent method we just talked about 💜

💅 is_protocol_supported

my $bool = $ua->is_protocol_supported( $scheme );

Use this to test whether your user agent object supports a given $scheme. (And the $scheme can be a plain string like http or ftp, or it can be a URI object reference — flexible queen 🌈)

So how's it decided? It comes down to the user agent's protocols_allowed or protocols_forbidden lists (if any exist), plus whatever LWP itself is capable of. Basically: it returns true only if LWP supports that protocol and it's permitted for this particular object. Both boxes gotta be checked ✔️

🌟 is_online

my $bool = $ua->is_online;

Tries to figure out if you've got internet access. Returns 1 (true) if the built-in heuristics decide the user agent can reach the Internet (over HTTP), or 0 (false) if not 📶

Also see LWP::Online — it's giving related reading 💖

🍓 mirror

my $res = $ua->mirror( $url, $filename );

This one grabs the document identified by $url and stores it in a file called $filename. If that file already exists, the request will include an If-Modified-Since header matching the modification time of the file 🕐 If the document on the server hasn't changed since then, nothing happens — we move on with our lives. But if it has been updated, it gets downloaded again, and the modification time of the file will be forced to match the server's.

It uses move in File::Copy to attempt to atomically replace the $filename — cute and tidy ✨

Return value is an HTTP::Response object, obviously 💅

🔥 patch

# Any version of HTTP::Message works with this form:
my $res = $ua->patch( $url, $field_name => $value, Content => $content );

# Using hash or array references requires HTTP::Message >= 6.12
use HTTP::Request 6.12;
my $res = $ua->patch( $url, \%form );
my $res = $ua->patch( $url, \@form );
my $res = $ua->patch( $url, \%form, $field_name => $value, ... );
my $res = $ua->patch( $url, $field_name => $value, Content => \%form );
my $res = $ua->patch( $url, $field_name => $value, Content => \@form );

This dispatches a PATCH request on the given $url, with %form or @form giving you the key/value pairs for the fill-in form content 🎀 Additional headers and content options are the same as for the get in LWP::UserAgent method.

CAVEAT: This method can only accept content that's in key-value pairs when using HTTP::Request::Common prior to version 6.12. Any use of hash or array references will result in an error prior to version 6.12. So check your version, bestie 😭

It uses the PATCH function from HTTP::Request::Common to build the request. See HTTP::Request::Common for all the details on passing form content and other advanced features 🫶

🌈 post

my $res = $ua->post( $url, \%form );
my $res = $ua->post( $url, \@form );
my $res = $ua->post( $url, \%form, $field_name => $value, ... );
my $res = $ua->post( $url, $field_name => $value, Content => \%form );
my $res = $ua->post( $url, $field_name => $value, Content => \@form );
my $res = $ua->post( $url, $field_name => $value, Content => $content );

Dispatches a POST request on the given $url, with %form or @form providing the key/value pairs for the fill-in form content 💖 Additional headers and content options are the same as for the get in LWP::UserAgent method.

It uses the POST function from HTTP::Request::Common to build the request — see HTTP::Request::Common for details on passing form content and other advanced features ✨

🧚 put

# Any version of HTTP::Message works with this form:
my $res = $ua->put( $url, $field_name => $value, Content => $content );

# Using hash or array references requires HTTP::Message >= 6.07
use HTTP::Request 6.07;
my $res = $ua->put( $url, \%form );
my $res = $ua->put( $url, \@form );
my $res = $ua->put( $url, \%form, $field_name => $value, ... );
my $res = $ua->put( $url, $field_name => $value, Content => \%form );
my $res = $ua->put( $url, $field_name => $value, Content => \@form );

Dispatches a PUT request on the given $url, with %form or @form giving the key/value pairs for the fill-in form content 💜 Additional headers and content options are the same as for the get in LWP::UserAgent method.

CAVEAT: This method can only accept content that's in key-value pairs when using HTTP::Request::Common prior to version 6.07. Any use of hash or array references will result in an error prior to version 6.07. Yeah, 6.07 this time, not 6.12 — don't mix 'em up 🙈

It uses the PUT function from HTTP::Request::Common to build the request. See HTTP::Request::Common for details on passing form content and other advanced features 🦋

👑 request

my $res = $ua->request( $request );
my $res = $ua->request( $request, $content_file );
my $res = $ua->request( $request, $content_cb );
my $res = $ua->request( $request, $content_cb, $read_size_hint );

This dispatches the $request object you hand it. Normally that's an instance of the HTTP::Request class, but honestly any object with a similar interface will do — we're not picky 💅 The return value is an HTTP::Response object.

The request method processes redirects and authentication responses transparently. Which means it may actually send several simple requests via the simple_request in LWP::UserAgent method described below. One call, many hops, no cap 🐇

All those request methods we just went through — get in LWP::UserAgent, head in LWP::UserAgent, post in LWP::UserAgent and mirror in LWP::UserAgent — dispatch the request they build via this method right here. They're convenience methods that just hide the request-object creation from you. Iconic labor-saving behaviour ✨

The $content_file, $content_cb and $read_size_hint all correspond to the options described with the get in LWP::UserAgent method above. And repeat after me: errors writing to the content file (permission denied, filesystem full, the usual 😭) will be reported via the Client-Aborted or X-Died response headers, and not the is_success method.

Also — and this is lowkey cool — you're allowed to use a CODE reference as content in the request object you pass in. The content function should return the content when called, and the content can be returned in chunks. The content function will be invoked repeatedly until it returns an empty string to signal that there's no more content 🌸

⭐ simple_request

my $request = HTTP::Request->new( ... );
my $res = $ua->simple_request( $request );
my $res = $ua->simple_request( $request, $content_file );
my $res = $ua->simple_request( $request, $content_cb );
my $res = $ua->simple_request( $request, $content_cb, $read_size_hint );

This dispatches a single request and returns the response it received. Arguments are the same as for the request in LWP::UserAgent described above 🫶

The difference from request in LWP::UserAgent? simple_request will not try to handle redirects or authentication responses. It's one and done, no detours 🚫 And funnily enough, the request in LWP::UserAgent method will, in fact, invoke this method for each simple request it sends. It's giving teamwork 💖

💅 CALLBACK METHODS

Okay so these are the methods that get called as your requests are being processed. They're documented right here because if you're subclassing LWP::UserAgent, you might lowkey want to override how they behave. 👑

🎀 get_basic_credentials

# This checks wantarray and can either return an array:
my ($user, $pass) = $ua->get_basic_credentials( $realm, $uri, $isproxy );
# or a string that looks like "user:pass"
my $creds = $ua->get_basic_credentials($realm, $uri, $isproxy);

So request in LWP::UserAgent calls this bad boy to go grab credentials for documents that are protected by Basic or Digest Authentication. The arguments it passes in are the $realm that the server handed over, the $uri you asked for, and a "boolean flag" telling you whether this auth is happening against a proxy server. 🕵️‍♀️

Your method should return a username and password. If you wanna abort the whole authentication resolution attempt, just return an empty list — that's the move. Subclasses can override this method to prompt the user for the info, and there's a whole example of that in the lwp-request program that ships with this library. ✨

The base implementation is pretty chill — it just checks a set of pre-stored member variables that were set up with the credentials method in LWP::UserAgent.

🦋 prepare_request

$request = $ua->prepare_request( $request );

simple_request in LWP::UserAgent invokes this one. Its whole job is to modify the $request object you handed it by setting up various headers based on the attributes of the user agent. The return value should normally be that same $request object you passed in — but if you return a different request object, that's the one that actually gets processed. No cap.

The headers the base implementation touches are: User-Agent, From, Range and Cookie. 🍪

🌸 progress

my $prog = $ua->progress( $status, $request_or_response );

This gets called frequently as the response comes in, no matter how the content is being processed. It's called with $status set to "begin" at the start of processing the request, and with $state set to "end" before the request method returns. In between those two, $status will be either the fraction of the response you've received so far, or the string "tick" if that fraction just can't be calculated. 🕰️

When $status is "begin", the second argument is the HTTP::Request object — otherwise it's the HTTP::Response object.

⭐ redirect_ok

my $bool = $ua->redirect_ok( $prospective_request, $response );

request in LWP::UserAgent calls this before it tries to follow a redirection to the request in $response. Return a true value if this redirection is permissible, mmkay? The $prospective_request is the request that'll get sent if this method returns true. 🚦

The base implementation is kinda picky: it returns false unless the method is in the object's requests_redirectable list, it returns false if the proposed redirection is to a file://... URL, and it returns true otherwise.

🌈 BEST PRACTICES

The default settings will get you up and running super fast, but there are some settings you can change to make your life genuinely easier. 💖

🍓 Handling Cookies

You are encouraged to install Mozilla::PublicSuffix and use HTTP::CookieJar::LWP as your cookie jar. HTTP::CookieJar::LWP gives you a better security model that matches what current web browsers do — but only when Mozilla::PublicSuffix is installed. 🔒

use HTTP::CookieJar::LWP ();

my $jar = HTTP::CookieJar::LWP->new;
my $ua = LWP::UserAgent->new( cookie_jar => $jar );

See cookie_jar for more info, bestie.

🧚 Managing Protocols

protocols_allowed gives you the ability to allow arbitrary protocols. Slay.

my $ua = LWP::UserAgent->new(
    protocols_allowed => [ 'http', 'https' ]
);

This will stop you from inadvertently following URLs like file:///etc/passwd. 💀 See protocols_allowed.

protocols_forbidden gives you the ability to deny arbitrary protocols. Also iconic.

my $ua = LWP::UserAgent->new(
    protocols_forbidden => [ 'file', 'mailto', 'ssh', ]
);

This can also prevent you from inadvertently following URLs like file:///etc/passwd. See protocols_forbidden.

👀 SEE ALSO

See LWP for the complete overview of libwww-perl5. See lwpcook and the scripts lwp-request and lwp-download for examples of usage. 📚

See HTTP::Request and HTTP::Response for a description of the message objects that get dispatched and received. See HTTP::Request::Common and HTML::Form for other ways to build request objects. 🛠️

See WWW::Mechanize and WWW::Search for examples of more specialized user agents based on LWP::UserAgent. 🤖

💜 COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. 🫶

And that's a wrap on the whole page, bestie! 🎉 This version is perl v5.42.0, dated 2026-05-18 — LWP::UserAgent(3), and you officially read the whole thing. Go forth and make requests responsibly 💅✨

📦 where this page comes from

opensuse-leap 16.0: perl-libwww-perl 6.770.0-160000.3.1 · Artistic-1.0 OR GPL-1.0-or-later
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms