LW2(3perl)

📦
version 2.5 · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

💖 LW2 💖 (the 3perl manual page)

okay bestie, gather round — this is LW2, a user-contributed Perl manual page. Perl HTTP library, version 2.5. She's iconic, she's a single file, and she does a LOT. Let's get into it. 🦋

🌸 NAME 🌸

LW2 — Perl HTTP library version 2.5. that's it, that's the name, short and slay. 👑

✨ SYNOPSIS ✨

use LW2;

require 'LW2.pm';

like, two whole ways to pull her in. pick your fave. 💅

🎀 DESCRIPTION 🎀

okay so. Libwhisker is a Perl library that's genuinely useful for HTTP testing scripts. 🕵️♀️

The way that it works is: it's a pure-Perl reimplementation of functionality you'd normally find scattered across a whole squad of other modules — specifically the stuff in LWP, URI, Digest::MD5, Digest::MD4, Data::Dumper, Authen::NTLM, HTML::Parser, HTML::FormParser, CGI::Upload, MIME::Base64, and GetOpt::Std. so basically she ate all of them and showed up as one file. 💀

And the design goals? libwhisker is:

  • portable — it's a single perl file. one. file. no dependency drama. 🌈
  • fast — general benchmarks show libwhisker is faster than LWP. no cap. 🔥
  • flexible — great care was taken to make sure the library does exactly what you want to do, even if it means breaking the protocol. yes really, she will let you color outside the lines. 🙈

👑 FUNCTIONS 👑

The following are the functions contained in Libwhisker. buckle up, there's a bunch. 🫶

🌸 auth_brute_force 🌸

Params: $auth_method, \%req, $user, \@passwords [, $domain, $fail_code ]

Return: $first_valid_password, undef if error/none found

so this one does a HTTP authentication brute force against a server — the host and URI come from %req, obviously. It'll try every single password in the password array for the given user, one by one. The first password (when paired with the given user) that does not come back with HTTP 401 gets returned, and the brute force stops right there, immediately. 🛑

bestie, please don't just trust that. Retry the request with the password it handed you and double-check you actually got a useful return code that means "auth worked" (like 200 or 302), and not something more unhinged (407, 500, etc.). 🙈

$domain is optional and it's only used for NTLM auth.

Note: set up any proxy settings and proxy auth in %req before you call this function. like, beforehand. ✨

You can also brute-force proxy authentication — set up the target proxy as proxy_host and proxy_port in %req, use an arbitrary host and uri (preferably one that's reachable once proxy authorization succeeds), and set $fail_code to 407. The $auth_method you pass in should be a proxy-based one ('proxy-basic', 'proxy-ntlm', etc).

And if your server returns something other than 401 when auth fails, then set $fail_code to whatever it actually returns. ⚠️ Heads up though: it needs to be something different than what you get on auth success, otherwise this function literally cannot tell the difference. it's giving blindfolded. 💀

🌸 auth_unset 🌸

Params: \%req

Return: nothing (modifies %req)

modifies %req to disable all authentication — regular and proxy. bye auth. 👋

Note: it only removes the values that were set by auth_set(). Manually-defined [Proxy-]Authorization headers will also be deleted — but honestly, you shouldn't be using the auth_* functions if you're manually handling your own auth anyway... 😭

🌸 auth_set 🌸

Params: $auth_method, \%req, $user, $password [, $domain]

Return: nothing (modifies %req)

modifies %req to use the authentication info you hand it. 🔐

Auth_method can be: 'basic', 'proxy-basic', 'ntlm', 'proxy-ntlm'.

Note: this function may not necessarily set any headers after being called. also, proxy-ntlm with SSL is not currently supported. ngl that one's a no-go. 🚫

Params: none

Return: $jar

creates a brand new cookie jar for use with the other functions. 🫙✨

Even though the jar is technically just a hash, you should still use this function to stay future-compatible — y'know, in case the jar format ever changes. future-proofing, we love her. 💖

Params: $jar, \%response [, \%request, $reject ]

Return: $num_of_cookies_read

reads cookies in from an %response hash and pops them into $jar.

Notice: cookie_read uses internal magic done by http_do_request so it can read cookies regardless of how the 'Set-Cookie[2]' header shows up. ✨

If the optional %request hash is supplied, it gets used to calculate default host and path values, in case the cookie doesn't explicitly specify them itself. And if $reject is set to 1, then the %request hash values get used to calculate and reject cookies that aren't appropriate for the path and domains of the given request. bye felicia. 🙅♀️

Params: $jar, $cookie [, $default_domain, $default_path, $reject ]

Return: nothing

parses the cookie into its various parts and then sets the appropriate values in the cookie $jar. If the cookie value is blank, it'll delete it from the $jar. 🗑️

for the full tea on how Libwhisker parses cookies and which RFC aspects are supported, go read the 'docs/cookies.txt' document. 📖

The optional $default_domain value is taken literally. Values with no leading dot (e.g. 'www.host.com') are considered to be strict hostnames and will only match the identical hostname. Values with leading dots (e.g. '.host.com') are treated as sub-domain matches for a single domain level. And if the cookie doesn't indicate a domain and no $default_domain was provided, then the cookie is considered to match all domains/hosts. it's giving universal. 🌈

The optional $default_path is used when the cookie doesn't specify a path. $default_path must be absolute (start with /), or it will just be ignored. If the cookie doesn't specify a path and $default_path isn't provided either, then the default value '/' gets used. 🛣️

Set $reject to 1 if you want to reject cookies based on the provided $default_domain and $default_path. ⚠️ Note that $default_domain and $default_path must be specified for $reject to actually do something meaningful — otherwise she's just vibing for nothing.

Params: $jar, \%request, $override

Return: nothing

goes through the given $jar and sets the Cookie header in %req, pending the correct domain and path. If $override is true, then the secure, domain and path restrictions of the cookies are ignored and all cookies are essentially included. all of them. no filter. 💅

Notice: cookie expiration is currently not implemented. URL restriction comparison is also case-insensitive.

Params: $jar, $name

Return: @elements

fetches the named cookie out of the $jar and returns the components. the returned items come back as an array in this exact order:

value, domain, path, expire, secure

  • value = cookie value, should always be a non-empty string
  • domain = domain root for cookie, can be undefined
  • path = URL path for cookie, should always be a non-empty string
  • expire = undefined (depreciated, but exists for backwards-compatibility)
  • secure = whether or not the cookie is limited to HTTPs; value is 0 or 1
Params: $jar

Return: @names

fetches all the cookie names from the jar, which then lets you cooke_get() them individually. one at a time, bestie. 🫶

Params: $jar, $domain, $url, $ssl

Return: @names

fetches all the cookie names from the jar that are valid for the given $domain, $url, and $ssl values. ✨

  • $domain should be a string scalar of the target host domain ('www.example.com', etc.).
  • $url should be the absolute URL for the page ('/index.html', '/cgi-bin/foo.cgi', etc.).
  • $ssl should be 0 for non-secure cookies, or 1 for all (secure and normal) cookies.

The return value is an array of names compatible with cookie_get(). 🎀

Params: $jar, $name, $value, $domain, $path, $expire, $secure

Return: nothing

sets the named cookie with the provided values into the %jar. 📥

  • $name is required to be a non-empty string.
  • $value is required, and will delete the named cookie from the $jar if it's an empty string.
  • $domain and $path can be strings or undefined.
  • $expire is ignored (but exists for backwards-compatibility).
  • $secure should be the numeric value 0 or 1.

🕷️ crawl_new 🕷️

Params: $START, $MAX_DEPTH, \%request_hash [, \%tracking_hash ]

Return: $crawl_object

crawl_new() initializes a crawl object (a hash) to the default values, then returns it so you can use it later with crawl(). 🧵

  • $START is the starting URL (in the form of 'http://www.host.com/url').
  • MAX_DEPTH is the maximum number of levels to crawl — and heads up, the START URL counts as 1, so a value of 2 will crawl the START URL plus all the URLs found on that page.
  • request_hash is a standard initialized request hash used for requests; you should set any authentication information or headers in this hash so the crawler actually uses them.
  • The optional tracking_hash lets you supply a hash for tracking URL results — otherwise crawl_new() will allocate a new anon hash itself. 🫙

🕷️ crawl 🕷️

Params: $crawl_object [, $START, $MAX_DEPTH ]

Return: $count [ undef on error ]

the heart of the crawl package, fr fr. 💖 Will perform an HTTP crawl on the specified HOST, starting at START URI, proceeding up to MAX_DEPTH.

Crawl_object needs to be the variable that crawl_new() returned. You can also indirectly call crawl() via the crawl_object itself, like this:

$crawl_object->{crawl}->($START,$MAX_DEPTH)

Returns the number of URLs actually crawled — not including the ones that got skipped. 🧮

💾 dump 💾

Params: $name, \@array [, $name, \%hash, $name, \$scalar ]

Return: $code [ undef on error ]

the dump function takes the given $name and data reference, and creates an ASCII perl code representation suitable for eval'ing later so you can recreate the same structure. $name is the name of the variable it'll be saved as. example:

$output = LW2::dump('request',\%request);
NOTE: dump() creates anonymous structures under the name given. So for example, if you dump the hash %hin under the name 'hin', then when you eval the dumped code you'll need to use %$hin, because $hin is now a reference to a hash. 🫠 don't forget the dollar sign, bestie.

💾 dump_writefile 💾

Params: $file, $name, \@array [, $name, \%hash, $name, \@scalar ]

Return: 0 if success; 1 if error

this calls dump() and saves the output to the specified $file. 📝

Note: LW does no checking on the validity of the file name, its creation, or anything of the sort. Files are opened in overwrite mode. so like... be careful what you point it at. 💀

🔤 encode_base64 🔤

Params: $data [, $eol]

Return: $b64_encoded_data

this function does Base64 encoding. ✨ If the binary MIME::Base64 module is available, it'll use that; otherwise it falls back to an internal perl version. That perl version carries the following copyright:

Copyright 1995-1999 Gisle Aas <gisle@aas.no>
NOTE: the $eol parameter will be inserted every 76 characters. This is used to format the data for output on an 80 character wide terminal. 🖥️

🔤 decode_base64 🔤

Params: $data

Return: $b64_decoded_data

and she decodes it right back. 🔁

A perl implementation of base64 decoding. The perl code for this function was actually taken from an older MIME::Base64 perl module, and bears the following copyright:

Copyright 1995-1999 Gisle Aas <gisle@aas.no>

encode_uri_hex

Params: $data

Return: $result

This one takes every single character and hex-encodes it the normal URL way — except for the / character, which gets left alone. bestie does not touch the slash. 🫶

encode_uri_randomhex

Params: $data

Return: $result

Same energy as above, but random — it picks characters at random to hex-encode, again skipping the / character. Chaos mode but make it URL-safe. ✨

encode_uri_randomcase

Params: $data

Return: $result

It just randomly flips the case of characters in the string. That's it. That's the whole function. little chaos gremlin behaviour, love that for it. 🦋

encode_unicode

Params: $data

Return: $result

Converts a normal string into Windows unicode format. Non-overlong, nothing fancy, no extra frills — just the conversion. 💅

decode_unicode

Params: $unicode_string

Return: $decoded_string

Attempts to decode a unicode (UTF-8) string by turning it into a single-byte-character string. Here's the tea on how it handles stuff:

  • Overlong characters get converted to their standard characters in place
  • Non-overlong (aka multi-byte) characters get substituted with the 0xff
  • Invalid encoding characters are just left as-is
Note: this function is useful for dealing with the various unicode exploits/vulnerabilities found in web servers; it is not good for doing actual UTF-8 parsing, since characters over a single byte are basically dropped/replaced with a placeholder.

encode_anti_ids

Params: \%request, $modes

Return: nothing

encode_anti_ids computes the proper anti-ids encoding/tricks specified by $modes, and sets up %hin in order to use those tricks. Valid modes are (and heads up, the mode numbers are the same as those found in whisker 1.4):

  • 1 — Encode some of the characters via normal URL encoding
  • 2 — Insert directory self-references (/./)
  • 3 — Premature URL ending (make it appear the request line is done)
  • 4 — Prepend a long random string in the form of "/string/../URL"
  • 5 — Add a fake URL parameter
  • 6 — Use a tab instead of a space as a request spacer
  • 7 — Change the case of the URL (works against Windows and Novell)
  • 8 — Change normal seperators ('/') to Windows version ('\')
  • 9 — Session splicing [NOTE: not currently available]
  • A — Use a carriage return (0x0d) as a request spacer
  • B — Use binary value 0x0b as a request spacer

You can set multiple modes by setting the string to contain all the modes desired; i.e. $modes="146" will use modes 1, 4, and 6. stack 'em, bestie, they're additive 💖

🎀 FORMS FUNCTIONS 🎀

ok so the goal here is to parse the variable, human-readable HTML into concrete structures useable by your program. The forms functions do a good job at making these structures, but ngl, I will admit: they are not exactly simple, and thus not a cinch to work with. But then again, representing something as complex as a HTML form is not a simple thing either. I think the results are acceptable for what's trying to be done. Anyways...

Forms are stored in perl hashes, with elements in the following format:

$form{'element_name'}=@([ 'type', 'value', @params ])

So every element in the hash is an array of anonymous arrays. The first array value contains the element type (which is 'select', 'textarea', 'button', or an 'input' value of the form 'input-text', 'input-hidden', 'input-radio', etc).

The second value is the value, if applicable (it could be undef if no value was specified). Note that select elements will always have an undef value — the actual values are in the subsequent options elements.

The third value, if defined, is an anonymous array of additional tag parameters found in the element (like 'onchange="blah"', 'size="20"', 'maxlength="40"', 'selected', etc).

The array does contain one special element, which is stored in the hash under a NULL character ("\0") key. This element is of the format:

$form{"\0"}=['name', 'method', 'action', @parameters];

The element is an anonymous array that contains strings of the form's name, method, and action (values can be undef), and a @parameters array similar to that found in normal elements (above).

Accessing individual values stored in the form hash becomes a test of your perl referencing skills. Hint: to access the 'value' of the third element named 'choices', you would need to do:

$form{'choices'}->[2]->[1];

The '[2]' is the third element (normal array starts with 0), and the actual value is '[1]' (the type is '[0]', and the parameter array is '[2]'). iykyk 🧚

forms_read

Params: \$html_data

Return: \@found_forms

This function parses the given $html_data into libwhisker form hashes. It returns a reference to an array of hash references to the found forms. ✨

forms_write

Params: \%form_hash

Return: $html_of_form [undef on error]

This function will take the given %form hash and compose a generic HTML representation of it, formatted with tabs and newlines in order to make it neat and tidy for printing. aesthetic 💅

Note: this function does not escape any special characters that were embedded in the element values.

html_find_tags

Params: \$data, \&callback_function [, $xml_flag, $funcref, \%tag_map]

Return: nothing

html_find_tags parses a piece of HTML and 'extracts' all found tags, passing the info to the given callback function. The callback function must accept two parameters: the current tag (as a scalar), and a hash ref of all the tag's elements. For example, the tag <a href="/file"> will pass 'a' as the current tag, and a hash reference which contains {'href'=>"/file"}.

The xml_flag, when set, causes the parser to do some extra processing and checks to accomodate XML style tags such as <tag foo="bar"/>.

The optional %tagmap is a hash of lowercase tag names. If a tagmap is supplied, then the parser will only call the callback function if the tag name exists in the tagmap.

The optional $funcref variable is passed straight to the callback function, allowing you to pass flags or references to more complex structures to your callback function. 🫶

html_find_tags_rewrite

Params: $position, $length, $replacement

Return: nothing

html_find_tags_rewrite() is used to 'rewrite' an HTML stream from within an html_find_tags() callback function. In general, you can think of html_find_tags_rewrite working as:

substr(DATA, $position, $length) = $replacement

Where DATA is the current HTML string the html parser is using. The reason you need to use this function and not substr() is because a few internal parser pointers and counters need to be adjusted to accomodate the changes.

If you want to remove a piece of the string, just set the replacement to an empty string (''). If you wish to insert a string instead of overwrite, just set $length to 0; your string will be inserted at the indicated $position. surgical. iconic. 💖

Params: \$html_data

Return: @urls

The html_link_extractor() function uses the internal crawl tests to extract all the HTML links from the given HTML data stream.

Note: html_link_extractor() does not unique the returned array of discovered links, nor does it attempt to remove javascript links or make the links absolute. It just extracts every raw link from the HTML stream and returns it. You'll have to do your own post-processing. no thoughts, just vibes and raw links 🍓

http_new_request

Params: %parameters

Return: \%request_hash

This function basically 'objectifies' the creation of whisker request hash objects. You would call it like:

$req = http_new_request( host=>'www.example.com', uri=>'/' )

where 'host' and 'uri' can be any number of {whisker} hash control values (see http_init_request for default list).

http_new_response

Params: [none]

Return: \%response_hash

This function basically 'objectifies' the creation of whisker response hash objects. You would call it like:

$resp = http_new_response()

http_init_request

Params: \%request_hash_to_initialize

Return: Nothing (modifies input hash)

Sets default values to the input hash for use. Sets the host to 'localhost', port 80, request URI '/', using HTTP 1.1 with GET method. The timeout is set to 10 seconds, no proxies are defined, and all URI formatting is set to standard HTTP syntax. It also sets the Connection (Keep-Alive) and User-Agent headers.

NOTICE!! It's important to use http_init_request before calling http_do_request, or http_do_request might puke. Thus, a special magic value is placed in the hash to let http_do_request know that the hash has been properly initialized. If you really must 'roll your own' and not use http_init_request before you call http_do_request, you will at least need to set the MAGIC value (amongst other things). fr fr do not skip this step 😭

http_do_request

Params: \%request, \%response [, \%configs]

Return: >=1 if error; 0 if no error (also modifies response hash)

THE core function of libwhisker. 👑

http_do_request actually performs the HTTP request, using the values submitted in %request, and placing result values in %response. This allows you to resubmit %request in subsequent requests (%response is automatically cleared upon execution). You can submit 'runtime' config directives as %configs, which will be spliced into $hin{whisker}->{} before anything else. That means you can do:

LW2::http_do_request(\%req,\%resp,{'uri'=>'/cgi-bin/'});

This will set $req{whisker}->{'uri'}='/cgi-bin/' before execution, and provides a simple shortcut (note: it does modify %req).

This function will also retry any requests that bomb out during the transaction (but not during the connecting phase). This is controlled by the {whisker}->{retry} value. Also note that the returned error message in hout is the last error received. All retry errors are put into {whisker}->{retry_errors}, which is an anonymous array. 💀

Also note that all NTLM auth logic is implemented in http_do_request(). NTLM requires multiple requests in order to work correctly, and so this function attempts to wrap that and make it all transparent, so that the final end result is what's passed to the application. the way it just handles all of that for you... main character energy honestly ✨

This function will return 0 on success, 1 on HTTP protocol error, and 2 on non-recoverable network connection error — and bestie, pls note the difference: you can retry error 1, but error 2 means the server is totally unreachable and there's literally no point in retrying. 💀

✨ More Functions, Let's Go ✨

💖 http_req2line

Params: \%request, $uri_only_switch

Return: $request

req2line is used internally by http_do_request, and it's also just a super convenient way to turn a %request configuration into an actual HTTP request line. If $switch is set to 1, then the $request you get back will be the URI only ('/requested/page.html'), versus the entire HTTP request ('GET /requested/page.html HTTP/1.0\n\n'). Also, if the 'full_request_override' whisker config variable is set in %hin, then that gets returned instead of the constructed URI. 🎀

🦋 http_resp2line

Params: \%response

Return: $response

http_resp2line gives you a convenient way to turn a %response hash back into the original HTTP response line. Slay. ✨

🌸 http_fixup_request

Params: $hash_ref

Return: Nothing

This function takes a %hin hash reference and makes sure the proper headers exist (for example, it will add the Host: header, calculate the Content-Length: header for POST requests, etc). For standard requests — i.e. you want the request to be HTTP RFC-compliant — you should call this function right before you call http_do_request. 👑

🧚 http_reset

Params: Nothing

Return: Nothing

The http_reset function will walk through the %http_host_cache, closing all open sockets and freeing SSL resources. It also clears out the host cache in case you need to rerun everything fresh. 🫶

Note: if you just want to close a single connection, and you have a copy of the %request hash you used, you should use the http_close() function instead. Don't do too much, ok?

⭐ ssl_is_available

Params: Nothing

Return: $boolean [, $lib_name, $version]

The ssl_is_available() function will inform you whether SSL requests are allowed, which is dependant on whether the appropriate SSL libraries are installed on the machine. In scalar context, the function will return 1 or 0. In array context, the second element will be the SSL library name that is currently being used by LW2, and the third elment will be the SSL library version number. Elements two and three (name and version) will be undefined if called in array context and no SSL libraries are available. 💅

🌈 http_read_headers

Params: $stream, \%in, \%out

Return: $result_code, $encoding, $length, $connection

Read HTTP headers from the given stream, storing the results in %out. On success, $result_code will be 1 and $encoding, $length, and $connection will hold the values of the Transfer-Encoding, Content-Length, and Connection headers, respectively. If any of those headers are not present, then it will have an 'undef' value. On an error, the $result_code will be 0 and $encoding will contain an error message. 😭

This function can be used to parse both request and response headers.

Note: if there are multiple Transfer-Encoding, Content-Length, or Connection headers, then only the last header value is the one returned by the function. Periodt.

🎀 http_read_body

Params: $stream, \%in, \%out, $encoding, $length

Return: 1 on success, 0 on error (and sets $hout->{whisker}->{error})

Read the body from the given stream, placing it in $out->{whisker}->{data}. Handles chunked encoding. Can be used to read HTTP (POST) request or HTTP response bodies. The $encoding parameter should be lowercase encoding type.

NOTE: $out->{whisker}->{data} is erased/cleared when this function is called, leaving {data} to just contain this particular HTTP body. So don't be shocked when your old data vanishes, it's giving intentional. 💅

💜 http_construct_headers

Params: \%in

Return: $data

This function assembles the headers in the given hash into a data string. Simple as that, no cap. ✨

🔥 http_close

Params: \%request

Return: nothing

This function will close any open streams for the given request.

Note: in order for http_close() to find the right connection, all original host/proxy/port parameters in %request must be the exact same as when the original request was made. Like, character for character, bestie. 🙈

🦋 http_do_request_timeout

Params: \%request, \%response, $timeout

Return: $result

This function is identical to http_do_request(), except that it wraps the entire request in a timeout wrapper. $timeout is the number of seconds to allow for the entire request to be completed. ⏰

Note: this function uses alarm() and signals, and thus will only work on Unix-ish platforms. It should be safe to call on any platform though. 🫶

🍓 md5

Params: $data

Return: $hex_md5_string

This function takes a data scalar, and composes a MD5 hash of it, and returns it in a hex ascii string. It will use the fastest MD5 function available. Main character energy from a hash function, honestly. 👑

💖 md4

Params: $data

Return: $hex_md4_string

This function takes a data scalar, and composes a MD4 hash of it, and returns it in a hex ascii string. It will use the fastest MD4 function available. ✨

🌸 multipart_set

Params: \%multi_hash, $param_name, $param_value

Return: nothing

This function sets the named parameter to the given value within the supplied multipart hash.

🧚 multipart_get

Params: \%multi_hash, $param_name

Return: $param_value, undef on error

This function retrieves the named parameter to the given value within the supplied multipart hash. There is a special case where the named parameter is actually a file — in which case the resulting value will be "\0FILE". In general, all special values will be prefixed with a NULL character. In order to get a file's info, use multipart_getfile(). 🎀

⭐ multipart_setfile

Params: \%multi_hash, $param_name, $file_path [, $filename]

Return: undef on error, 1 on success

NOTE: this function does not actually add the contents of $file_path into the %multi_hash; instead, multipart_write() inserts the content when generating the final request. Sneaky but on purpose. 💅

🌈 multipart_getfile

Params: \%multi_hash, $file_param_name

Return: $path, $name ($path=undef on error)

multipart_getfile is used to retrieve information for a file parameter contained in %multi_hash. To use this you would most likely do:

($path,$fname)=LW2::multipart_getfile(\%multi,"param_name");

💜 multipart_boundary

Params: \%multi_hash [, $new_boundary_name]

Return: $current_boundary_name

multipart_boundary is used to retrieve, and optionally set, the multipart boundary used for the request.

NOTE: the function does no checking on the supplied boundary, so if you want things to work make sure it's a legit boundary. Libwhisker does *not* prefix it with any '---' characters. So like, don't be messy with it. 😭

🔥 multipart_write

Params: \%multi_hash, \%request

Return: 1 if successful, undef on error

multipart_write is used to parse and construct the multipart data contained in %multi_hash, and place it ready to go in the given whisker hash (%request) structure, to be sent to the server. 🫶

NOTE: file contents are read into the final %request, so it's possible for the hash to get *very* large if you have (a) large file(s). Like, watch your memory bestie.

🎀 multipart_read

Params: \%multi_hash, \%hout_response [, $filepath ]

Return: 1 if successful, undef on error

multipart_read will parse the data contents of the supplied %hout_response hash, by passing the appropriate info to multipart_read_data(). Please see multipart_read_data() for more info on parameters and behaviour. 👑

NOTE: this function will return an error if the given %hout_response Content-Type is not set to "multipart/form-data". Vibe check failed, basically. 💀

🦋 multipart_read_data

Params: \%multi_hash, \$data, $boundary [, $filepath ]

Return: 1 if successful, undef on error

multipart_read_data parses the contents of the supplied data using the given boundary and puts the values in the supplied %multi_hash. Embedded files will *not* be saved unless a $filepath is given, which should be a directory suitable for writing out temporary files.

NOTE: currently only application/octet-stream is the only supported file encoding. All other file encodings will not be parsed/saved. That's just how it is, periodt. ✨

🍓 multipart_files_list

Params: \%multi_hash

Return: @files

multipart_files_list returns an array of parameter names for all the files that are contained in %multi_hash.

🌸 multipart_params_list

Params: \%multi_hash

Return: @params

multipart_files_list returns an array of parameter names for all the regular parameters (non-file) that are contained in %multi_hash. (Yes, the original says multipart_files_list here too — we're keeping it exactly as written, no editing the lore. 🙈)

🧚 ntlm_new

Params: $username, $password [, $domain, $ntlm_only]

Return: $ntlm_object

Returns a reference to an array (otherwise known as the 'ntlm object') which contains the various informations specific to a user/pass combo. If $ntlm_only is set to 1, then only the NTLM hash (and not the LanMan hash) will be generated. This results in a speed boost, and is typically fine for using against IIS servers. 🔥

The array contains the following items, in order: username, password, domain, lmhash(password), ntlmhash(password)

⭐ ntlm_decode_challenge

Params: $challenge

Return: @challenge_parts

Splits the supplied challenge into the various parts. The returned array contains elements in the following order:

unicode_domain, ident, packet_type, domain_len, domain_maxlen, domain_offset, flags, challenge_token, reserved, empty, raw_data

💜 ntlm_client

Params: $ntlm_obj [, $server_challenge]

Return: $response

ntlm_client() is responsible for generating the base64-encoded text you include in the HTTP Authorization header. If you call ntlm_client() without a $server_challenge, the function will return the initial NTLM request packet (message packet #1). You send this to the server, and take the server's response (message packet #2) and pass that as $server_challenge, causing ntlm_client() to generate the final response packet (message packet #3). It's giving three-act structure, honestly. 👑

Note: $server_challenge is expected to be base64 encoded.

🌈 get_page

Params: $url [, \%request]

Return: $code, $data ($code will be set to undef on error, $data will contain error message)

...okay so this one fetches the page at the URL you give it and hands you back BOTH the HTTP response code and the page contents. Obsessed with a two-for-one. You use it like this:

($code,$html)=LW2::get_page("http://host.com/page.html")

The optional %request gets used if you supply it — that's your little backstage pass for setting headers and other parameters. 💅

✨ get_page_hash ✨

Params: $url [, \%request]

Return: $hash_ref (undef on no URL)

This one fetches the page at the given URL and gives you back the whisker HTTP response hash instead. The return code of the function gets set to $hash_ref->{whisker}->{get_page_hash}, and it uses the http_do_request() return values. Fancy.

Note: undef is returned if no URL is supplied. No URL, no vibes. 🫥

🌸 get_page_to_file 🌸

Params: $url, $filepath [, \%request]

Return: $code ($code will be set to undef on error)

Fetches the page at the given URL, drops the resulting HTML into the file you specified, and returns the HTTP response code. The optional %request hash sets the default parameters used in the request.

NOTE: libwhisker does not do any file checking, bestie. It will open the $filepath you gave it for writing and overwrite any previously-existing files — no cap, it does not care. It also does not differentiate between a bad request and a bad file open. So if you're having troubles making this function work, make sure your $filepath is legal and valid, and that you have the appropriate write permissions to create/overwrite that file. 🔥

⏰ time_mktime ⏰

Params: $seconds, $minutes, $hours, $day_of_month, $month, $year_minus_1900

Return: $seconds [ -1 on error ]

Does a general mktime calculation with the time components you hand it. Heads up: the input parameter values are expected to be in the format that localtime/gmtime output. So like, $seconds is 0-60 (yes, there can be a leap second value of 60 occasionally, iconic), $minutes is 0-59, $hours is 0-23, $days is 1-31, $month is 0-11, and $year is 70-127. This function is limited though — it will not process dates prior to 1970 or after 2037. That's on purpose, so the 32-bit time_t overflow calculations aren't required. 💀

Any additional parameters you pass get ignored, so it's totally safe to just use the full localtime/gmtime output, like:

$seconds = LW2::time_mktime( localtime( time ) );
Note: this function does not adjust for time zone, daylight savings time, etc. You gotta do that yourself, girl. 🕰️

🦋 time_gmtolocal 🦋

Params: $seconds_gmt

Return: $seconds_local_timezone

Takes a seconds value in UTC/GMT time and adjusts it to reflect the current timezone. Lowkey this one is a bit expensive: it takes the gmtime() and localtime() representations of the current time, calculates the delta difference by turning them back into seconds via time_mktime, and then applies that delta difference to $seconds_gmt.

Note that if you give this function a time and then subtract the return value from the original time, you get the delta value. At that point you can just apply the delta directly and skip calling this function — which is a massive performance boost, slay. BUT. This will cause problems if you have a long running program which crosses daylight savings time boundaries, because the DST adjustment will not be accounted for unless you recalculate the new delta. So keep that in mind. 🌈

🔗 uri_split 🔗

Params: $uri_string [, \%request_hash]

Return: @uri_parts

Returns an array of the following values, in order: uri, protocol, host, port, params, frag, user, password. Any values that aren't defined get an undef value — everyone gets a seat at the table, even the undefs. 💜 If a %request hash is passed in, then uri_split() will also set the appropriate values in that hash.

Note: uri_split() will only set the %request hash if the protocol is HTTP or HTTPS! Any other protocol and the hash gets nothing. Periodt.

🎀 uri_join 🎀

Params: @vals

Return: $url

Takes the @vals array output from http_split_uri, and returns a single scalar/string with them joined back together again, in the form of:

protocol://user:pass@host:port/uri?params#frag

Putting the little URL back together like a friendship bracelet. 🫶

✨ uri_absolute ✨

Params: $uri, $base_uri [, $normalize_flag ]

Return: $absolute_uri

Double checks that the $uri you gave is in absolute form (that is, "http://host/file"), and if it's not (i.e. it's in the form "/file"), then it appends the given $base_uri to make it absolute. This provides compatibility similar to that found in the URI subpackage.

If $normalize_flag is set to 1, then the output gets passed through uri_normalize before being returned. ✨

🧚 uri_normalize 🧚

Params: $uri [, $fix_windows_slashes ]

Return: $normalized_uri [ undef on error ]

Takes the given $uri and does any /./ and /../ dereferencing so it can come up with the correct absolute URL. If the $fix_windows_slashes parameter is set to 1, all \ (back slashes) will be converted to / (forward slashes). Windows girlies, this one's for you.

Non-http/https URIs return an error. Sorry. 💀

📁 uri_get_dir 📁

Params: $uri

Return: $uri_directory

Will take a URI and return the directory base of it — i.e. /rfp/page.php will return /rfp/. It's giving "just the folder, thanks." 👑

🍓 uri_strip_path_parameters 🍓

Params: $uri [, \%param_hash ]

Return: $stripped_uri

This function removes all URI path parameters of the form

/blah1;foo=bar/blah2;baz

and returns the stripped URI ('/blah1/blah2'). If the optional parameter hash reference is provided, the stripped parameters are saved in the form of 'blah1'=>'foo=bar', 'blah2'=>'baz'.

Note: only the last value of a duplicate name is saved into the param_hash, if provided. So a $uri of '/foo;A/foo;B/' will result in a single hash entry of 'foo'=>'B'. The rest got left on read. 😭

💖 uri_parse_parameters 💖

Params: $parameter_string [, $decode, $multi_flag ]

Return: \%parameter_hash

This function takes a string in the form of:

foo=1&bar=2&baz=3&foo=4

And parses it into a hash. In the example above, the element 'foo' has two values (1 and 4). If $multi_flag is set to 1, then the 'foo' hash entry will hold an anonymous array of both values. Otherwise, the default is to just contain the last value (in this case, '4').

If $decode is set to 1, then normal hex decoding is done on the characters, where needed — and yes, both the name and the value get decoded. 💅

Note: if a URL parameter name appears without a value, then the value will be set to undef. E.g. for the string "foo=1&bar&baz=2", the 'bar' hash element will have an undef value. Bar said nothing and meant it.

🌈 uri_escape 🌈

Params: $data

Return: $encoded_data

This function encodes the given $data so it is safe to be used in URIs. Making it URL-safe, as a treat. ✨

🦋 uri_unescape 🦋

Params: $encoded_data

Return: $data

This function decodes the given $data out of URI format. The reverse glow-up. 💜

🔥 utils_recperm 🔥

Params: $uri, $depth, \@dir_parts, \@valid, \&func, \%track, \%arrays, \&cfunc

Return: nothing

This is a special function which is used to recursively-permutate through a given directory listing. Real talk, this is really only used by whisker, in order to traverse down directories, testing them as it goes. See whisker 2.0 for exact usage examples. 📚

🎀 utils_array_shuffle 🎀

Params: \@array

Return: nothing

This function will randomize the order of the elements in the given array. Shuffling the deck, no cap. 🔀

⭐ utils_randstr ⭐

Params: [ $size, $chars ]

Return: $random_string

This function generates a random string between 10 and 20 characters long, or of $size if you specify it. If $chars is specified, then the random function picks characters from the supplied string. For example, to have a random string of 10 characters composed of only the characters 'abcdef', you would run:

utils_randstr(10,'abcdef');

The default character string is alphanumeric. Very demure, very random. 🎲

🫶 utils_port_open 🫶

Params: $host, $port

Return: $result

Quick function to attempt to make a connection to the given host and port. If a connection was successfully made, the function returns true (1). Otherwise it returns false (0). That's it, that's the function. 💅

Note: this uses standard TCP connections, so it is not recommended for use in port-scanning type applications. Extremely slow. Like, watching-paint-dry slow. 😭

💜 utils_lowercase_keys 💜

Params: \%hash

Return: $number_changed

Will lowercase all the header names (but not the values) of the given hash. Names only, the values stay exactly as they are. 👑

🌸 utils_find_lowercase_key 🌸

Params: \%hash, $key

Return: $value, undef on error or not exist

Searches the given hash for the $key (regardless of case), and returns the value. If the return value is placed into an array, it will dereference any multi-value references and return an array of all values.

WARNING! In scalar context, $value can either be a single-value scalar or an array reference for multiple scalar values. That means you either need to check the return value and act appropriately, or use an array context (even if you only want a single value). This is very important, even if you know there are no multi-value hash keys. This function may still return an array of multiple values even if all hash keys are single value, since lowercasing the keys could result in multiple keys matching. For example, a hash with the values { 'Foo'=>'a', 'fOo'=>'b' } technically has two keys with the lowercase name 'foo', and so this function will either return an array or array reference with both 'a' and 'b'. So like, be careful bestie. 😭

🎀 utils_find_key 🎀

Params: \%hash, $key

Return: $value, undef on error or not exist

Searches the given hash for the $key (case-sensitive), and returns the value. If the return value is placed into an array, it will dereference any multi-value references and return an array of all values. Exact match only, this one does not play around with cases. 💅

🧚 utils_delete_lowercase_key 🧚

Params: \%hash, $key

Return: $number_found

Searches the given hash for the $key (regardless of case), and deletes the key out of the hash if found. The function returns the number of keys found and deleted — because multiple keys can exist under the names 'Key', 'key', 'keY', 'KEY', etc. All of them. Gone. 🫥

🍓 utils_getline 🍓

Params: \$data [, $resetpos ]

Return: $line (undef if no more data)

Fetches the next \n terminated line from the given data. Use the optional $resetpos to reset the internal position pointer. Does *NOT* return trailing \n. She's tidy like that. ✨

🦋 utils_getline_crlf 🦋

Params: \$data [, $resetpos ]

Return: $line (undef if no more data)

Okay so this next one fetches the next \r\n terminated line from the given data. Use the optional $resetpos to reset the internal position pointer. And it does NOT return the trialing \r\n. Periodt. 💅

utils_save_page

Params: $file, \%response

Return: 0 on success, 1 on error

Saves the data portion of the given whisker %response hash to the file you indicate. You can technically save the data portion of a %request hash too. If there's no data, no file gets written.

Note: LW does not do any special file checking; files are opened in overwrite mode. So like, be careful, it'll just overwrite whatever's there. 💀

utils_getopts

Params: $opt_str, \%opt_results

Return: 0 on success, 1 on error

This is a general implementation of GetOpts::Std. It parses @ARGV, looking for the options specified in $opt_str, and puts the results in %opt_results. Behavior and parameter values are similar to GetOpts::Std's getopts().

Note: This function does not support long options (--option), option grouping (-opq), or options with immediate values (-ovalue). If an option is indicated as having a value, it will take the next argument regardless. No cap. 🧢

utils_text_wrapper

Params: $long_text_string [, $crlf, $width ]

Return: $formatted_test_string

A simple function used to format a long line of text for display on a typical limited-character screen, such as a unix shell console.

$crlf defaults to "\n", and $width defaults to 76. Slay. ✨

utils_bruteurl

Params: \%req, $pre, $post, \@values_in, \@values_out

Return: Nothing (adds to @out)

Bruteurl will perform a brute force against the host/server specified in %req. However, it will make one request per entry in @in, taking the value and setting $hin{'whisker'}->{'uri'}= $pre.value.$post. Any URI responding with an HTTP 200 or 403 response is pushed into @out. An example of this would be to brute force usernames, putting a list of common usernames in @in, setting $pre='/~' and $post='/'. So iconic. 🔥

utils_join_tag

Params: $tag_name, \%attributes

Return: $tag_string [undef on error]

This function takes the $tag_name (like 'A') and a hash full of attributes (like {href=>'http://foo/'}) and returns the constructed HTML tag string (<A href="http://foo">). Cute. 🎀

utils_request_clone

Params: \%from_request, \%to_request

Return: 1 on success, 0 on error

This function takes the connection/request-specific values from the given %from_request hash, and copies them to the %to_request hash. Simple as that. 💖

utils_request_fingerprint

Params: \%request [, $hash ]

Return: $fingerprint [undef on error]

This function constructs a 'fingerprint' of the given request by using a cryptographic hashing function on the constructed original HTTP request.

Note: $hash can be 'md5' (default) or 'md4'.

utils_flatten_lwhash

Params: \%lwhash

Return: $flat_version [undef on error]

This function takes a %request or %response libwhisker hash, and creates an approximate flat data string of the original request/response (i.e. before it was parsed into components and placed into the libwhisker hash). So like, it flattens it back out. 🦋

utils_carp

Params: [ $package_name ]

Return: nothing

This function acts like Carp's carp function. It warn's with the file and line number of user's code which causes a problem. It traces up the call stack and reports the first function that is not in the LW2 or optional $package_name package package. (Yes, the original literally says "package package" twice. We're keeping it. 😭)

utils_croak

Params: [ $package_name ]

Return: nothing

This function acts like Carp's croak function. It die's with the file and line number of user's code which causes a problem. It traces up the call stack and reports the first function that is not in the LW2 or optional $package_name package package. (Again, "package package" is in the original. We're not fixing it. 💀)

SEE ALSO

LWP

Copyright 2009 Jeff Forristal

This page is from perl v5.32.1, dated 2022-01-01, and it's the LW2(3perl) manpage. Bye bestie! 💖✨

📦 where this page comes from