LW2(3)

📦
version 2.5 · 2 versions
, in fedora 43

💖 NAME 💖

LW2 — it's the Perl HTTP library, version 2.5, bestie. That's the whole vibe. (It's the LW2(3) page from User Contributed Perl Documentation, and yes you're gonna read all of it. 💅)

✨ SYNOPSIS ✨

use LW2;

require 'LW2.pm';

🦋 DESCRIPTION 🦋

okay so, Libwhisker is a Perl library that's literally made for HTTP testing scripts. 🎀 And the way it does it? It's a pure-Perl reimplementation of all the functionality you'd normally go hunting for across the LWP, URI, Digest::MD5, Digest::MD4, Data::Dumper, Authen::NTLM, HTML::Parser, HTML::FormParser, CGI::Upload, MIME::Base64, and GetOpt::Std modules. all of them. in one place. iconic. 👑

Libwhisker is designed to be portable (it's a single perl file, no cap), fast (general benchmarks show libwhisker is faster than LWP, and we do love to see it 🔥), and flexible — great care was taken to make sure the library does exactly what you want it to do, even if that means breaking the protocol. yeah. it'll do crimes for you if you ask nicely. 💀

🌸 FUNCTIONS 🌸

The following are the functions contained in Libwhisker — this is the whole roster, no skips:

💅 auth_brute_force

Params: $auth_method, \%req, $user, \@passwords [, $domain, $fail_code ]

Return: $first_valid_password, or undef if there's an error / nothing found

Perform an HTTP authentication brute force against a server (the host and URI are defined inside %req). It'll try every password in the password array for the given user. The first password (paired with the given user) that doesn't come back with HTTP 401 is the one it returns — and the brute force stops right there, we're done, no more guessing. 🛑

You should retry the request with that password and double-check you got a useful HTTP return code that indicates successful authentication (200, 302), and not something a bit more abnormal (407, 500, etc). don't trust the first thing you see, bestie.

$domain is optional, and it's only used for NTLM auth.

Note: set up any proxy settings and proxy auth in %req before calling this function. 🔑

You can brute-force proxy authentication by setting up the target proxy as proxy_host and proxy_port in %req, using an arbitrary host and uri (preferably one that is reachable upon successful proxy authorization), and setting the $fail_code to 407. The $auth_method you pass to this function should be a proxy-based one ('proxy-basic', 'proxy-ntlm', etc).

if your server returns something other than 401 upon auth failure, then set $fail_code to whatever is returned (and it needs to be something *different* than what is received on auth success, or this function won't be able to tell the difference). fr fr, it's gotta be different or the whole thing is confused. 😭

🧚 auth_unset

Params: \%req

Return: nothing (it modifies %req)

Modifies %req to disable all authentication (regular and proxy).

Note: it only removes the values set by auth_set(). Manually-defined [Proxy-]Authorization headers will also be deleted (but you shouldn't be using the auth_* functions if you're manually handling your own auth... just saying 🙈).

⭐ auth_set

Params: $auth_method, \%req, $user, $password [, $domain]

Return: nothing (it modifies %req)

Modifies %req to use the indicated authentication info.

Auth_method can be: 'basic', 'proxy-basic', 'ntlm', 'proxy-ntlm'.

Note: this function may not necessarily set any headers after being called. Also, proxy-ntlm with SSL is not currently supported. 🚫

Params: none

Return: $jar

Create a new cookie jar, for use with the other functions. Even though the jar is technically just a hash, you should still use this function in order to be future-compatible (should the jar format change). yes it looks like extra steps, no it's not, do it anyway 💖

Params: $jar, \%response [, \%request, $reject ]

Return: $num_of_cookies_read

Read in cookies from an %response hash, and put them in $jar.

Notice: cookie_read uses internal magic done by http_do_request in order to read cookies regardless of 'Set-Cookie[2]' header appearance. ✨ it's giving wizardry, honestly.

If the optional %request hash is supplied, then it will be used to calculate default host and path values, in case the cookie doesn't specify them explicitly. If $reject is set to 1, then the %request hash values are used to calculate and reject cookies which are not appropriate for the path and domains of the given request. bye felicia 👋

Params: $jar, $cookie [, $default_domain, $default_path, $reject ]

Return: nothing

Parses the cookie into the various parts and then sets the appropriate values in the cookie $jar. If the cookie value is blank, it will delete it from the $jar. See the 'docs/cookies.txt' document for a full explanation of how Libwhisker parses cookies and what RFC aspects are supported. 📚

The optional $default_domain value is taken literally. Values with no leading dot (e.g. 'www.host.com') are considered to be strict hostnames and will only match the identical hostname. Values with leading dots (e.g. '.host.com') are treated as sub-domain matches for a single domain level. If the cookie does not indicate a domain, and a $default_domain is not provided, then the cookie is considered to match all domains/hosts. so... everything. wild, but okay. 💀

The optional $default_path is used when the cookie does not specify a path. $default_path must be absolute (start with '/'), or it will be ignored. If the cookie does not specify a path, and $default_path is not provided, then the default value '/' will be used.

Set $reject to 1 if you wish to reject cookies based upon the provided $default_domain and $default_path. Note that $default_domain and $default_path must be specified for $reject to actually do something meaningful. ⚠️

Params: $jar, \%request, $override

Return: nothing

Goes through the given $jar and sets the Cookie header in %req pending the correct domain and path. If $override is true, then the secure, domain and path restrictions of the cookies are ignored and all cookies are essentially included. no rules, just vibes. 💅

Notice: cookie expiration is currently not implemented. URL restriction comparision is also case-insensitive. 📝

Params: $jar, $name

Return: @elements

Fetch the named cookie from the $jar, and return the components. The returned items will be an array in the following order:

value, domain, path, expire, secure
  • value = cookie value, should always be non-empty string
  • domain = domain root for cookie, can be undefined
  • path = URL path for cookie, should always be a non-empty string
  • expire = undefined (depreciated, but exists for backwards-compatibility)
  • secure = whether or not the cookie is limited to HTTPs; value is 0 or 1

Params: $jar

Return: @names

Fetch all the cookie names from the jar, which then let you cooke_get() them individually. (yes, that spelling, we keep it 💖)

Params: $jar, $domain, $url, $ssl

Return: @names

Fetch all the cookie names from the jar which are valid for the given $domain, $url, and $ssl values. $domain should be string scalar of the target host domain ('www.example.com', etc.). $url should be the absolute URL for the page ('/index.html', '/cgi-bin/foo.cgi', etc.). $ssl should be 0 for non-secure cookies, or 1 for all (secure and normal) cookies. The return value is an array of names compatible with cookie_get(). ✨

Params: $jar, $name, $value, $domain, $path, $expire, $secure

Return: nothing

Set the named cookie with the provided values into the %jar. $name is required to be a non-empty string. $value is required, and will delete the named cookie from the $jar if it is an empty string. $domain and $path can be strings or undefined. $expire is ignored (but exists for backwards-compatibility). $secure should be the numeric value of 0 or 1.

🧵 crawl_new

Params: $START, $MAX_DEPTH, \%request_hash [, \%tracking_hash ]

Return: $crawl_object

The crawl_new() functions initializes a crawl object (hash) to the default values, and then returns it for later use by crawl(). $START is the starting URL (in the form of 'http://www.host.com/url'), and MAX_DEPTH is the maximum number of levels to crawl (the START URL counts as 1, so a value of 2 will crawl the START URL and all URLs found on that page). The request_hash is a standard initialized request hash to be used for requests; you should set any authentication information or headers in this hash in order for the crawler to use them. The optional tracking_hash lets you supply a hash for use in tracking URL results (otherwise crawl_new() will allocate a new anon hash). 🕸️

👑 crawl

Params: $crawl_object [, $START, $MAX_DEPTH ]

Return: $count [ undef on error ]

The heart of the crawl package. 💓 Will perform an HTTP crawl on the specified HOST, starting at START URI, proceeding up to MAX_DEPTH.

Crawl_object needs to be the variable returned by crawl_new(). You can also indirectly call crawl() via the crawl_object itself:

$crawl_object->{crawl}->($START,$MAX_DEPTH)

Returns the number of URLs actually crawled (not including those skipped).

💀 dump

Params: $name, \@array [, $name, \%hash, $name, \$scalar ]

Return: $code [ undef on error ]

The dump function will take the given $name and data reference, and will create an ASCII perl code representation suitable for eval'ing later to recreate the same structure. $name is the name of the variable that it will be saved as. Example:

$output = LW2::dump('request',\%request);

NOTE: dump() creates anonymous structures under the name given. For example, if you dump the hash %hin under the name 'hin', then when you eval the dumped code you will need to use %$hin, since $hin is now a *reference* to a hash. don't get caught out by this one bestie 😭

🎁 dump_writefile

Params: $file, $name, \@array [, $name, \%hash, $name, \@scalar ]

Return: 0 if success; 1 if error

This calls dump() and saves the output to the specified $file.

Note: LW does not checking on the validity of the file name, it's creation, or anything of the sort. Files are opened in overwrite mode. ⚠️ so, like, be careful what name you hand it.

🌸 encode_base64

Params: $data [, $eol]

Return: $b64_encoded_data

This function does Base64 encoding. If the binary MIME::Base64 module is available, it will use that; otherwise, it falls back to an internal perl version. The perl version carries the following copyright:

Copyright 1995-1999 Gisle Aas <gisle@aas.no>

NOTE: the $eol parameter will be inserted every 76 characters. This is used to format the data for output on a 80 character wide terminal. 📐

💖 decode_base64

Params: $data

Return: $b64_decoded_data

A perl implementation of base64 decoding. The perl code for this function was actually taken from an older MIME::Base64 perl module, and bears the following copyright:

Copyright 1995-1999 Gisle Aas <gisle@aas.no>

💅 encode_uri_hex

  • Params: $data
  • Return: $result

This one encodes every character (except the / character, she stays unbothered) with normal URL hex encoding. 🎀

✨ encode_uri_randomhex

  • Params: $data
  • Return: $result

Same energy, but it randomly picks which characters (again, not the / character, she's exempt) get the normal URL hex encoding treatment. Chaos, but controlled. 🙈

🦋 encode_uri_randomcase

  • Params: $data
  • Return: $result

Randomly flips the case of characters in the string. So SeLeCt vibes. No cap.

🌸 encode_unicode

  • Params: $data
  • Return: $result

Converts a normal string into Windows unicode format — and like, the plain kind, non-overlong, nothing fancy or extra. Just straight up.

💜 decode_unicode

  • Params: $unicode_string
  • Return: $decoded_string

Okay so this function attempts to decode a unicode (UTF-8) string by converting it into a single-byte-character string. Here's the tea:

  • Overlong characters get converted to their standard characters in place
  • Non-overlong (aka multi-byte) characters get substituted with 0xff
  • Invalid encoding characters are left as-is
Note: this function is useful for dealing with the various unicode exploits/vulnerabilities found in web servers; it is not good for doing actual UTF-8 parsing, since characters over a single byte are basically dropped/replaced with a placeholder. So like — great for the hacker girlies, terrible for your legit UTF-8 needs. Periodt.

🔥 encode_anti_ids

  • Params: \%request, $modes
  • Return: nothing

encode_anti_ids computes the proper anti-ids encoding/tricks specified by $modes, and sets up %hin in order to use those tricks. Valid modes are (the mode numbers are the same as those found in whisker 1.4):

  1. Encode some of the characters via normal URL encoding
  2. Insert directory self-references (/./)
  3. Premature URL ending (make it appear the request line is done)
  4. Prepend a long random string in the form of "/string/../URL"
  5. Add a fake URL parameter
  6. Use a tab instead of a space as a request spacer
  7. Change the case of the URL (works against Windows and Novell)
  8. Change normal seperators ('/') to Windows version ('\')
  9. Session splicing [NOTE: not currently available] A. Use a carriage return (0x0d) as a request spacer B. Use binary value 0x0b as a request spacer

You can set multiple modes by setting the string to contain all the modes desired; i.e. $modes="146" will use modes 1, 4, and 6. Stack 'em, bestie. 💅

🎀 FORMS FUNCTIONS 🎀

The goal is to parse the variable, human-readable HTML into concrete structures useable by your program. The forms functions does do a good job at making these structures, but I will admit: they are not exactly simple, and thus not a cinch to work with. But then again, representing something as complex as a HTML form is not a simple thing either. I think the results are acceptable for what's trying to be done. Anyways...

Forms are stored in perl hashes, with elements in the following format:

$form{'element_name'}=@([ 'type', 'value', @params ])

So every element in the hash is an array of anonymous arrays. The first array value contains the element type (which is 'select', 'textarea', 'button', or an 'input' value of the form 'input-text', 'input-hidden', 'input-radio', etc).

The second value is the value, if applicable (it could be undef if no value was specified). Note that select elements will always have an undef value — the actual values are in the subsequent options elements.

The third value, if defined, is an anonymous array of additional tag parameters found in the element (like 'onchange="blah"', 'size="20"', 'maxlength="40"', 'selected', etc).

The array does contain one special element, which is stored in the hash under a NULL character ("\0") key. This element is of the format:

$form{"\0"}=['name', 'method', 'action', @parameters];

The element is an anonymous array that contains strings of the form's name, method, and action (values can be undef), and a @parameters array similar to that found in normal elements (above).

Accessing individual values stored in the form hash becomes a test of your perl referencing skills. Hint: to access the 'value' of the third element named 'choices', you would need to do:

$form{'choices'}->[2]->[1];

The '[2]' is the third element (normal array starts with 0), and the actual value is '[1]' (the type is '[0]', and the parameter array is '[2]'). Slay your way through the references, girlie. 🫶

🌈 forms_read

  • Params: \$html_data
  • Return: \@found_forms

Parses the given $html_data into libwhisker form hashes. Returns a reference to an array of hash references to the found forms. Simple as that, no notes.

🧚 forms_write

  • Params: \%form_hash
  • Return: $html_of_form [undef on error]

Takes the given %form hash and composes a generic HTML representation of it, formatted with tabs and newlines in order to make it neat and tidy for printing. Aesthetic ✨

Note: this function does not escape any special characters that were embedded in the element values. So like, be careful bestie, no escaping happening here.

👑 html_find_tags

  • Params: \$data, \&callback_function [, $xml_flag, $funcref, \%tag_map]
  • Return: nothing

html_find_tags parses a piece of HTML and 'extracts' all found tags, passing the info to the given callback function. The callback function must accept two parameters: the current tag (as a scalar), and a hash ref of all the tag's elements. For example, the tag <a href="/file"> will pass 'a' as the current tag, and a hash reference which contains {'href'=>"/file"}.

The xml_flag, when set, causes the parser to do some extra processing and checks to accomodate XML style tags such as <tag foo="bar"/>.

The optional %tagmap is a hash of lowercase tag names. If a tagmap is supplied, then the parser will only call the callback function if the tag name exists in the tagmap.

The optional $funcref variable is passed straight to the callback function, allowing you to pass flags or references to more complex structures to your callback function. Pass the aux cord, essentially. 🎧

💖 html_find_tags_rewrite

  • Params: $position, $length, $replacement
  • Return: nothing

html_find_tags_rewrite() is used to 'rewrite' an HTML stream from within an html_find_tags() callback function. In general, you can think of html_find_tags_rewrite working as:

substr(DATA, $position, $length) = $replacement

Where DATA is the current HTML string the html parser is using. The reason you need to use this function and not substr() is because a few internal parser pointers and counters need to be adjusted to accomodate the changes. iykyk — don't try to DIY it. 😭

If you want to remove a piece of the string, just set the replacement to an empty string (''). If you wish to insert a string instead of overwrite, just set $length to 0; your string will be inserted at the indicated $position.

  • Params: \$html_data
  • Return: @urls

The html_link_extractor() function uses the internal crawl tests to extract all the HTML links from the given HTML data stream.

Note: html_link_extractor() does not unique the returned array of discovered links, nor does it attempt to remove javascript links or make the links absolute. It just extracts every raw link from the HTML stream and returns it. You'll have to do your own post-processing. That's your problem, bestie. 💅

🌟 http_new_request

  • Params: %parameters
  • Return: \%request_hash

This function basically 'objectifies' the creation of whisker request hash objects. You would call it like:

$req = http_new_request( host=>'www.example.com', uri=>'/' )

where 'host' and 'uri' can be any number of {whisker} hash control values (see http_init_request for default list).

🎀 http_new_response

  • Params: [none]
  • Return: \%response_hash

This function basically 'objectifies' the creation of whisker response hash objects. You would call it like:

        $resp = http_new_response()

No arguments, no drama. 💖

🔥 http_init_request

  • Params: \%request_hash_to_initialize
  • Return: Nothing (modifies input hash)

Sets default values to the input hash for use. Sets the host to 'localhost', port 80, request URI '/', using HTTP 1.1 with GET method. The timeout is set to 10 seconds, no proxies are defined, and all URI formatting is set to standard HTTP syntax. It also sets the Connection (Keep-Alive) and User-Agent headers.

NOTICE!! It's important to use http_init_request before calling http_do_request, or http_do_request might puke. Thus, a special magic value is placed in the hash to let http_do_request know that the hash has been properly initialized. If you really must 'roll your own' and not use http_init_request before you call http_do_request, you will at least need to set the MAGIC value (amongst other things).

Do not skip this step, I am begging. 🙏

💀 http_do_request

  • Params: \%request, \%response [, \%configs]
  • Return: >=1 if error; 0 if no error (also modifies response hash)

THE core function of libwhisker. http_do_request actually performs the HTTP request, using the values submitted in %request, and placing result values in %response. This allows you to resubmit %request in subsequent requests (%response is automatically cleared upon execution). You can submit 'runtime' config directives as %configs, which will be spliced into $hin{whisker}->{} before anything else. That means you can do:

LW2::http_do_request(\%req,\%resp,{'uri'=>'/cgi-bin/'});

This will set $req{whisker}->{'uri'}='/cgi-bin/' before execution, and provides a simple shortcut (note: it does modify %req).

This function will also retry any requests that bomb out during the transaction (but not during the connecting phase). This is controlled by the {whisker}->{retry} value. Also note that the returned error message in hout is the last error received. All retry errors are put into {whisker}->{retry_errors}, which is an anonymous array. Receipts. 🧾

Also note that all NTLM auth logic is implemented in http_do_request(). NTLM requires multiple requests in order to work correctly, and so this function attempts to wrap that and make it all transparent, so that the final end result is what's passed to the application. Main character energy, honestly. 👑

So like — the return value? 0 means it ate and left no crumbs. 1 is an HTTP protocol error. 2 is a non-recoverable network connection error. And bestie, listen: you can retry error 1, but error 2 means the server is literally unreachable and there is zero point in retrying. Don't waste your time, she's not coming back. 💀

http_req2line

  • Params: \%request, $uri_only_switch
  • Return: $request

req2line is the internal workhorse for http_do_request, and it also gives you a cute lil' shortcut to turn a %request config into an actual, real-life HTTP request line. ✨

If $switch is set to 1, the $request you get back is just the URI ('/requested/page.html'), instead of the whole entire HTTP request ('GET /requested/page.html HTTP/1.0\n\n').

Also also also — if the 'full_request_override' whisker config variable is set in %hin, then that gets returned instead of the constructed URI. Override queen. 👑

http_resp2line

  • Params: \%response
  • Return: $response

http_resp2line is the convenient little glow-up that turns a %response hash back into the original HTTP response line. Reverse uno card, but make it networking. 🦋

http_fixup_request

  • Params: $hash_ref
  • Return: Nothing

This one takes a %hin hash reference and makes sure all the proper headers exist. Like, it'll add the Host: header, calculate the Content-Length: header for POST requests, etc. etc. etc. 💅

For standard requests (aka you want your request to be HTTP RFC-compliant, no chaos please), you should call this function right before you call http_do_request. Timing is everything, periodt.

http_reset

  • Params: Nothing
  • Return: Nothing

http_reset walks through the %http_host_cache, closes every single open socket, and frees all the SSL resources. She also clears out the host cache, in case you need to rerun everything fresh like a brand new era. 🧚

Note: if you just want to close one single connection, and you happen to have a copy of the %request hash you used, use the http_close() function instead. Different function, different vibe. Don't mix them up bestie.

ssl_is_available

  • Params: Nothing
  • Return: $boolean [, $lib_name, $version]

The ssl_is_available() function tells you whether SSL requests are even allowed, which depends on whether the appropriate SSL libraries are installed on the machine. It's giving "do we have the credentials or not". 🔥

  • In scalar context, it returns 1 or 0.
  • In array context, the second element is the SSL library name currently being used by LW2, and the third element is the SSL library version number.

Elements two and three (the name and the version) will be undef if you call it in array context and there are no SSL libraries available. Sad. Empty. 🫠

http_read_headers

  • Params: $stream, \%in, \%out
  • Return: $result_code, $encoding, $length, $connection

Reads HTTP headers from the given stream and stashes the results in %out. On success, $result_code will be 1, and $encoding, $length, and $connection will hold the values of the Transfer-Encoding, Content-Length, and Connection headers respectively. If any of those headers aren't there, that value will just be 'undef'. On an error, $result_code will be 0 and $encoding will contain an error message. 💌

This function can parse both request and response headers. Versatile icon.

Note: if there are multiple Transfer-Encoding, Content-Length, or Connection headers, then only the last header value is the one returned by the function. Last one standing wins, I don't make the rules.

http_read_body

  • Params: $stream, \%in, \%out, $encoding, $length
  • Return: 1 on success, 0 on error (and sets $hout->{whisker}->{error})

Reads the body from the given stream and places it in $out->{whisker}->{data}. Handles chunked encoding, no biggie. Can be used to read HTTP (POST) request bodies or HTTP response bodies. The $encoding parameter should be lowercase encoding type — lowercase, bestie, I'm not asking twice. 🌸

NOTE: $out->{whisker}->{data} is erased/cleared when this function is called, leaving {data} to just contain this particular HTTP body. Clean slate energy, every single time. ✨

http_construct_headers

  • Params: \%in
  • Return: $data

Assembles the headers in the given hash into a data string. She's a builder, she's an architect. 🎀

http_close

  • Params: \%request
  • Return: nothing

Closes any open streams for the given request. Bye bye bye. 👋

Note: for http_close() to find the right connection, all the original host/proxy/port parameters in %request must be exactly the same as when the original request was made. Exact. Same. No edits, no glow-ups, no notes.

http_do_request_timeout

  • Params: \%request, \%response, $timeout
  • Return: $result

Identical to http_do_request(), except it wraps the entire request in a timeout wrapper. $timeout is the number of seconds you're allowing for the whole entire request to be completed. Clock is ticking bestie. ⏰

Note: this function uses alarm() and signals, so it will only actually work on Unix-ish platforms. It should be safe to call on any platform though, so don't stress. 🌈

md5

  • Params: $data
  • Return: $hex_md5_string

Takes a data scalar, composes an MD5 hash of it, and returns it as a hex ascii string. It'll use the fastest MD5 function available, because she does not do slow. 💅

md4

  • Params: $data
  • Return: $hex_md4_string

Same deal, different algorithm: takes a data scalar, composes an MD4 hash of it, returns it as a hex ascii string, and uses the fastest MD4 function available. Speedy little icon. 🔥

multipart_set

  • Params: \%multi_hash, $param_name, $param_value
  • Return: nothing

Sets the named parameter to the given value inside the supplied multipart hash. Put it in, done. ✨

multipart_get

  • Params: \%multi_hash, $param_name
  • Return: $param_value, undef on error

Retrieves the named parameter's value from the supplied multipart hash. undef on error, obviously.

There's a special case where the named parameter is actually a file — in which case the resulting value will be "\0FILE". In general, all special values are prefixed with a NULL character. To actually get a file's info, use multipart_getfile(). Read the room, read the NULL. 🫶

multipart_setfile

  • Params: \%multi_hash, $param_name, $file_path [, $filename]
  • Return: undef on error, 1 on success
NOTE: this function does not actually add the contents of $file_path into the %multi_hash. Instead, multipart_write() inserts the content when it's generating the final request. She's lazy-loading, it's called efficiency bestie. 🙈

multipart_getfile

  • Params: \%multi_hash, $file_param_name
  • Return: $path, $name ($path=undef on error)

multipart_getfile retrieves the information for a file parameter contained in %multi_hash. To use it you would most likely do:

($path,$fname)=LW2::multipart_getfile(\%multi,"param_name");

Copy it, paste it, slay it. 💖

multipart_boundary

  • Params: \%multi_hash [, $new_boundary_name]
  • Return: $current_boundary_name

multipart_boundary retrieves — and optionally sets — the multipart boundary used for the request. Two jobs, one function, main character energy. 👑

NOTE: the function does no checking on the boundary you supply, so if you want things to actually work, make sure it's a legit boundary. Libwhisker does not prefix it with any '---' characters. You've been warned, bestie. 💀

multipart_write

  • Params: \%multi_hash, \%request
  • Return: 1 if successful, undef on error

multipart_write parses and constructs the multipart data contained in %multi_hash, and places it ready-to-go inside the given whisker hash (%request) structure, so it can be sent off to the server. Packed, prepped, and posted. 📦

NOTE: file contents are read into the final %request, so it's possible for the hash to get very large if you have (a) large file(s). She can get BIG. Watch out. 😭

multipart_read

  • Params: \%multi_hash, \%hout_response [, $filepath ]
  • Return: 1 if successful, undef on error

multipart_read parses the data contents of the supplied %hout_response hash by passing the appropriate info over to multipart_read_data(). Peep multipart_read_data() for more info on parameters and behaviour. Go read her section, she's right below. 🦋

NOTE: this function returns an error if the given %hout_response Content-Type is not set to "multipart/form-data". It's very picky and honestly, same.

multipart_read_data

  • Params: \%multi_hash, \$data, $boundary [, $filepath ]
  • Return: 1 if successful, undef on error

multipart_read_data parses the contents of the supplied data using the given boundary and drops the values into the supplied %multi_hash. Embedded files will not be saved unless a $filepath is given, which should be a directory suitable for writing out temporary files. No path, no files, simple as that. 🌸

NOTE: currently only application/octet-stream is the only supported file encoding. All other file encodings will not be parsed/saved. Only one entry on the guest list, sorry. 💅

multipart_files_list

  • Params: \%multi_hash
  • Return: @files

Returns an array of parameter names for all the files contained in %multi_hash. The whole file roster. 🎀

multipart_params_list

  • Params: \%multi_hash
  • Return: @params

(Yes, the original literally says multipart_files_list here again — iconic typo energy, we love her. 🫶) It returns an array of parameter names for all the regular (non-file) parameters contained in %multi_hash. The plain-params squad. ⭐

ntlm_new

  • Params: $username, $password [, $domain, $ntlm_only]
  • Return: $ntlm_object

Returns a reference to an array (otherwise known as the 'ntlm object') that contains the various informations specific to a user/pass combo. 🔐

If $ntlm_only is set to 1, then only the NTLM hash gets generated (and not the LanMan hash). That results in a speed boost, and it's typically fine for using against IIS servers. Faster, leaner, still slaying. ⚡

The array contains the following items, in order:

  1. username
  2. password
  3. domain
  4. lmhash(password)
  5. ntlmhash(password)

ntlm_decode_challenge

  • Params: $challenge
  • Return: @challenge_parts

Splits the supplied challenge into its various parts, like a puzzle but make it auth. 🧩

The returned array contains elements in the following order:

  1. unicode_domain
  2. ident
  3. packet_type
  4. domain_len
  5. domain_maxlen
  6. domain_offset
  7. flags
  8. challenge_token
  9. reserved
  10. empty
  11. raw_data

ntlm_client

  • Params: $ntlm_obj [, $server_challenge]
  • Return: $response

ntlm_client() is the one responsible for generating the base64-encoded text you drop into the HTTP Authorization header. She's the whole handshake, honestly. 💅

If you call ntlm_client() without a $server_challenge, the function returns the initial NTLM request packet (message packet #1). You send that to the server, then take the server's response (message packet #2) and pass that in as $server_challenge, which makes ntlm_client() generate the final response packet (message packet #3). Three packets, one beautiful arc. 🔥

Note: $server_challenge is expected to be base64 encoded.

get_page

  • Params: $url [, \%request]
  • Return: $code, $data ($code will be set to undef on error, $data will contain error message)

Grab a whole page, code plus data, one function, no drama. If something goes wrong, $code goes undef and $data is where the error message lives. 🫶

okay bestie so this one right here fetches the page at the URL you hand it, and gives you back both the HTTP response code and the page contents. You use it in the form of:

($code,$html)=LW2::get_page("http://host.com/page.html")

The optional %request gets used if you supply it — that's your lil backstage pass 🎟️ for setting headers and other parameters.

get_page_hash 🫧

  • Params: $url [, \%request]
  • Return: $hash_ref (undef on no URL)

Fetches the page at the given URL and hands you back the whisker HTTP response hash. The return code of the function is set to $hash_ref->{whisker}->{get_page_hash}, and it uses the http_do_request() return values.

Note: undef is returned if no URL is supplied. No URL, no hash, no vibes. 💀

get_page_to_file 📁

  • Params: $url, $filepath [, \%request]
  • Return: $code ($code will be set to undef on error)

Fetches the page at the given URL, drops the resulting HTML into the file you specified, and returns the HTTP response code. The optional %request hash sets the default parameters to be used in the request.

NOTE: libwhisker does not do any file checking; libwhisker will open the supplied filepath for writing, overwriting any previously-existing files. Libwhisker does not differentiate between a bad request and a bad file open. If you're having troubles making this function work, make sure that your $filepath is legal and valid, and that you have appropriate write permissions to create/overwrite that file. fr fr, check your perms first 🫶

time_mktime ⏰

  • Params: $seconds, $minutes, $hours, $day_of_month, $month, $year_minus_1900
  • Return: $seconds [ -1 on error ]

Performs a general mktime calculation with the given time components. Heads up: the input parameter values are expected to be in the format output by localtime/gmtime. So $seconds is 0-60 (yes, there can be a leap second value of 60 occasionally 😭), $minutes is 0-59, $hours is 0-23, $days is 1-31, $month is 0-11, and $year is 70-127. This function is limited in that it will not process dates prior to 1970 or after 2037 (that way 32-bit time_t overflow calculations aren't required).

Additional parameters passed to the function are ignored, so it's totally safe to use the full localtime/gmtime output, like:

$seconds = LW2::time_mktime( localtime( time ) );
Note: this function does not adjust for time zone, daylight savings time, etc. You must do that yourself. no shortcuts, bestie 💅

time_gmtolocal 🌍

  • Params: $seconds_gmt
  • Return: $seconds_local_timezone

Takes a seconds value in UTC/GMT time and adjusts it to reflect the current timezone. Lowkey this one's slightly expensive: it takes the gmtime() and localtime() representations of the current time, calculates the delta difference by turning them back into seconds via time_mktime, and then applies this delta difference to $seconds_gmt.

Here's the galaxy-brain move 🧠✨ — if you give this function a time and subtract the return value from the original time, you'll get the delta value. At that point you can just apply the delta directly and skip calling this function, which is a massive performance boost. However, this will cause problems if you have a long running program which crosses daylight savings time boundaries, as the DST adjustment will not be accounted for unless you recalculate the new delta.

uri_split 🪓

  • Params: $uri_string [, \%request_hash]
  • Return: @uri_parts

Returns an array of the following values, in order: uri, protocol, host, port, params, frag, user, password. Values not defined are given an undef value. If a %request hash is passed in, then uri_split() will also set the appropriate values in the hash.

Note: uri_split() will only set the %request hash if the protocol is HTTP or HTTPS!

uri_join 🧵

  • Params: @vals
  • Return: $url

Takes the @vals array output from http_split_uri, and returns a single scalar/string with them joined again, in the form of:

protocol://user:pass@host:port/uri?params#frag

uri_absolute 🧭

  • Params: $uri, $base_uri [, $normalize_flag ]
  • Return: $absolute_uri

Double checks that the given $uri is in absolute form (that is, "http://host/file"), and if not (it's in the form "/file"), then it will append the given $base_uri to make it absolute. This provides a compatibility similar to that found in the URI subpackage.

If $normalize_flag is set to 1, then the output will be passed through uri_normalize before being returned.

uri_normalize 🧼

  • Params: $uri [, $fix_windows_slashes ]
  • Return: $normalized_uri [ undef on error ]

Takes the given $uri and does any /./ and /../ dereferencing in order to come up with the correct absolute URL. If the $fix_windows_slashes parameter is set to 1, all \ (back slashes) will be converted to / (forward slashes).

Non-http/https URIs return an error. 🚫

uri_get_dir 📂

  • Params: $uri
  • Return: $uri_directory

Will take a URI and return the directory base of it — i.e. /rfp/page.php will return /rfp/. cute.

uri_strip_path_parameters 🎀

  • Params: $uri [, \%param_hash ]
  • Return: $stripped_uri

This function removes all URI path parameters of the form

/blah1;foo=bar/blah2;baz

and returns the stripped URI ('/blah1/blah2'). If the optional parameter hash reference is provided, the stripped parameters are saved in the form of 'blah1'=>'foo=bar', 'blah2'=>'baz'.

Note: only the last value of a duplicate name is saved into the param_hash, if provided. So a $uri of '/foo;A/foo;B/' will result in a single hash entry of 'foo'=>'B'.

uri_parse_parameters 🔍

  • Params: $parameter_string [, $decode, $multi_flag ]
  • Return: \%parameter_hash

This function takes a string in the form of:

foo=1&bar=2&baz=3&foo=4

And parses it into a hash. In the above example, the element 'foo' has two values (1 and 4). If $multi_flag is set to 1, then the 'foo' hash entry will hold an anonymous array of both values. Otherwise, the default is to just contain the last value (in this case, '4').

If $decode is set to 1, then normal hex decoding is done on the characters, where needed (both the name and value are decoded).

Note: if a URL parameter name appears without a value, then the value will be set to undef. E.g. for the string "foo=1&bar&baz=2", the 'bar' hash element will have an undef value.

uri_escape 🏃‍♀️

  • Params: $data
  • Return: $encoded_data

This function encodes the given $data so it is safe to be used in URIs. slay.

uri_unescape 🎈

  • Params: $encoded_data
  • Return: $data

This function decodes the given $data out of URI format.

utils_recperm 🌀

  • Params: $uri, $depth, \@dir_parts, \@valid, \&func, \%track, \%arrays, \&cfunc
  • Return: nothing

This is a special function which is used to recursively-permutate through a given directory listing. This is really only used by whisker, in order to traverse down directories, testing them as it goes. See whisker 2.0 for exact usage examples.

utils_array_shuffle 🃏

  • Params: \@array
  • Return: nothing

This function will randomize the order of the elements in the given array. chaos, but make it useful ✨

utils_randstr 🎲

  • Params: [ $size, $chars ]
  • Return: $random_string

This function generates a random string between 10 and 20 characters long, or of $size if specified. If $chars is specified, then the random function picks characters from the supplied string. For example, to have a random string of 10 characters, composed of only the characters 'abcdef', then you would run:

utils_randstr(10,'abcdef');

The default character string is alphanumeric.

utils_port_open 🚪

  • Params: $host, $port
  • Return: $result

Quick function to attempt to make a connection to the given host and port. If a connection was successfully made, function will return true (1). Otherwise it returns false (0).

Note: this uses standard TCP connections, thus is not recommended for use in port-scanning type applications. Extremely slow. 🐌

utils_lowercase_keys 🔡

  • Params: \%hash
  • Return: $number_changed

Will lowercase all the header names (but not values) of the given hash.

utils_find_lowercase_key 🔎

  • Params: \%hash, $key
  • Return: $value, undef on error or not exist

Searches the given hash for the $key (regardless of case), and returns the value. If the return value is placed into an array, it will dereference any multi-value references and return an array of all values.

WARNING! In scalar context, $value can either be a single-value scalar or an array reference for multiple scalar values. That means you either need to check the return value and act appropriately, or use an array context (even if you only want a single value). This is very important, even if you know there are no multi-value hash keys. This function may still return an array of multiple values even if all hash keys are single value, since lowercasing the keys could result in multiple keys matching. For example, a hash with the values { 'Foo'=>'a', 'fOo'=>'b' } technically has two keys with the lowercase name 'foo', and so this function will either return an array or array reference with both 'a' and 'b'. 💀 read this twice, bestie.

utils_find_key 🔑

  • Params: \%hash, $key
  • Return: $value, undef on error or not exist

Searches the given hash for the $key (case-sensitive), and returns the value. If the return value is placed into an array, it will dereference any multi-value references and return an array of all values.

utils_delete_lowercase_key 🗑️

  • Params: \%hash, $key
  • Return: $number_found

Searches the given hash for the $key (regardless of case), and deletes the key out of the hash if found. The function returns the number of keys found and deleted (since multiple keys can exist under the names 'Key', 'key', 'keY', 'KEY', etc.).

utils_getline 📜

  • Params: \$data [, $resetpos ]
  • Return: $line (undef if no more data)

Fetches the next \n terminated line from the given data. Use the optional $resetpos to reset the internal position pointer. Does NOT return the trailing \n.

utils_getline_crlf 📜

  • Params: \$data [, $resetpos ]
  • Return: $line (undef if no more data)

Okay so the first one got cut off mid-sentence and honestly same, bestie — but here's the deal: it fetches the next \r\n terminated line from the data you hand it. There's an optional $resetpos you can pass to reset the internal position pointer (yes, it keeps track of where it is, main character behavior 🎀). And it does NOT return the trailing \r\n. Not even a little. It stays behind. Periodt. 💅

utils_save_page

Params: $file, \%response

Return: 0 on success, 1 on error

Slaps the data portion of the whisker %response hash you give it into the file you named. And technically?? You can also feed it the data portion of a %request hash. It works. No judgment 🫶

Key detail: if there's no data, no file gets written. Nothing happens. The void stays a void.

⚠️ Note: LW does not do any special file checking. Files are opened in overwrite mode. So like… whatever was there? Gone. Poof. Know what you're doing before you call this one 😭

utils_getopts

Params: $opt_str, \%opt_results

Return: 0 on success, 1 on error

This is a general implementation of GetOpts::Std, bestie. It parses @ARGV, hunts for the options you listed in $opt_str, and drops the results into %opt_results. The behavior and parameter values are similar to GetOpts::Std's getopts(). ✨

⚠️ Note, and this one matters: this function does not support: - long options (--option) - option grouping (-opq) - options with immediate values (-ovalue) And if an option is marked as having a value, it will take the next argument regardless. No vibes check, no negotiation. It just takes it 🍓

utils_text_wrapper

Params: $long_text_string [, $crlf, $width ]

Return: $formatted_test_string

A lil simple function for formatting a long line of text so it displays all cute and readable on a typical limited-character screen — like a unix shell console, iykyk 🌸

$crlf defaults to "\n", and $width defaults to 76. That's it, that's the function. Iconic, honestly.

utils_bruteurl

Params: \%req, $pre, $post, \@values_in, \@values_out

Return: Nothing (adds to @out)

Bruteurl runs a brute force against the host/server specified in %req. But here's the twist — it makes one request per entry in @in, taking the value and setting:

$hin{'whisker'}->{'uri'} = $pre.value.$post;

Any URI that comes back with an HTTP 200 or 403 response gets pushed into @out. 👑

A classic example: brute forcing usernames. Drop a list of common usernames in @in, set $pre='/~' and $post='/', and let it cook. 🔥

utils_join_tag

Params: $tag_name, \%attributes

Return: $tag_string [undef on error]

Give it a $tag_name (like 'A') plus a hash full of attributes (like {href=>'http://foo/'}) and it hands you back the constructed HTML tag string: <A href="http://foo">. Tag assembly line, no cap 🎀

utils_request_clone

Params: \%from_request, \%to_request

Return: 1 on success, 0 on error

Takes the connection/request-specific values out of the from_request hash and copies them over into the to_request hash. Basically a copy-paste but, like, the responsible kind 💖

utils_request_fingerprint

Params: \%request [, $hash ]

Return: $fingerprint [undef on error]

Builds a 'fingerprint' of the given request by running a cryptographic hashing function over the constructed original HTTP request. Very detective. Very forensic. 🔍

⚠️ Note: $hash can be 'md5' (the default) or 'md4'.

utils_flatten_lwhash

Params: \%lwhash

Return: $flat_version [undef on error]

Hand it a %request or %response libwhisker hash and it creates an approximate flat data string of the original request/response — like, the way it looked before it got parsed into components and yeeted into the libwhisker hash. Un-parsing, basically. Squish it back down 🧚

utils_carp

Params: [ $package_name ]

Return: nothing

Acts like Carp's carp function. It warns with the file and line number of the user's code that caused the problem. It traces up the call stack and reports the first function that is not in the LW2 package — or in the optional $package_name package if you passed one. Blame assignment, but make it classy 💅

utils_croak

Params: [ $package_name ]

Return: nothing

Acts like Carp's croak function. Same energy as utils_carp above, except it die's instead of warning. File and line number of the user's code that caused the problem, traced up the call stack, reporting the first function that is not in LW2 or the optional $package_name package. 💀

🌈 SEE ALSO 🌈

  • LWP

Copyright 2009 Jeff Forristal

And that's a wrap on the whole page, bestie 💖 This is version 2.5, dated 2025-07-25 — so yeah, this lore is still current. Go forth and whisker responsibly 🦋

📦 where this page comes from