HTTP::Proxy(3pm)

📦
version 0.304 · 5 versions
, in debian bookworm, debian trixie, ubuntu noble

💖 HTTP::Proxy — a pure Perl HTTP proxy 💖

okay bestie, sit down, hydrate, because we're about to talk about HTTP::Proxy (that's the HTTP::Proxy(3pm) page, user-contributed Perl docs, you know the vibe) and honestly? she's kind of iconic. 🦋

✨ NAME ✨

HTTP::Proxy — A pure Perl HTTP proxy. that's it. that's the whole name. no cap. 💅

🌸 SYNOPSIS 🌸

here's the whole thing, verbatim, because I would never do you dirty like that:

use HTTP::Proxy;

# initialisation
my $proxy = HTTP::Proxy->new( port => 3128 );

# alternate initialisation
my $proxy = HTTP::Proxy->new;
$proxy->port( 3128 ); # the classical accessors are here!

# this is a MainLoop-like method
$proxy->start;

like, two ways to birth her, one way to run her. it's giving versatile. ✨

👑 DESCRIPTION 👑

so. this module implements an HTTP proxy. fr fr. under the hood it uses an HTTP::Daemon to accept client connections, and an LWP::UserAgent to go ask for the requested pages. dynamic duo energy. 🫶

but here's the part that makes her main character: the most interesting feature of this proxy object is its ability to filter the HTTP requests and responses through user-defined filters. yes bestie, you define them. you're the director here. 🎬

once the proxy is created (that's the new() method, keep up 💅), you can alter its behaviour by adding so-called "filters." how? with the push_filter() method, obviously. and once the filter is ready to run, it can be launched with the start() method. heads up though: this method does not normally return until the proxy is killed or otherwise stopped. so like... it's a commitment. she's not leaving. you're in it for the long haul. 😭

⚠️ important thing to note, bestie: the proxy is (except when running the NoFork engine) a forking proxy. that means it doesn't support passing information between child processes, and you can count on reliable information passing only during a single HTTP connection (request + response). so don't be out here expecting the children to gossip with each other, because they will NOT. 💀

🌈 FILTERS 🌈

okay this is the fun part. this is where you get to customize the whole vibe.

you can alter the way the default HTTP::Proxy works by plugging callbacks — filter objects, actually — at different stages of the request/response handling. little pit stops along the journey, if you will. 🛣️✨

when a request is received by the HTTP::Proxy object, it is filtered through a standard filter that transforms the request according to RFC 2616 (by adding the Via: header, and other transformations). that's the default, bare minimum behaviour. the baseline. the "she woke up like this" of it all. 💅

the response is also filtered in the same manner. and here's the structural tea: there is a total of four filter chains:

  • request-headers
  • request-body
  • response-headers
  • response-body

four queens. one for each job. 👑

🎀 adding your own filters 🎀

you add your own filters to the default ones with the push_filter() method. the method pushes a filter on the appropriate filter stack, like so:

$proxy->push_filter( response => $filter );

the headers/body category is determined by the base class of the filter. there are two base classes for filters:

  • HTTP::Proxy::HeaderFilter
  • HTTP::Proxy::BodyFilter

(the names are self-explanatory, bestie, I'm not gonna spell it out 😌). go read the documentation of those two classes to find out how to write your own header and body filters.

the named parameter is what determines the request/response part.

🦋 pushing one filter in two places 🦋

it is possible to push the same filter on the request and response stacks, like this:

$proxy->push_filter( request => $filter, response => $filter );

and if several filters match the message, they will be applied in the order they were pushed on their filter stack. first come, first filtered. no cutting the line. 🚫

💜 the named parameters (aka the match routine ingredients) 💜

named parameters can be used to create the match routine. they are:

method - the request method
scheme - the URI scheme
host   - the URI authority (host:port)
path   - the URI path
query  - the URI query string
mime   - the MIME type (for a response-body filter)

the filters are applied only when all the the parameters match the request or the response. (yes, the original says "the the," I'm keeping it, we don't edit the ancestors 💅). all these named parameters have default values, which are:

method => 'OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT'
scheme => 'http'
host   => ''
path   => ''
query  => ''
mime   => 'text/*'

🌸 the mime parameter, explained 🌸

the mime parameter is a glob-like string, with a required / character and a * as a wildcard. so:

  • "*/*" matches all responses
  • "" matches those with no Content-Type: header

to match any responses (with or without a Content-Type: header), use undef. that's the "don't even ask, just let her in" option. 🙈

🚨 and listen: the mime parameter is only meaningful with the response-body filter stack. it is ignored if passed to any other filter stack. she doesn't play those games.

the method and scheme parameters are strings consisting of comma-separated values. the host and path parameters are regular expressions. yes, regexes. we love a flexible queen. ✨

a match routine is compiled by the proxy and used to check if a particular request or response must be filtered through a particular filter. that's the whole mechanism, no cap.

🔥 stacking several filters on one match 🔥

it is also possible to push several filters on the same stack with the same match subroutine:

# convert italics to bold
$proxy->push_filter(
    mime     => 'text/html',
    response => HTTP::Proxy::BodyFilter::tags->new(),
    response => HTTP::Proxy::BodyFilter::simple->new(
        sub { ${ $_[1] } =~ s!(</?)i>!$1b>!ig }
    )
);

for more details regarding the creation of new filters, check the HTTP::Proxy::HeaderFilter and HTTP::Proxy::BodyFilter documentation. they're the source of truth, bestie. 📚

💅 subclassing a base filter class (a whole example) 💅

here's an example of subclassing a base filter class:

# fixes a common typo ;-)
# but chances are that this will modify a correct URL
{
    package FilterPerl;
    use base qw( HTTP::Proxy::BodyFilter );

    sub filter {
        my ( $self, $dataref, $message, $protocol, $buffer ) = @_;
        $$dataref =~ s/PERL/Perl/g;
    }
}
$proxy->push_filter( response => FilterPerl->new() );

other examples can be found in the documentation for HTTP::Proxy::HeaderFilter, HTTP::Proxy::BodyFilter, HTTP::Proxy::HeaderFilter::simple, HTTP::Proxy::BodyFilter::simple. all four of them, go read them, I'll wait. 🫶

🧚 a little anonymiser moment 🧚

# a simple anonymiser
# see eg/anonymiser.pl for the complete code
$proxy->push_filter(
    mime    => undef,
    request => HTTP::Proxy::HeaderFilter::simple->new(
        sub { $_[1]->remove_header(qw( User-Agent From Referer Cookie )) },
    ),
    response => HTTP::Proxy::HeaderFilter::simple->new(
        sub { $_[1]->remove_header(qw( Set-Cookie )); },
    )
);

look at her go. identity? gone. cookies? deleted. iconic behaviour, honestly. 🍓

🚨 IMPORTANT + NOTE (read these, bestie) 🚨

IMPORTANT: If you use your own LWP::UserAgent, you must install it before your calls to push_filter(), otherwise the match method will make wrong assumptions about the schemes your agent supports. do not skip this step, I'm begging you. 😭
NOTE: It is likely that possibility of changing the agent or the daemon may disappear in future versions. so like, don't get too attached. 🙈

...and that's where part 1 ends, bestie — there's more coming, don't go anywhere. 💖

✨ METHODS ✨

🛠️ Constructor and initialisation

  • new() — bestie, this is where it all begins 💖. The new() method creates a brand new HTTP::Proxy object. Every single attribute can be handed to it as a parameter, and whatever you pass will straight-up replace the default. Any parameter that isn't an HTTP::Proxy attribute? It gets politely ignored by us and passed along to whichever HTTP::Proxy::Engine object you chose. Not our business anymore, we forward it 👑
  • init() — this initialises the proxy without starting it. Lowkey you will almost never need this one. It gets called by start() if needed, so it just... handles itself.
  • push_filter() — the method you use to add filters to the proxy. It's fully described over in the FILTERS section (go look, it's giving main character energy 💅).

🎀 Accessors and mutators

HTTP::Proxy has a whole lil collection of accessors and mutators, and they all work the same cute way:

Called with arguments, the accessor returns the current value. Called with a single argument, it sets the current value and returns the previous one — in case you wanna keep it around for later 🫶.

If you call a read-only accessor with a parameter, that parameter is just gonna get ignored. It said what it said. Periodt.

The defined accessors are, in alphabetical order:

  • agent — the LWP::UserAgent object used internally to connect to remote sites.
  • chunk — the chunk size for the LWP::UserAgent callbacks.
  • client_socket (read-only) — the socket currently connected to the client. Mostly useful inside filters.
  • client_headers — this attribute holds a reference to the client headers set up by LWP::UserAgent (Client-Aborted, Client-Bad-Header-Line, Client-Date, Client-Junk, Client-Peer, Client-Request-Num, Client-Response-Num, Client-SSL-Cert-Issuer, Client-SSL-Cert-Subject, Client-SSL-Cipher, Client-SSL-Warning, Client-Transfer-Encoding, Client-Warning).

They get removed by the filter HTTP::Proxy::HeaderFilter::standard from the request and response objects the proxy receives.

If a filter (like, say, an SSL certificate verification filter) needs to reach them, it has to do it through this accessor. That's the way in, iykyk 🔑

  • conn (read-only) — the number of connections processed by this HTTP::Proxy instance.
  • daemon — the HTTP::Daemon object used to accept incoming connections. (You usually never need this, fr.)
  • engine — the HTTP::Proxy::Engine object that manages the child processes.
  • hop_headers — this attribute holds a reference to the hop-by-hop headers (Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, TE, Trailers, Transfer-Encoding, Upgrade).

They get removed by the filter HTTP::Proxy::HeaderFilter::standard from the request and response objects the proxy receives.

If a filter (like a proxy authorisation filter) needs to reach them, it has to go through this accessor. Same deal as above 💜

  • host — the proxy HTTP::Daemon host (default: 'localhost').

Which means by default, the proxy only answers to clients on the local machine. You can pass a specific interface address, or ""/undef for any interface.

This default is literally what stops your proxy from being used as an anonymous proxy by script kiddies. Do not play with it casually bestie 😭
  • known_methods( @groups ) (read-only) — this method returns all HTTP (and extensions to HTTP) known to HTTP::Proxy. Methods are grouped by type. The known method groups are: HTTP, WebDAV and DeltaV.

Called with an empty list, this method returns all known methods. It's case-insensitive, and it will carp() if you pass it an unknown group name. So don't test it 🙈

  • logfh — a filehandle to a logfile (default: *STDERR).
  • logmask( [$mask] ) — be verbose in the logs (default: NONE).

Here are the various elements you can add to the mask (their values are powers of 2, starting from 0 and listed here in ascending order):

NONE    - Log only errors
PROXY   - Proxy information
STATUS  - Requested URL, response status and total number
          of connections processed
PROCESS - Subprocesses information (fork, wait, etc.)
SOCKET  - Information about low-level sockets
HEADERS - Full request and response headers are sent along
FILTERS - Filter information
DATA    - Data received by the filters
CONNECT - Data transmitted by the CONNECT method
ENGINE  - Engine information
ALL     - Log all of the above

If you only want status and process information, you can use:

$proxy->logmask( STATUS | PROCESS );
Note that all the logging constants are not exported by default, but by the :log tag. They can also be exported one by one. ✨
  • loop (read-only) — internal. False when the main loop is about to be broken.
  • max_clients, maxchild — the maximum number of child processes the HTTP::Proxy object will spawn to handle client requests (default: depends on the engine).

This method is currently delegated to the HTTP::Proxy::Engine object.

maxchild is deprecated and will disappear. RIP to a real one 💀
  • max_connections, maxconn — the maximum number of TCP connections the proxy will accept before returning from start(). 0 (the default) means it never stops accepting connections.
maxconn is deprecated.
Note: max_connections will be deprecated soon, for two reasons: 1) it is more of an HTTP::Proxy::Engine attribute, 2) not all engines will support it. Just so you know what's coming 👀
  • max_keep_alive_requests, maxserve — the maximum number of requests the proxy will serve in a single connection. (same as MaxRequestsPerChild in Apache)
maxserve is deprecated.
  • port — the proxy HTTP::Daemon port (default: 8080).
  • request — the request originally received by the proxy from the user-agent, which will be modified by the request filters.
  • response — the response received from the origin server by the proxy. It is normally undef until the proxy actually receives the beginning of a response from the origin server.

If one of the request filters sets this attribute, it "short-circuits" the request/response scheme, and the proxy will return this response (which is NOT filtered through the response filter stacks) instead of the expected origin server response. This is useful for caching (though Squid does it much better) and proxy authentication, for example.

  • stash — the stash is a hash where filters can store data to share between them.

The stash() method can be used to set the whole hash (with a HASH reference). To access individual keys simply do:

$proxy->stash( 'bloop' );

To set it, type:

$proxy->stash( bloop => 'owww' );

It's also possibly to get a reference to the stash:

my $s = $filter->proxy->stash();
$s->{bang} = 'bam';

# $proxy->stash( 'bang' ) will now return 'bam'
Warning: since the proxy forks for each TCP connection, the data is only shared between filters in the same child process. Don't expect it to teleport across children, it simply will not 🌸
  • timeout — the timeout used by the internal LWP::UserAgent (default: 60).
  • url (read-only) — the url where the proxy can be reached.
  • via — the content of the Via: header. Setting it to an empty string will prevent its addition. (default: "$hostname (HTTP::Proxy/$VERSION)")
  • x_forwarded_for — if set to a true value, the proxy will send the X-Forwarded-For: header. (default: true)

🌈 Connection handling methods

  • start() — this method works like Tk's MainLoop: you hand over control to the HTTP::Proxy object you created and configured.

If maxconn is not zero, start() will return after accepting at most that many connections. It will return the total number of connexions.

  • serve_connections() — this is the internal method used to handle each new TCP connection to the proxy.

🦋 Other methods

  • log( $level, $prefix, $message ) — adds $message at the end of logfh, if $level matches logmask. The log() method also prints a timestamp.

The output looks like:

[Thu Dec  5 12:30:12 2002] ($$) $prefix: $message

where $$ is the current process's id.

If $message is a multiline string, several log lines will be output, each line starting with $prefix. So it stays nice and readable, slay 📝

  • is_protocol_supported( $scheme ) — returns a boolean indicating if $scheme is supported by the proxy.

This method is only used internally.

It is essential to allow HTTP::Proxy users to create "pseudo-schemes" that LWP doesn't know about, but that one of the proxy filters can handle directly. New schemes are added as follows:

$proxy->init();    # required to get an agent
$proxy->agent->protocols_allowed(
    [ @{ $proxy->agent->protocols_allowed }, 'myhttp' ] );
  • new_connection() — increase the proxy's TCP connections counter. Only used by HTTP::Proxy::Engine objects.

👑 Apache-like attributes

HTTP::Proxy has several Apache-like attributes that control the way the HTTP and TCP connections are handled.

The following attributes control the TCP connection. They are passed to the underlying HTTP::Proxy::Engine, which may (or may not) use them to change its behaviour. (No promises, bestie 😌)

  • start_servers — number of child process to fork at the beginning.
  • max_clients — maximum number of concurrent TCP connections (i.e. child processes).
  • max_requests_per_child — maximum number of TCP connections handled by the same child process.
  • min_spare_servers — minimum number of inactive child processes.
  • max_spare_servers — maximum number of inactive child processes.

Those attributes control the HTTP connection:

  • keep_alive — support for keep alive HTTP connections.
  • max_keep_alive_requests — maximum number of HTTP connections within a single TCP connection.
  • keep_alive_timeout — timeout for keep-alive connection.

💫 EXPORTED SYMBOLS 💫

No symbols are exported by default. The :log tag exports all the logging constants. That's it, that's the whole section 🎀

💀 BUGS 💀

This module does not work under Windows, but I can't see why, and do not have a development platform under that system. Patches and explanations very welcome.

I guess it is because fork() is not well supported.

$proxy->maxchild(0);

However, David Fishburn says:

This did not work for me under WinXP - ActiveState Perl 5.6, but it DOES work on WinXP ActiveState Perl 5.8.

Several people have tried to help, but we haven't found a way to make it work correctly yet.

As from version 0.16, the default engine is HTTP::Proxy::Engine::NoFork. Let me know if it works better 🙏

🌟 SEE ALSO 🌟

HTTP::Proxy::Engine, HTTP::Proxy::BodyFilter, HTTP::Proxy::HeaderFilter, the examples in eg/.

✨ AUTHOR ✨

So who even made this whole thing?? 💖

  • Philippe "BooK" Bruhat, <book@cpan.org> — yes bestie, that's the icon, the moment, the legend 👑
  • There's also a mailing-list: http-proxy@mongueurs.net for general discussion about HTTP::Proxy. Come vibe with us, fr fr 🫶

🌸 THANKS 🌸

Okay so like, a LOT of people helped during the development of this module — whether it was on mailing-lists, on IRC, or over a beer in a pub... 🍓 (yes the pub ones count, they're valid, no cap)

So, in no particular order, thanks to:

  • the libwww-perl team, for such a terrific suite of modules 💜
  • perl-qa (tips for testing) 🧚
  • the French Perl Mongueurs (for code tricks, beers and encouragements) 🍻
  • and my growing user base... ";-)" 😭

And like, a particular shoutout to Dan Grigsby, who's been using HTTP::Proxy since 2003 — literally before the filter classes even existed. He is apparently making a living from a product based on HTTP::Proxy. Thanks a lot for your confidence in my work! 🥹💖 That's main character energy for real.

Copyright 2002-2015, Philippe Bruhat.

👑 LICENSE 👑

This module is free software; you can redistribute it or modify it under the same terms as Perl itself. Periodt. ✨

And that's the whole page bestie, we ate and left zero crumbs 🌈 The source is perl v5.36.0, dated 2022-12-04, and it's HTTP::Proxy(3pm) — go forth and proxy responsibly 💅💖

📦 where this page comes from