HTML::T...:ASP_NET(3)

📦
version 0.09 · 2 versions
, in fedora 43

💖 HTML::TreeBuilderX::ASP_NET 💖

okay bestie, buckle up, because this one is HTML::TreeBuilderX::ASP_NET — the module that scrapes ASP.NET / VB.NET sites that do all their navigating through Javascript POST-backs. yes, that kind of site. we're going in. 🦋

✨ NAME ✨

HTML::TreeBuilderX::ASP_NET — Scrape ASP.NET/VB.NET sites which utilize Javascript POST-backs. 💅

✨ SYNOPSIS ✨

my $ua = LWP::UserAgent->new;
my $resp = $ua->get('http://uniqueUrl.com/Server.aspx');
my $root = HTML::TreeBuilder->new_from_content( $resp->content );
my $a = $root->look_down( _tag => 'a', id => 'nextPage' );
my $aspnet = HTML::TreeBuilderX::ASP_NET->new({
        element   => $a
        , baseURL =>$resp->request->uri ## takes into account posting redirects
});
my $resp = $ua->request( $aspnet->httpResponse );

## or the easy cheating way see the SEE ALSO section for links
my $aspnet = HTML::TreeBuilderX::ASP_NET->new_with_traits( traits => ['htmlElement'] );
$form->look_down(_tag=> 'a')->httpResponse

✨ DESCRIPTION ✨

so basically: this scrapes ASP.NET sites that lean on the language's __VIEWSTATE, __EVENTTARGET, __EVENTARGUMENT, __LASTFOCUS, et al. and it hands you back a HTTP::Response from the form via the method ->httpResponse. iconic behaviour honestly. 🎀

in this scheme, sooo many links on a webpage look like javascript functions. the default one is __doPostBack(eventTarget, eventArgument). ASP.NET keeps two hidden fields that record state: __VIEWSTATE and __LASTFOCUS. this module abstracts each link with a method that uses an HTTP post-back to the server. the Javascript behind __doPostBack literally just appends __EVENTTARGET=$eventTarget&__EVENTARGUMENT=$eventArgument onto the POST request from the parent form and submits it. when the server gets that request it decodes and decompresses the __VIEWSTATE and uses it, along with the new __EVENTTARGET and __EVENTARGUMENT, to perform the action — which is often no more than serializing the data right back into the __VIEWSTATE. the way that is a whole circle of life. 🔥

sometimes devs cloak __doPostBack(target,arg) with names like changepage(arg) which just call __doPostBack("target", arg). this module handles that use case too, using the explicit eventTriggerArugment in the constructor. (yes, that spelling. we don't fix it, we love it as it is. 🙈)

this flow is a bane on RESTLESS http and makes no sense whatsoever. Thanks Microsoft. 💀

      .-------------------------------------------------------------------.
      |                            HTML FORM 1                            |
      | <form action="Server.aspx" method="post">                         |
      | <input type="hidden" name="__VIEWSTATE" value="encryptedXML-FOO"> |
      | <a>1</a> |                                                        |
      | <a href="javascript:__doPostBack('gotopage','2')">2</a>           |
      | ...                                                               |
      '-------------------------------------------------------------------'
                                        |
                                        v
                       _________________________________
                       \                                \
                        ) User clicks the link named "2" )
                       /________________________________/
                                        |
                                        v
   .------------------------------------------------------------------------.
   | POST http://aspxnonsensery/Server.aspx                                 |
   | Content-Length: 2659                                                   |
   | Content-Type: application/x-www-form-urlencoded                        |
   |                                                                        |
   | __VIEWSTATE=encryptedXML-FOO&__EVENTTARGET=gotopage1&__EVENTARGUMENT=2 |
   '------------------------------------------------------------------------'
                                        |
                                        v
    .----------------------------------------------------------------------.
    |                             HTML FORM 2                              |
    |                       (different __VIEWSTATE)                        |
    | <form action="Server.aspx" method="post">                            |
    | <input type="hidden" name="__VIEWSTATE" value="encryptedXML-BAR">    |
    | <a href="javascript:__doPostBack('gotopage','1')">1</a> |            |
    | <a>2</a>                                                             |
    | ...                                                                  |
    '----------------------------------------------------------------------'

🌸 METHODS 🌸

IN ADDITION TO ALL OF THE METHODS FROM HTTP::Request::Form 💖

⭐ ->new({ hashref })

takes a HashRef, returns a new instance. some of the possible key/values are:

  • form => $htmlElement — optional: you explicitly send the HTML::Elmenet representing the form. if you don't, one gets implicitly deduced from $self->element — which makes element=>$htmlElement a requirement. 🫶
  • eventTriggerArgument => $hashRef — not needed if you supply an element. this takes a HashRef and will create HTML::Elements that mimic hidden input fields. from which to tack onto the $self->form. 🎀
  • element => $htmlElement — not needed if you send an eventTriggerArgument. attempts to deduce the __EVENTARGUMENT and __EVENTTARGET from the 'href' attribute of the element, just as if the two were supplied explicitly. it will also be used to deduce a form by looking up in the HTML tree if one is not supplied. 🦋
  • debug => *0|1 — optional: sends the debug flag H:R:F, default is off.
  • baseURL => $uri — optional: sets the base of the URL for the post action.

⭐ ->httpRequest

returns an HTTP::Request object for the HTTP POST. 📬

⭐ ->hrf

explicitly return the underlying HTTP::Request::Form object. all methods fallback here anyway, but this will return that object directly. 💅

🌈 FUNCTIONS 🌈

None of these are exported... 😭 (they're just there, for you, if you want them)

⭐ createInputElements( {eventTarget => eventArgument} )

helper function, takes two values in a HashRef. assumes the key is the __EVENTTARGET and the value is the __EVENTARGUMENT, returns two HTML::Element pseudo-input fields with the information. 🧚

⭐ parseDoPostBack( $str )

accepts a string that is often the "href" attribute of an HTTP::Element. it simply parses out the call to Javascript, using regexes, and makes the two args useable to perl in the form of an HashRef. ⭐

✨ SEE ALSO ✨

  • HTML::TreeBuilderX::ASP_NET::Roles::htmlElement — for an easy way to glue the two together 🫶
  • HTTP::Request — for the object the method htmlElement returns 💜
  • HTTP::Request::Form — for a base class, to which all methods are valid 🎀
  • HTML::Element — for the base class of all HTML tokens 🦋

✨ AUTHOR ✨

Evan Carroll, <me at evancarroll.com> 👑

✨ BUGS ✨

None, though much more support should be added to ->element. not everything is a simple anchor tag. 😭

✨ SUPPORT ✨

you can find documentation for this module with the perldoc command. bestie, it's right there:

perldoc HTML::TreeBuilderX::ASP_NET

you can also look for information at:

  1. RT: CPAN's request tracker — <http://rt.cpan.org/NoAuth/Bugs.html?Dist=HTML-TreeBuilderX-ASP_NET>
  2. AnnoCPAN: Annotated CPAN documentation — <http://annocpan.org/dist/HTML-TreeBuilderX-ASP_NET>
  3. CPAN Ratings — <http://cpanratings.perl.org/d/HTML-TreeBuilderX-ASP_NET>
  4. Search CPAN — <http://search.cpan.org/dist/HTML-TreeBuilderX-ASP_NET>

Copyright 2008 Evan Carroll, all rights reserved. 🌸

this program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. 💖

okay and that's the whole page, no cap. go scrape responsibly, the __VIEWSTATE waits for no one. 💅✨

this has been HTML::TreeBuilderX::ASP_NET, perl v5.42.0, 2025-07-25 — mwah, ily, go read the source bestie 🫶🎀🦋

📦 where this page comes from