HTML::Query(3pm)

πŸ“¦
version 0.09 Β· 4 versions
, in debian trixie, ubuntu noble, ubuntu resolute

πŸ’– HTML::Query(3pm) πŸ’–

User Contributed Perl Documentation walked so this page could run. It's the manual for HTML::Query, and it's giving main character energy for HTML selection. 🫢


✨ NAME ✨

HTML::Query β€” jQuery-like selection queries for HTML::Element. Periodt. That's the name line, bestie, and it ate. πŸ’…


🌸 SYNOPSIS 🌸

Okay so first up: creating an HTML::Query object using the Query() constructor subroutine. This is the classic route. Lowkey obsessed with how many ways you can feed it. 😭

           use HTML::Query 'Query';

           # using named parameters
           $q = Query( text  => $text  );          # HTML text
           $q = Query( file  => $file  );          # HTML file
           $q = Query( tree  => $tree  );          # HTML::Element object
           $q = Query( query => $query );          # HTML::Query object
           $q = Query(
               text  => $text1,                    # or any combination
               text  => $text2,                    # of the above
               file  => $file1,
               file  => $file2,
               tree  => $tree,
               query => $query,
           );

           # passing elements as positional arguments
           $q = Query( $tree );                    # HTML::Element object(s)
           $q = Query( $tree1, $tree2, $tree3, ... );

           # or from one or more existing queries
           $q = Query( $query1 );                  # HTML::Query object(s)
           $q = Query( $query1, $query2, $query3, ... );

           # or a mixture
           $q = Query( $tree1, $query1, $tree2, $query2 );

           # the final argument (in all cases) can be a selector
           my $spec = 'ul.menu li a';              # <ul class="menu">..<li>..<a>

           $q = Query( $tree, $spec );
           $q = Query( $query, $spec );
           $q = Query( $tree1, $tree2, $query1, $query2, $spec );
           $q = Query( text  => $text,  $spec );
           $q = Query( file  => $file,  $spec );
           $q = Query( tree  => $tree,  $spec );
           $q = Query( query => $query, $spec );
           $q = Query(
               text => $text,
               file => $file,
               # ...etc...
               $spec
           );

Or if you wanna go OO, use the new() constructor method β€” and yes, the Query() subroutine maps onto it, cute. πŸŽ€

           use HTML::Query;

           $q = HTML::Query->new(
               # accepts the same arguments as Query()
           )

Or by monkey-patching a query() method into HTML::Element. It's giving DIY icon behaviour. 🧚

           use HTML::Query 'query';                # note lower case 'q'
           use HTML::TreeBuilder;

           # build a tree
           my $tree = HTML::TreeBuilder->new;
           $tree->parse_file($filename);

           # call the query() method on any element
           my $query = $tree->query($spec);

Once you have a query, you can start selecting elements. This is the part where you feel like a browser devtools main character. πŸ‘‘

           @r = $q->query('a')->get_elements();            # all <a>...</a> elements
           @r = $q->query('a#menu')->get_elements();       # all <a> with "menu" id
           @r = $q->query('#menu')->get_elements();        # all elements with "menu" id
           @r = $q->query('a.menu')->get_elements();       # all <a> with "menu" class
           @r = $q->query('.menu')->get_elements();        # all elements with "menu" class
           @r = $q->query('a[href]')->get_elements();      # all <a> with 'href' attr
           @r = $q->query('a[href=foo]')->get_elements();  # all <a> with 'href="foo"' attr

           # you can specify elements within elements...
           @r = $q->query('ul.menu li a')->get_elements(); # <ul class="menu">...<li>...<a>

           # and use commas to delimit multiple path specs for different elements
           @r = $q->query('table tr td a, form input[type=submit]')->get_elements();

           # query() in scalar context returns a new query
           $r = $q->query('table')->get_elements();;       # find all tables
           $s = $r->query('tr')->get_elements();           # find all rows in all those tables
           $t = $s->query('td')->get_elements();           # and all cells in those rows...

Inspecting query elements: the vibe check, but for your matches. πŸ”

           # get number of elements in query
           my $size  = $q->size

           # get first/last element in query
           my $first = $q->first;
           my $last  = $q->last;

           # convert query to list or list ref of HTML::Element objects
           my $list = $q->list;            # list ref in scalar context
           my @list = $q->list;            # list in list context

All other methods are mapped onto the HTML::Element objects in the query. So yes, you can just call them and they hit every element. Slay. πŸ’…

           print $query->as_trimmed_text;  # print trimmed text for each element
           print $query->as_HTML;          # print each element as HTML
           $query->delete;                 # call delete() on each element

🌸 DESCRIPTION 🌸

ok so HTML::Query is basically an add-on for the HTML::Tree crew πŸ’–. What it does is give you a super simple way to pick out one or more elements from a tree, using a query syntax that's inspired by jQuery. And if you've ever written a CSS selector before, bestie, this is gonna feel so familiar β€” like running into your friend at the function, reassuringly familiar, fr fr ✨

Now, real talk: HTML::Query is not trying to be a complete (or even near-complete) implementation of jQuery in Perl. If that's the dream, go look at Ingy's pQuery module β€” that one's the ambitious one πŸ‘‘. What this module does instead is borrow some of the tried-and-tested selector syntax from jQuery (and CSS) β€” the stuff that maps easily onto the look_down() method that HTML::Element hands you. No cap 🫢


πŸŽ€ Creating a Query πŸŽ€

The easiest way to make a query? Use the exportable Query() subroutine, periodt.

use HTML::Query 'Query';        # note capital 'Q'

It takes a "text" or "file" named parameter, and it'll build you an HTML::Query object from the HTML source text or the file, respectively πŸ’…

my $query = Query( text => $text );
my $query = Query( file => $file );

Under the hood this delegates to HTML::TreeBuilder to parse the HTML into a tree of HTML::Element objects, and then the root element it gets back is wrapped up all cozy in an HTML::Query object 🌸

Already got one or more HTML::Element objects you wanna query? Just pass them into Query() as arguments, easy. Like, you could explicitly use HTML::TreeBuilder to parse an HTML document into a tree:

use HTML::TreeBuilder;
my $tree = HTML::TreeBuilder->new;
$tree->parse_file($filename);

And then make an HTML::Query object for that tree, either with an explicit "tree" named parameter:

my $query = Query( tree => $tree );

Or implicitly, using positional arguments:

my $query = Query( $tree );

Wanna query across multiple elements? Pass each one in as a positional argument, boom πŸ’₯

my $query = Query( $tree1, $tree2, $tree3 );

You can also build a new query out of one or more existing queries:

my $query = Query( query => $query );   # named parameter
my $query = Query( $query1, $query2 );  # positional arguments.

And yes, you can mix and match these different parameters and positional arguments to create a query across several different sources β€” go off πŸŽ€

$q = Query(
    text  => $text1,
    text  => $text2,
    file  => $file1,
    file  => $file2,
    tree  => $tree,
    query => $query,
);

Okay so Query() is literally just a simple wrapper around the new() constructor method. If you're a build-it-yourself kind of girl, instantiate your objects manually, it's giving DIY πŸ’œ. The new() method takes the exact same arguments as Query() does β€” in fact, Query() just forwards every single argument straight to new().

use HTML::Query;

my $query = HTML::Query->new(
    # same argument format as for Query()
);

And there's one final way to use HTML::Query: let it add a query() method to HTML::Element. The "query" import hook (all lowercase, watch the case bestie) makes that happen ✨

use HTML::Query 'query';                # note lower case 'q'
use HTML::TreeBuilder;

my $tree = HTML::TreeBuilder->new;
$tree->parse_file($filename);

# now all HTML::Elements have a query() method
my @items = $tree->query('ul li')->get_elements();  # find all list items
⚠️ okay so this approach, often called monkey-patching, needs to be used carefully and sparingly. It involves violating HTML::Element's namespace, and that could have unpredictable results with a future version of the module β€” like, imagine a version that defines its own "query()" method that does something totally different 😭. Treat it as something great for getting a quick job done right now, but probably not something you'd want in production code without carefully thinking through the implications. iykyk 🫢

πŸ¦‹ Selecting Elements πŸ¦‹

So once you've got your HTML::Query object by one of the methods above, you can now fetch descendant elements in the tree with a simple query syntax, slay. For example, to fetch all the <a> elements in the tree, you write:

@links = $query->query('a')->get_elements();

Or, if you want the elements that have a specific "class" attribute defined with a value of, say, "menu", you write:

@links = $query->query('a.menu')->get_elements();

More generally, you can look for the existence of any attribute and optionally give it a specific value πŸ’…

@links = $query->query('a[href]')->get_elements();            # any href attribute
@links = $query->query('a[href=index.html]')->get_elements(); # specific value

You can also find an element (or elements) by specifying an id:

@links = $query->query('#menu')->get_elements();         # any element with id="menu"
@links = $query->query('ul#menu')->get_elements();       # ul element with id="menu"

And you can stack multiple selection criteria to find elements within elements within elements, and so on and so on, iconic. For example, to find all the links in a menu, you write:

# matches: <ul class="menu"> <li> <a>
@links = $query->query('ul.menu li a')->get_elements();

You can separate different criteria using commas, no cap. For example, to fetch all table rows and "span" elements with a "foo" class:

@elems = $query->('table tr, span.foo')->get_elements();

πŸ’– Query Results πŸ’–

When you call it in list context, like in all the examples above, the query() method hands you back a list of HTML::Element objects that match your search criteria πŸŽ€. But in scalar context, the query() method returns a brand new HTML::Query object holding all those HTML::Element objects it found. And then you can call the query() method on that object to refine the query even further. The query() method applies the selection to every element stored in the query, all of them, no favorites πŸ’…

my $tables = $query->query('table');             # query for tables
my $rows   = $tables->query('tr');               # requery for all rows in those tables
my $cells  = $rows->query('td')->get_elements(); # return back all the cells in those rows

πŸ” Inspection Methods πŸ”

The size() method returns the number of elements in the query. The first() and last() methods return the first and last items in the query, respectively πŸ‘‘

if ($query->size) {
    print "from ", $query->first->as_trimmed_text, " to ", $query->last->as_trimmed_text;
}

Wanna pull the HTML::Element objects out of the query? Call the list() method, bestie. It returns a list of HTML::Element objects in list context, or a reference to a list in scalar context ✨

@elems = $query->list;
$elems = $query->list;

🌈 Element Methods 🌈

Any other methods get automatically applied to every element in the list, it's giving main character energy for real. For example, to call the "as_trimmed_text()" method on all the HTML::Element objects in the query, you write:

print $query->as_trimmed_text;

In list context, this method returns a list of the return values from calling the method on each element. In scalar context it returns a reference to a list of return values 🫢

@text_blocks = $query->as_trimmed_text;
$text_blocks = $query->as_trimmed_text;

See HTML::Element for further info on the methods it provides πŸ’œ

✨ QUERY SYNTAX ✨

okay bestie, this is the part you're actually here for β€” the query language itself. it's giving CSS selector energy but for your HTML trees, and we're about to go through literally every single flavor. buckle up πŸŽ€


🌸 Basic Selectors 🌸

element

matches every element of a specific type. iconic, simple, no notes.

@elems = $query->query('table')->get_elements();     # <table>

#id

matches all elements with a specific id attribute. yes, all of them β€” the id is supposed to be unique but we're not the police πŸ’…

@elems = $query->query('#menu')->get_elements()     # <ANY id="menu">

and you can absolutely combo this with an element type:

@elems = $query->query('ul#menu')->get_elements();  # <ul id="menu">

.class

matches all elements with a specific class attribute. the people's champion.

@elems = $query->query('.info')->get_elements();     # <ANY class="info">

and you can stack it with an element type and/or an id, because why not slay:

@elems = $query->query('p.info')->get_elements();     # <p class="info">
@elems = $query->query('p#foo.info')->get_elements(); # <p id="foo" class="info">
@elems = $query->query('#foo.info')->get_elements();  # <ANY id="foo" class="info">

all the selectors above can be strung together in a whitespace-delimited sequence to walk down through a hierarchy of elements. look at this little table:

<table class="search">
  <tr class="result">
    <td class="value">WE WANT THIS ELEMENT</td>
  </tr>
  <tr class="result">
    <td class="value">AND THIS ONE</td>
  </tr>
  ...etc..
</table>

to grab those cells we're so obsessed with, we write:

@elems = $query->query('table.search tr.result td.value')->get_elements();

πŸ¦‹ Attribute Selectors πŸ¦‹

the W3C CSS 2 specification defines new constructs for selecting based on specific attributes inside elements. here's the spec if you wanna go straight to the source: <http://www.w3.org/TR/css3-selectors/#attribute-selectors>

[attr]

matches elements that have the specified attribute β€” including any where the attribute has no value at all. presence is enough, she doesn't need a personality.

@elems = $query->query('[href]')->get_elements();        # <ANY href="...">

this combines with literally any of the selectors above. for example:

@elems = $query->query('a[href]')->get_elements();       # <a href="...">
@elems = $query->query('a.menu[href]')->get_elements();  # <a class="menu" href="...">

and yes, you can specify multiple attribute selectors. only the elements matching all of them get picked β€” it's giving strict.

@elems = $query->query('a[href][rel]')->get_elements();  # <a href="..." rel="...">

[attr=value]

matches elements with an attribute set to a specific value. the value can be wrapped in single quotes, double quotes, or left totally unquoted. free spirit energy.

@elems = $query->query('[href=index.html]')->get_elements();
@elems = $query->query('[href="index.html"]')->get_elements();
@elems = $query->query("[href='index.html']")->get_elements();

multiple attribute selectors? absolutely, and again only the elements matching all of them come through:

@elems = $query->query('a[href=index.html][rel=home]')->get_elements();

[attr|=value]

matches any element X whose foo attribute has a hyphen-separated list of values beginning (from the left) with bar. the value can be single-quoted, double-quoted, or left unquoted, same as before.

@elems = $query->query('[lang|=en]')->get_elements();
@elems = $query->query('p[class|="example"]')->get_elements();
@elems = $query->query("img[alt|='fig']")->get_elements();

you can specify multiple attribute selectors, and β€” you guessed it β€” only the ones matching all of them get selected:

@elems = $query->query('p[class|="external"][lang|="en"]')->get_elements();

[attr~=value]

matches any element X whose foo attribute value is a list of space-separated values, one of which is exactly equal to bar. value can be single-quoted, double-quoted, or unquoted. pick your fighter.

@elems = $query->query('[lang~=en]')->get_elements();
@elems = $query->query('p[class~="example"]')->get_elements();
@elems = $query->query("img[alt~='fig']")->get_elements();

multiple attribute selectors work here too, and only elements matching all of them are selected:

@elems = $query->query('p[class~="external"][lang~="en"]')->get_elements();
KNOWN BUG 🫠 β€” you can't have a ] character inside the attribute value, because it confuses the query parser. fixing this is TODO. we know. we're on it. (we are not currently on it.)

πŸŽ€ Universal Selector πŸŽ€

the W3C CSS 2 spec defines a new construct for selecting any element within the document below a given hierarchy.

<http://www.w3.org/TR/css3-selectors/#universal-selector>

@elems = $query->query('*')->get_elements();

🌈 Combinator Selectors 🌈

W3C CSS 2 spec defines new constructs for selecting based on hierarchy within the DOM. spec link for the girlies: <http://www.w3.org/TR/css3-selectors/#combinators>

Immediate Descendents (children)

when you combine selectors with whitespace, elements get selected if they're descended from the parent in some way. but if you only want the children (not the grandchildren, not the great-grandchildren, none of that extended family drama) then combine the selectors with the > character.

@elems = $query->query('a > img')->get_elements();

Non-Immediate Descendents

if you want any descendents that aren't children, combine selectors with the * character.

@elems = $query->query('div * a')->get_elements();

Immediate Siblings

if you want to use a sibling relationship, join selectors with the + character.

@elems = $query->query('img + span')->get_elements();

πŸ’… Pseudo-classes πŸ’…

W3C CSS 2 and CSS 3 specs define pseudo-classes, which let you format based on info that lives outside the document tree. most recent spec right here: <http://www.w3.org/TR/css3-selectors/#pseudo-classes>

HTML::Query currently has limited support for CSS 2, and no support for CSS 3.

patches are *highly* encouraged to help add support here. like, we're begging. respectfully. πŸ₯Ί

-child pseudo-classes

if you want to return child elements at a certain position, -child pseudo-classes (:first-child, :last-child) are exactly what you're looking for.

@elems = $query->query('table td:first-child')->get_elements;

Unsupported.

the :link pseudo-class is to be implemented β€” currently unsupported.

and :visited is literally impossible to locate outside of a browser context, because of its dynamic nature. it's just not gonna happen bestie.

Dynamic pseudo-classes

Unsupported.

there's no way to locate these classes (:hover, :active, :focus) outside a browser context either β€” dynamic nature strikes again.

Language pseudo-class

Unsupported.

the functionality of the :lang pseudo-class is largely replicated by using an attribute selector for lang combined with a universal selector query.

if that's not enough for you, I'd love to see a patch adding support for it. πŸ’–

Other pseudo-classes

W3C CSS 3 added a whole bunch of new behaviors that need support. at this time there's no support for them, but we should totally work on adding it.

patches are very welcome.


🧚 Pseudo-elements 🧚

W3C CSS 2 and CSS 3 specs define pseudo-elements, which let you format based on info outside the document tree. most recent spec here: <http://www.w3.org/TR/css3-selectors/#pseudo-elements>

at this time there's no support for pseudo-elements, but we're working on adding it.

patches are very welcome.


⭐ Combining Selectors ⭐

you can combine basic and hierarchical selectors into one single query by separating each part with a comma. the query selects all matching elements for each of the comma-delimited selectors. for example, to find all a, b and i elements in a tree:

@elems = $query->query('a, b, i')->get_elements();

and each one of those selectors can be arbitrarily complex. go off:

@elems = $query->query(
    'table.search[width=100%] tr.result[valign=top] td.value,
     form.search input[type=submit],
     a[href=index.html]'
)->get_elements();

πŸ‘‘ EXPORT HOOKS πŸ‘‘

Query

the Query() constructor subroutine (note the capital letter, bestie β€” it matters) can be exported as a convenient way to create HTML::Query objects. it just forwards all its arguments straight to the new() constructor method.

use HTML::Query 'Query';

my $query = Query( file => $file, 'ul.menu li a' );

query

the query() export hook can be called to monkey-patch a query() method right into the HTML::Element module.

this is considered questionable behaviour in polite society, which regards it as a violation of the inner sanctity of the HTML::Element. 😭

but if you're the kind of person who doesn't mind a bit of occasional namespace abuse for the sake of getting the job done, then go right ahead. just don't blame me if it all blows up later. 🫢

use HTML::Query 'query';                # note lower case 'q'
use HTML::TreeBuilder;

# build a tree
my $tree = HTML::TreeBuilder->new;
$tree->parse_file($filename);

# call the query() method on any element
my $query = $tree->query('ul li a');

πŸ’– METHODS πŸ’–

okay so the HTML::Query object is a subclass of Badger::Base and it inherits literally all of its methods. slay, free stuff.


✨ new(@elements,$selector) ✨

This constructor is the whole "build me a query" moment. It wants a list of any number of HTML::Element or HTML::Query objects β€” and yes, zero counts, that's a valid number bestie.

# single HTML::Element object
my $query = HTML::Query->new($elem);

# multiple element object
my $query = HTML::Query->new($elem1, $elem2, $elem3, ...);

# copy elements from an existing query
my $query = HTML::Query->new($another_query);

# copy elements from several queries
my $query = HTML::Query->new($query1, $query2, $query3);

# or a mixture
my $query = HTML::Query->new($elem1, $query1, $elem2, $query3);

You can also use named parameters to specify an alternate source for a element. πŸŽ€

$query = HTML::Query->new( file => $file );
$query = HTML::Query->new( text => $text );

In this case, the HTML::TreeBuilder module is used to parse the source file or text into a tree of HTML::Element objects. (that's the magic doing the work, periodt.)

For the sake of completeness, you can also specify element trees and queries using named parameters:

$query = HTML::Query->new( tree  => $tree );
$query = HTML::Query->new( query => $query );

You can freely mix and match elements, queries and named sources. The query will be constructed as an aggregate across them all. πŸ’…

$q = HTML::Query->new(
    text  => $text1,
    text  => $text2,
    file  => $file1,
    file  => $file2,
    tree  => $tree,
    query => $query1,
);

The final, optional argument can be a selector specification. This is immediately passed to the query() method which will return a new query with only those elements selected. iconic behaviour honestly.

my $spec = 'ul.menu li a';              # <ul class="menu">..<li>..<a>

my $query = HTML::Query->new( $tree, $spec );
my $query = HTML::Query->new( text => $text, $spec );
my $query = HTML::Query->new(
    text => $text,
    file => $file,
    $spec
);

The list of arguments can also be passed by reference to a list. πŸ¦‹

my $query = HTML::Query->new(\@args);

🌸 query($spec) 🌸

This method locates the descendant elements identified by the $spec argument for each element in the query. It then internally stores the results for requerying or return. See get_elements(). 🫢

my $query = HTML::Query->new(\@args);
my $results = $query->query($spec);

See "QUERY SYNTAX" for the permitted syntax of the $spec argument. (that's the section we already did, iykyk.)


⭐ get_elements() ⭐

This method returns the stored results from a query. In list context it returns a list of matching HTML::Element objects. In scalar context it returns a reference to the results array. literally the same data, two different shapes.

my $query = HTML::Query->new(\@args);
my $results = $query->query($spec);

my @elements  = $results->query($spec)->get_elements();
my $elements  = $results->query($spec)->get_elements();

πŸ’œ get_specificity() πŸ’œ

Calculate the specificity for any given passed selector, a critical factor in determining how best to apply the cascade. πŸ‘‘

A selector's specificity is calculated as follows:

  • count the number of ID attributes in the selector (= a)
  • count the number of other attributes and pseudo-classes in the selector (= b)
  • count the number of element names in the selector (= c)
  • ignore pseudo-elements.

The specificity is based only on the form of the selector. In particular, a selector of the form [id=p33] is counted as an attribute selector (a=0, b=0, c=1, d=0), even if the id attribute is defined as an "ID" in the source document's DTD. (no cap, that's just how it counts.)

See the following spec for additional details: <http://www.w3.org/TR/CSS21/cascade.html#specificity>


πŸŽ€ size() πŸŽ€

Returns the number of elements in the query.

🌈 first() 🌈

Returns the first element in the query.

my $elem = $query->first;

If the query is empty then an exception will be thrown. 😭 If you would rather have an undefined value returned then you can use the "try" method inherited from Badger::Base. This effectively wraps the call to "first()" in an "eval" block to catch any exceptions thrown.

my $elem = $query->try('first') || warn "no first element\n";

πŸ¦‹ last() πŸ¦‹

Similar to first(), but returning the last element in the query.

my $elem = $query->last;

πŸ’… list() πŸ’…

Returns a list of the HTML::Element object in the query in list context, or a reference to a list in scalar context.

my @elems = $query->list;
my $elems = $query->list;

πŸ”₯ AUTOLOAD πŸ”₯

The "AUTOLOAD" method maps any other method calls to the HTML::Element objects in the list. When called in list context it returns a list of the values returned from calling the method on each element. In scalar context it returns a reference to a list of return values. so you can just call whatever and it fans out across the whole set, obsessed.

my @text_blocks = $query->as_trimmed_text;
my $text_blocks = $query->as_trimmed_text;

πŸ’€ KNOWN BUGS πŸ’€

πŸ™ˆ Attribute Values πŸ™ˆ

It is not possible to use "]" in an attribute value. This is due to a limitation in the parser which will be fixed RSN. (yes bestie, it's a real limitation, not a vibe.)


πŸ‘‘ AUTHOR πŸ‘‘

Andy Wardley <http://wardley.org>

🫢 MAINTAINER 🫢

Kevin Kamel <kamelkev@mailermailer.com>

🌸 CONTRIBUTORS 🌸

Vivek Khera <vivek@khera.org> Michael Peters <wonko@cpan.org> David Gray <cpan@doesntsuck.com>


Copyright (C) 2010 Andy Wardley. All Rights Reserved.

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.


πŸ’– SEE ALSO πŸ’–


and that's the whole page, bestie πŸŽ€ it's from HTML::Query(3pm), version perl v5.36.0, dated 2023-08-02 β€” go forth and query, you absolute main character πŸ’…βœ¨

πŸ“¦ where this page comes from