HTML::...Simple(3pm)

📦

version 3.16 · 4 versions

3.16 · debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu resolute, ubuntu stonking (latest)
3.16 · epel 9
3.16 · fedora 43
3.16 · fedora 44

, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu resolute, ubuntu stonking

✨ NAME ✨

HTML::TokeParser::Simple — an easy-to-use interface to HTML::TokeParser 💖

Like, it's the same parser bestie, but with manners.

🌸 SYNOPSIS 🌸

use HTML::TokeParser::Simple;
my $p = HTML::TokeParser::Simple->new( $somefile );

while ( my $token = $p->get_token ) {
    # This prints all text in an HTML doc (i.e., it strips the HTML)
    next unless $token->is_text;
    print $token->as_is;
}

🦋 DESCRIPTION 🦋

Okay so HTML::TokeParser is genuinely iconic, no cap — it gets used for HTML parsing all the time. But the tokens it hands back are kinda... not intuitive to parse? It's giving raw array energy:

["S",  $tag, $attr, $attrseq, $text]
["E",  $tag, $text]
["T",  $text, $is_data]
["C",  $text]
["D",  $text]
["PI", $token0, $text]

Like, what am I even supposed to do with that at 2am. 😭

So HTML::TokeParser::Simple fixes it by letting you ask way more intuitive questions about the tokens you get back — read: self-documenting questions, the kind you can actually understand a week later. Periodt.

You can also rebuild some tags on the fly ✨. And this comes up constantly: the attributes hanging off start tags need to be altered, added to, or deleted. That whole functionality is built right in. Obsessed.

Since this is a subclass of HTML::TokeParser, every single HTML::TokeParser method is available to you. To truly appreciate the power of this module, please go read the docs for HTML::TokeParser and HTML::Parser. Fr fr, do it.

🎀 CONTRUCTORS 🎀

💜 `new($source)`

The constructor for HTML::TokeParser::Simple works exactly like HTML::TokeParser's constructor does:

my $parser = HTML::TokeParser::Simple->new($filename);
# or
my $parser = HTML::TokeParser::Simple->new($filehandle);
# or
my $parser = HTML::TokeParser::Simple->new(\$html_string);

💜 `new($source_type, $source)`

If you wanna be extra explicit about things (main character energy, honestly), there's a whole new style of constructor available:

my $parser = HTML::TokeParser::Simple->new(file   => $filename);
# or
my $parser = HTML::TokeParser::Simple->new(handle => $filehandle);
# or
my $parser = HTML::TokeParser::Simple->new(string => $html_string);

Quick note bestie: you do not have to hand over a reference for the string when you're using the string constructor. That's the whole perk. 🫶

And as a lil convenience, you can also try to grab the HTML straight from a URL:

my $parser = HTML::TokeParser::Simple->new(url => 'http://some.url');

This one leans on LWP::Simple. So if that module isn't found, or the page can't be fetched, the constructor will croak() at you. 💀

🔥 PARSER METHODS 🔥

⭐ `get_token`

Returns the next token — literally the same one HTML::TokeParser::get_token() would give you. But it comes back blessed into a class that actually matches its token type. Glow-up. 💅

⭐ `get_tag`

Returns the next token that HTML::TokeParser::get_tag() would return — except it's blessed into either the HTML::TokeParser::Simple::Token::Tag::Start or the HTML::TokeParser::Simple::Token::Tag::End class. Pick your fighter.

⭐ `peek`

As of version 3.14, you can now peek() at the upcoming tokens without messing with the state of the parser at all. By default peek() hands you the text of the next token, but if you pass in an integer $count, you get the text of the next $count tokens. 🌈

This is so useful when you're debugging and you're like "where even am I in this document":

warn $parser->peek(3); # show the next 3 tokens

🌸 ACCESSORS 🌸

Heads up: all of these get called on the token object that got returned, not on the parser object itself. Important distinction, iykyk. 🎀

💖 Boolean Accessors

These accessors return true or false. Simple as.

is_tag([$tag]) — use this to figure out if you've got any tag at all. You can optionally pass a "tag type", which lets you match whether it's one particular tag. The tag you supply is case-insensitive.

if ( $token->is_tag ) { ... }

Optionally, you may pass a regular expression as an argument.

is_start_tag([$tag]) — use this to figure out if you've got a start tag. You can optionally pass a "tag type" so you can match whether it's one particular start tag. The tag you supply is case-insensitive.

if ( $token->is_start_tag ) { ... }
if ( $token->is_start_tag( 'font' ) ) { ... }

Optionally, you may pass a regular expression as an argument. So to match all the header tags (h1, h2, ... h6):

if ( $token->is_start_tag( qr/^h[123456]$/ ) ) { ... }

is_end_tag([$tag]) — use this to figure out if you've got an end tag. You can optionally pass a "tag type" so you can match whether it's one particular end tag. The tag you supply is case-insensitive.

Little trick: when you're testing for an end tag, the forward slash on the tag is optional.

while ( $token = $p->get_token ) {
  if ( $token->is_end_tag( 'form' ) ) { ... }
}

Or:

while ( $token = $p->get_token ) {
  if ( $token->is_end_tag( '/form' ) ) { ... }
}

Optionally, you may pass a regular expression as an argument.

is_text() — use this to figure out if you've got text. Do not confuse this with the return_text (deprecated) method described further down, bestie, they are NOT the same thing. is_text identifies the text that a user typically sees displayed in the Web browser. 👑
is_comment() — okay be honest, are you still reading this? Nobody reads POD. Don't you know you're supposed to go to CLPM, ask a question that's literally answered in the POD, and get flamed for it? It's a rite of passage. 😭

Really. 💀

is_comment is what you use to identify comments. Go see the HTML::Parser documentation for more info about comments — there's more going on there than you might think. 🌸

is_declaration() — this matches the DTD at the top of your HTML. (You do use DTDs, don't you? 🙈)
is_process_instruction() — Process Instructions come from XML. Super handy if you need to parse out PHP and similar things with a parser.

Currently there appear to be some problems with process instructions. You can override HTML::TokeParser::Simple::Token::ProcessInstruction if you need to.

is_pi() — shorthand for is_process_instruction(). Same vibe, shorter name. ✨

💖 Data Accessors

Some of these were originally return_ methods, but that name was not only unwieldy, it also went against reasonable conventions. The get_ methods listed below still have their return_ methods available for backwards compatibility reasons — but they literally just call their get_ counterpart. Like, calling return_tag() actually calls get_tag() internally. That's it, that's the whole relationship. 🫶

get_tag() — do you have a start tag or an end tag? This returns the type (lower case). Note: this is not the same thing as the get_tag() method on the actual parser object.
get_attr([$attribute]) — if you've got a start tag, this returns a hash ref with the attribute names as keys and the values as values.

And if you pass in an attribute name, it returns the value for just that one attribute. 🎯

Returns false if the token is not a start tag.

get_attrseq() — for a start tag, this is an array reference holding the sequence of the attributes, if there are any.

Returns false if the token is not a start tag.

return_text() — this method has been heavily deprecated (for a couple of years now) in favor of as_is. The reason? Programmers kept getting confused over the difference between is_text, return_text, and some parser methods like HTML::TokeParser::get_text and friends. Understandable honestly. 😭

Using this method still succeeds, but it will now carp at you, and it will be removed in the next major release of this module. Consider this your warning bestie. 💀

as_is() — this is the exact text of whatever the token is representing. Verbatim. No notes. ✨
get_token0() — for processing instructions, this returns the token found immediately after the opening tag. Example: for <?php, "php" will be the start of the returned string.

Note: process instruction handling appears to be incomplete in HTML::TokeParser.

Returns false if the token is not a process instruction.

✨ MUTATORS ✨

okay bestie, so the delete_attr() and set_attr() methods let you rewrite start tag attributes on the fly. It should be noted that bad HTML will be "corrected" by this. Specifically, the new tag will have all attributes lower-cased with the values properly quoted. 💅

Self-closing tags (e.g. <hr />) are also handled correctly. Some older browsers require a space prior to the final slash in a self-closed tag. If such a space is detected in the original HTML, it will be preserved. ✨

Calling a mutator on an token type that does not support that property is a no-op. For example:

if ($token->is_comment) {
   $token->set_attr(foo => 'bar'); # does nothing
}

delete_attr($name) — This method attempts to delete the attribute specified. It will silently fail if called on anything other than a start tag. The argument is case-insensitive, but must otherwise be an exact match of the attribute you are attempting to delete. If the attribute is not found, the method will return without changing the tag.

# <body bgcolor="#FFFFFF">
$token->delete_attr('bgcolor');
print $token->as_is;
# <body>

After this method is called, if successful, the as_is(), get_attr() and get_attrseq() methods will all return updated results. 🎀

set_attr($name,$value) — This method will set the value of an attribute. If the attribute is not found, then get_attrseq() will have the new attribute listed at the end.

# <p>
$token->set_attr(class => 'some_class');
print $token->as_is;
# <p class="some_class">

# <body bgcolor="#FFFFFF">
$token->set_attr('bgcolor','red');
print $token->as_is;
# <body bgcolor="red">

After this method is called, if successful, the as_is(), get_attr() and get_attrseq() methods will all return updated results. 💖

set_attr($hashref) — Under the premise that set_ methods should accept what their corresponding get_ methods emit, the following works:

$tag->set_attr($tag->get_attr);

Theoretically that's a no-op and for purposes of rendering HTML, it should be. However, internally this calls $tag->rewrite_tag, so see that method to understand how this may affect you.

Of course, this is useless if you want to actually change the attributes, so you can do this:

my $attrs = {
  class  => 'headline',
  valign => 'top'
};
$token->set_attr($attrs)
  if $token->is_start_tag('td') &&  $token->get_attr('class') eq 'stories';

rewrite_tag() — This method rewrites the tag. The tag name and the name of all attributes will be lower-cased. Values that are not quoted with double quotes will be. This may be called on both start or end tags. Note that both set_attr() and delete_attr() call this method prior to returning.

If called on a token that is not a tag, it simply returns. Regardless of how it is called, it returns the token.

# <body alink=#0000ff BGCOLOR=#ffffff class='none'>
$token->rewrite_tag;
print $token->as_is;
# <body alink="#0000ff" bgcolor="#ffffff" class="none">

A quick cleanup of sloppy HTML is now the following:

my $parser = HTML::TokeParser::Simple->new( string => $ugly_html );
while (my $token = $parser->get_token) {
    $token->rewrite_tag;
    print $token->as_is;
}

🔍 PARSER VERSUS TOKENS 🔍

The parser returns tokens that are blessed into appropriate classes. Some people get confused and try to call parser methods on tokens and token methods on the parser. To prevent this, HTML::TokeParser::Simple versions 1.4 and above now bless all tokens into appropriate token classes. Please keep this in mind while using this module (and many thanks to PodMaster http://www.perlmonks.org/index.pl?node_id=107642 for pointing out this issue to me.) 🙏

🌈 EXAMPLES 🌈

🕵️ Finding comments 🕵️

For some strange reason, your Pointy-Haired Boss (PHB) is convinced that the graphics department is making fun of him by embedding rude things about him in HTML comments. You need to get all HTML comments from the HTML. 😭

use strict;
use HTML::TokeParser::Simple;

my @html_docs = glob( "*.html" );

open PHB, "> phbreport.txt" or die "Cannot open phbreport for writing: $!";

foreach my $doc ( @html_docs ) {
    print "Processing $doc\n";
    my $p = HTML::TokeParser::Simple->new( file => $doc );
    while ( my $token = $p->get_token ) {
        next unless $token->is_comment;
        print PHB $token->as_is, "\n";
    }
}

close PHB;

🧹 Stripping Comments 🧹

Uh oh. Turns out that your PHB was right for a change. Many of the comments in the HTML weren't very polite. Since your entire graphics department was just fired, it falls on you need to strip those comments from the HTML. 💀

use strict;
use HTML::TokeParser::Simple;

my $new_folder = 'no_comment/';
my @html_docs  = glob( "*.html" );

foreach my $doc ( @html_docs ) {
    print "Processing $doc\n";
    my $new_file = "$new_folder$doc";

    open PHB, "> $new_file" or die "Cannot open $new_file for writing: $!";

    my $p = HTML::TokeParser::Simple->new( $file => doc );
    while ( my $token = $p->get_token ) {
        next if $token->is_comment;
        print PHB $token->as_is;
    }
    close PHB;
}

🔄 Changing form tags 🔄

Your company was foo.com and now is bar.com. Unfortunately, whoever wrote your HTML decided to hardcode http://www.foo.com/ into the action attribute of the form tags. You need to change it to http://www.bar.com/. 💅

use strict;
use HTML::TokeParser::Simple;

my $new_folder = 'new_html/';
my @html_docs  = glob( "*.html" );

foreach my $doc ( @html_docs ) {
    print "Processing $doc\n";
    my $new_file = "$new_folder$doc";

    open FILE, "> $new_file" or die "Cannot open $new_file for writing: $!";

    my $p = HTML::TokeParser::Simple->new( file => $doc );
    while ( my $token = $p->get_token ) {
        if ( $token->is_start_tag('form') ) {
            my $action = $token->get_attr(action);
            $action =~ s/www\.foo\.com/www.bar.com/;
            $token->set_attr('action', $action);
        }
        print FILE $token->as_is;
    }
    close FILE;
}

⚠️ CAVEATS ⚠️

For compatibility reasons with HTML::TokeParser, methods that return references are violating encapsulation and altering the references directly will alter the state of the object. Subsequent calls to rewrite_tag() can thus have unexpected results. Do not alter these references directly unless you are following behavior described in these docs. In the future, certain methods such as get_attr, get_attrseq and others may return a copy of the reference rather than the original reference. This behavior has not yet been changed in order to maintain compatibility with previous versions of this module. At the present time, your author is not aware of anyone taking advantage of this "feature," but it's better to be safe than sorry. 😬

Use of $HTML::Parser::VERSION which is less than 3.25 may result in incorrect behavior as older versions do not always handle XHTML correctly. It is the programmer's responsibility to verify that the behavior of this code matches the programmer's needs. 👑

Note that HTML::Parser processes text in 512 byte chunks. This sometimes will cause strange behavior and cause text to be broken into more than one token. You can suppress this behavior with the following command:

$p->unbroken_text( [$bool] );

See the HTML::Parser documentation and http://www.perlmonks.org/index.pl?node_id=230667 for more information. 🔥

🐛 BUGS 🐛

There are no known bugs, but that's no guarantee. 😭

Address bug reports and comments to: eop_divo_sitruc@yahoo.com. When sending bug reports, please provide the version of HTML::Parser, HTML::TokeParser, HTML::TokeParser::Simple, the version of Perl, and the version of the operating system you are using. 🫶

Reverse the name to email the author. 💌

🧬 SUBCLASSING 🧬

You may wish to change the behavior of this module. You probably do not want to subclass HTML::TokeParser::Simple. Instead, you'll want to subclass one of the token classes. HTML::TokeParser::Simple::Token is the base class for all tokens. Global behavioral changes should go there. Otherwise, see the appropriate token class for the behavior you wish to alter. 🌈

👀 SEE ALSO 👀

HTML::TokeParser::Simple::Token
HTML::TokeParser::Simple::Token::Tag
HTML::TokeParser::Simple::Token::Text
HTML::TokeParser::Simple::Token::Comment
HTML::TokeParser::Simple::Token::Declaration
HTML::TokeParser::Simple::Token::ProcessInstruction

👑 AUTHOR 👑

Curtis "Ovid" Poe eop_divo_sitruc@yahoo.com