HTML::...eParser(3pm)

📦
version 0.43 · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

HTML::TableParser(3pm) 💖

okay so this is the User Contributed Perl Documentation page for HTML::TableParser, and bestie, it's giving extremely useful table-extraction energy. This is part 1, and there's more coming, so don't go anywhere. ✨

NAME 🎀

HTML::TableParser — HTML::TableParser — Extract data from an HTML table

VERSION 🦋

version 0.43

SYNOPSIS 🌸

use HTML::TableParser;

@reqs = (
         {
          id => 1.1,                    # id for embedded table
          hdr => \&header,              # function callback
          row => \&row,                 # function callback
          start => \&start,             # function callback
          end => \&end,                 # function callback
          udata => { Snack => 'Food' }, # arbitrary user data
         },
         {
          id => 1,                      # table id
          cols => [ 'Object Type',
                    qr/object/ ],       # column name matches
          obj => $obj,                  # method callbacks
         },
        );

# create parser object
$p = HTML::TableParser->new( \@reqs,
                 { Decode => 1, Trim => 1, Chomp => 1 } );
$p->parse_file( 'foo.html' );

# function callbacks
sub start {
  my ( $id, $line, $udata ) = @_;
  #...
}

sub end {
  my ( $id, $line, $udata ) = @_;
  #...
}

sub header {
  my ( $id, $line, $cols, $udata ) = @_;
  #...
}

sub row  {
  my ( $id, $line, $cols, $udata ) = @_;
  #...
}

DESCRIPTION 💅

okay so HTML::TableParser uses HTML::Parser to extract data from an HTML table. The data is returned via a series of user defined callback functions or methods. Specific tables may be selected either by matching a unique table id or by matching against the column names. Multiple (even nested) tables may be parsed in a document in one pass.

Table Identification 🎀

Each table is given a unique id, relative to its parent, based upon its order and nesting. The first top level table has id 1, the second 2, etc. The first table nested in table 1 has id 1.1, the second 1.2, etc. The first table nested in table 1.1 has id 1.1.1, etc. These, as well as the tables' column names, may be used to identify which tables to parse.

Data Extraction 🔥

As the parser traverses a selected table, it will pass data to user provided callback functions or methods after it has digested particular structures in the table. All functions are passed the table id (as described above), the line number in the HTML source where the table was found, and a reference to any table specific user provided data.

Table Start ✨

The start callback is invoked when a matched table has been found.

Table End ✨

The end callback is invoked after a matched table has been parsed.

The hdr callback is invoked after the table header has been read in. Some tables do not use the <th> tag to indicate a header, so this function may not be called. It is passed the column names.

Row ✨

The row callback is invoked after a row in the table has been read. It is passed the column data.

Warn ✨

The warn callback is invoked when a non-fatal error occurs during parsing. Fatal errors croak.

New ✨

This is the class method to call to create a new object when HTML::TableParser is supposed to create new objects upon table start.

Callback API 💖

Callbacks may be functions or methods or a mixture of both. In the latter case, an object must be passed to the constructor. (More on that later.)

The callbacks are invoked as follows:

start( $tbl_id, $line_no, $udata );

end( $tbl_id, $line_no, $udata );

hdr( $tbl_id, $line_no, \@col_names, $udata );

row( $tbl_id, $line_no, \@data, $udata );

warn( $tbl_id, $line_no, $message, $udata );

new( $tbl_id, $udata );

Data Cleanup 🧚

There are several cleanup operations that may be performed automatically:

  • Chomp — chomp() the data
  • Decode — Run the data through HTML::Entities::decode.
  • DecodeNBSP — Normally HTML::Entitites::decode changes a non-breaking space into a character which doesn't seem to be matched by Perl's whitespace regexp. Setting this attribute changes the HTML "nbsp" character to a plain 'ol blank.
  • Trim — remove leading and trailing white space.

Data Organization 🌈

Column names are derived from cells delimited by the <th> and </th> tags. Some tables have header cells which span one or more columns or rows to make things look nice. HTML::TableParser determines the actual number of columns used and provides column names for each column, repeating names for spanned columns and concatenating spanned rows and columns. For example, if the table header looks like this:

+----+--------+----------+-------------+-------------------+
|    |        | Eq J2000 |             | Velocity/Redshift |
| No | Object |----------| Object Type |-------------------|
|    |        | RA | Dec |             | km/s |  z  | Qual |
+----+--------+----------+-------------+-------------------+

The columns will be:

No
Object
Eq J2000 RA
Eq J2000 Dec
Object Type
Velocity/Redshift km/s
Velocity/Redshift z
Velocity/Redshift Qual

Row data are derived from cells delimited by the <td> and </td> tags. Cells which span more than one column or row are handled correctly, i.e. the values are duplicated in the appropriate places.

METHODS 👑

new 🫶

$p = HTML::TableParser->new( \@reqs, \%attr );

This is the class constructor. It is passed a list of table requests as well as attributes which specify defaults for common operations. Table requests are documented in "Table Requests".

The %attr hash provides default values for some of the table request attributes, namely the data cleanup operations ("Chomp", "Decode", "Trim"), and the multi match attribute "MultiMatch", i.e.,

$p = HTML::TableParser->new( \@reqs, { Chomp => 1 } );

will set Chomp on for all of the table requests, unless overridden by them. The data cleanup operations are documented above; "MultiMatch" is documented in "Table Requests".

Decode defaults to on; all of the others default to off.

parse_file 🦋

This is the same function as in HTML::Parser.

parse 🦋

This is the same function as in HTML::Parser.

💖 Table Requests 💖

A table request is just a hash — HTML::TableParser reads it to figure out which tables to parse, which callbacks to fire, and any data cleanup you want done. One call to the parser can carry multiple requests, and every table gets tied to a single request (even if like, several requests match it).

A single request can match several tables, but bestie, unless you set the MultiMatch attribute on that request, it's only gonna be used for the first matching table. That's it. One and done.

A table request that matches a table id of "DEFAULT" becomes the catch-all — it matches every table that no other request already grabbed. And heads up: tables are compared to requests in the order you passed them to the new() method, so put the DEFAULT one LAST or it will not behave properly. Order matters, periodt.

🔍 Identifying tables to parse

HTML::TableParser needs you to tell it which tables to parse. You can do that by matching table ids, column names, or honestly a combo of both. Here are the request-hash keys that do the work:

  • id — this one matches on table id. It can take one of these forms:

exact match

id => $match
id => '1.2'

Here $match is a scalar that gets compared directly to the table id.

regular expression

id => $re
id => qr/1\.\d+\.2/

$re is a regex, and it must be built with the qr// operator. No shortcuts.

subroutine

id => \&my_match_subroutine
id => sub { my ( $id, $oids ) = @_ ;
         $oids[0] > 3 && $oids[1] < 2 }

Here id gets a coderef to a sub that returns true if the table matches and false if it doesn't. That sub gets handed two arguments: the table id as a scalar string (e.g. 1.2.3) and the table id as an arrayref (e.g. $oids = [ 1, 2, 3]).

And id can also be given an array holding any combo of the above:

id => [ '1.2', qr/1\.\d+\.2/, sub { ... } ]

Elements in that array may be preceded by a modifier that says what happens if the table matches on that element. The modifiers and what they mean:

  • "-" — if the id matches, it's explicitly excluded from being processed by this request.
  • "--" — if the id matches, it's skipped by all requests. Like, gone gone.
  • "+" — if the id matches, it will be processed by this request. This is the default action.

An example:

id => [ '-', '1.2', 'DEFAULT' ]

which says: use this request for all tables, except table 1.2. And:

id => [ '--', '1.2' ]

means Table 2 is just plain skipped altogether. 💀

  • cols — this one matches on column names. It can take one of these forms:

exact match

cols => $match
cols => 'Snacks01'

$match is a scalar compared directly to the column names. If any column matches, the table gets processed.

regular expression

cols => $re
cols => qr/Snacks\d+/

$re is a regex, and again it must be built with the qr// operator. And again, a successful match against any column name means the table gets processed.

subroutine

cols => \&my_match_subroutine
cols => sub { my ( $id, $oids, $cols ) = @_ ;
              ... }

Here cols gets a coderef to a sub that returns true if the table matches, false if not. That sub gets three arguments: the table id as a scalar string (e.g. 1.2.3), the table id as an arrayref (e.g. $oids = [ 1, 2, 3]), and the column names as an arrayref (e.g. $cols = [ 'col1', 'col2' ]). This option hands your calling routine the power to make arbitrary selections based on table id and columns. Iconic, honestly.

cols can be given an arrayref holding any combination of the above:

cols => [ 'Snacks01', qr/Snacks\d+/, sub { ... } ]

Elements in that array may be preceded by a modifier saying what happens if the table matches on that element. They're the same as the table id modifiers up above. 👑

  • colre — this one is deprecated, and it's here purely for backwards compatibility. It's an arrayref containing the regular expressions to match, or a scalar containing a single reqular expression.

More than one of these can be used for a single table request. A request can match more than one table. But by default a request is used only once (yes, even the "DEFAULT" id match!). Set the MultiMatch attribute to unlock multiple matches per request.

When it tries to match a table, here's the lil algorithm it runs:

  1. The table id is compared to the requests that contain an id match. The first such match is used (in the order given in the passed array).
  2. If no explicit id match is found, column name matches are attempted. The first such match is used (in the order given in the passed array).
  3. If no column name match is found (or there were none requested), the first request that matches an id of "DEFAULT" is used.

🎀 Specifying the data callbacks

Callback functions get specified with the callback attributes start, end, hdr, row, and warn. Set them to code references, like:

%table_req = ( ..., start => \&start_func, end => \&end_func )

Wanna use methods instead? Then specify the object with the obj key, and pass the method names through the callback attributes, set to strings. If you don't specify method names, they'll default to (you guessed it 💅) start, end, hdr, row, and warn.

$obj = SomeClass->new();
# ...
%table_req_1 = ( ..., obj => $obj );
%table_req_2 = ( ..., obj => $obj, start => 'start',
                           end => 'end' );

You can also have HTML::TableParser whip up a brand new object for you for each table by specifying the class attribute. By default the constructor is assumed to be the class's new() method; if it's not, tell it with the new attribute:

use MyClass;
%table_req = ( ..., class => 'MyClass', new => 'mynew' );

To use a plain function instead of a method for one particular callback, just set that callback attribute to a code reference:

%table_req = ( ..., obj => $obj, end => \&end_func );

You don't have to provide all the callbacks. But you should not use both obj and class in the same table request. Pick a lane bestie.

HTML::TableParser automatically figures out if your object or class has one of the required methods. If you want it to not use a particular method, set it equal to undef. For example:

%table_req = ( ..., obj => $obj, end => undef )

means the object's end method should not be called, even if it exists. Snubbed. 🙈

You can also pass arbitrary data through to the callback functions via the udata attribute:

%table_req = ( ..., udata => \%hash_of_my_special_stuff )

✨ Specifying Data cleanup operations

Data cleanup operations can be specified uniquely for each table. The available keys are Chomp, Decode, Trim. Set them to a non-zero value if you want that operation to happen.

🌸 Other Attributes

The MultiMatch key is what you reach for when a request is capable of handling multiple tables in the document. Ordinarily a request processes a single table only (yes, even "DEFAULT" requests). Set it to a non-zero value to let the request handle more than one table.

🐛 BUGS

Please report any bugs or feature requests on the bugtracker website <https://rt.cpan.org/Public/Dist/Display.html?Name=HTML-TableParser> or by email to bug-HTML-TableParser@rt.cpan.org <mailto:bug-HTML-TableParser@rt.cpan.org>.

When you submit a bug or a request, pretty please include a test-file or a patch to an existing test-file that shows off the bug or the feature you want. Receipts help everyone. 🫶

📦 SOURCE

The development version lives on github at <https://github.com/djerius/html-tableparser> and may be cloned from <git://github.com/djerius/html-tableparser.git>

👑 AUTHOR

Diab Jerius <djerius@cpan.org>

This software is Copyright (c) 2018 by Smithsonian Astrophysical Observatory.

This is free software, licensed under:

The GNU General Public License, Version 3, June 2007

okay that's literally the whole page, bestie, you made it to the end 🎉 this was HTML::...eParser(3pm) from the User Contributed Perl Documentation, perl v5.26.1, dated 2018-03-25 — go forth and parse those tables, you absolute icon 💅✨🦋

📦 where this page comes from