HTML::...xtract(3pm)

📦
version 2.15 · 7 versions
, in debian bookworm, debian trixie, ubuntu noble, ubuntu resolute, ubuntu stonking

HTML::...xtract(3pm) 💖

User Contributed Perl Documentation ✨ HTML::...xtract(3pm)

okay bestie, buckle up, because this is the page for HTML::TableExtract — the Perl module that reaches into an HTML document and pulls out exactly the tables you want. 🎀

🌸 NAME 🌸

HTML::TableExtract — a Perl module for extracting the content contained in tables within an HTML document, either as text or encoded element trees. no cap, that's the whole job. 💅

✨ SYNOPSIS ✨

so the vibe is: matched tables come back to you as table objects, and you can match them by column headers, by depth, by count within a depth, by table tag attributes, or by literally any combination of those four. 🦋

# Matched tables are returned as table objects; tables can be matched
# using column headers, depth, count within a depth, table tag
# attributes, or some combination of the four.

# Example: Using column header information.
# Assume an HTML document with tables that have "Date", "Price", and
# "Cost" somewhere in a row. The columns beneath those headings are
# what you want to extract. They will be returned in the same order as
# you specified the headers since 'automap' is enabled by default.

use HTML::TableExtract;
my $te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] );
$te->parse($html_string);

# Examine all matching tables
foreach my $ts ($te->tables) {
  print "Table (", join(',', $ts->coords), "):\n";
  foreach my $row ($ts->rows) {
     print join(',', @$row), "\n";
  }
}

# Shorthand...top level rows() method assumes the first table found in
# the document if no arguments are supplied.
foreach my $row ($te->rows) {
   print join(',', @$row), "\n";
}

# Example: Using depth and count information.
# Every table in the document has a unique depth and count tuple, so
# when both are specified it is a unique table. Depth and count both
# begin with 0, so in this case we are looking for a table (depth 2)
# within a table (depth 1) within a table (depth 0, which is the top
# level HTML document). In addition, it must be the third (count 2)
# such instance of a table at that depth.

my $te = HTML::TableExtract->new( depth => 2, count => 2 );
$te->parse_file($html_file);
foreach my $ts ($te->tables) {
   print "Table found at ", join(',', $ts->coords), ":\n";
   foreach my $row ($ts->rows) {
      print "   ", join(',', @$row), "\n";
   }
}

# Example: Using table tag attributes.
# If multiple attributes are specified, all must be present and equal
# for match to occur.

my $te = HTML::TableExtract->new( attribs => { border => 1 } );
$te->parse($html_string);
foreach my $ts ($te->tables) {
  print "Table with border=1 found at ", join(',', $ts->coords), ":\n";
  foreach my $row ($ts->rows) {
     print "   ", join(',', @$row), "\n";
  }
}

# Example: Extracting as an HTML::Element tree structure
# Rather than extracting raw text, the html can be converted into a
# tree of element objects. The HTML document is composed of
# HTML::Element objects and the tables are HTML::ElementTable
# structures. Using this, the contents of tables within a document can
# be edited in-place.

use HTML::TableExtract qw(tree);
my $te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) );
$te->parse_file($html_file);
my $table = $te->first_table_found;
my $table_tree = $table->tree;
$table_tree->cell(4,4)->replace_content('Golden Goose');
my $table_html = $table_tree->as_HTML;
my $table_text = $table_tree->as_text;
my $document_tree = $te->tree;
my $document_html = $document_tree->as_HTML;

💜 DESCRIPTION 💜

HTML::TableExtract is a subclass of HTML::Parser, and its whole main character moment is extracting the info from the tables you actually care about inside an HTML document. 🫶 Every extracted table's info gets stored in table objects. You can pull those tables out as text, as HTML, or as HTML::ElementTable structures (which is the move if you want to edit or manipulate them in-place). iconic.

Right now there are four constraints you can use to say which tables you want pulled out of a document: Headers, Depth, Count, and Attributes. 🌈

  • Headers — the most flexible and adaptive technique, fr fr. You hand it an array of text you expect to show up above the data in the tables you want. Once every one of those headers has been located in a row of that table, all the further cells underneath the columns that matched your headers get extracted. Every other column gets ignored — think of it as vertical slices through a table. 🍰 On top of that, TableExtract automatically rearranges each row into the same order as the headers you gave it. Wanna turn that off? Set automap to 0 when you create the object, and instead lean on the column_map() method to find out the order the headers were actually found in. Also, TableExtract automatically compensates for cell span issues, so the columns are really the same columns you'd see with your eyeballs in a browser. 👀 You can kill that behavior by setting the gridmap parameter to 0. And heads up: HTML is stripped from the entire textual content of a cell before any header matching is attempted — unless the keep_html parameter was enabled. 🧚
  • Depth and Count — the more specific way to point at tables in relation to each other. Depth is how deeply a table lives inside other tables. A top-level table in the document has a depth of 0. A table inside a top-level table has a depth of 1, and so on. Each depth is basically a layer; tables sharing the same depth are on the same layer. 💅 Within each of those layers, Count is the order in which a table was seen at that depth, starting with 0. Give both a depth and a count and you have uniquely specified one single table in the document. 🔥
  • Attributes — these match based on the attributes of the html <table> tag. Think border widths, or background color. 🎨

Each of the Headers, Depth, Count, and Attributes specifications is cumulative in how it affects the overall extraction. So, for instance:

  • If you specify only a Depth, you get all tables at that depth (and note that these could very well live in separate higher-level tables all over the document, since depth extends across tables).
  • If you specify only a Count, you get the tables at that Count from all depths (i.e., the nth occurrence of a table at each depth).
  • If you only specify Headers, you get all tables in the document that contain those column headers.
  • If you've specified multiple constraints out of Headers, Depth, Count, and Attributes, then each constraint has veto power over whether a particular table gets extracted. 👑

If no Headers, Depth, Count, or Attributes are specified at all, then all tables match. slay, everything is invited to the party. 🎉

When you're extracting only text from tables, the text is decoded with HTML::Entities by default — you can disable that by setting the decode parameter to 0. ✨

🦋 Extraction Modes 🦋

The default mode of extraction for HTML::TableExtract is raw text or HTML. In this mode, embedded tables are completely decoupled from one another. In this case, HTML::TableExtract is a subclass of HTML::Parser:

use HTML::TableExtract;

Alternatively, tables can be extracted as HTML::ElementTable structures, which are in turn embedded in an HTML::Element tree representing the entire HTML document. Embedded tables are not decoupled from one another here, because that tree structure has to be maintained. In this case, HTML::TableExtract is a subclass of HTML::TreeBuilder (itself a subclass of HTML:::Parser):

use HTML::TableExtract qw(tree);

Either way, the basic interface for HTML::TableExtract and the resulting table objects stays exactly the same — the only thing that changes is what you can do with the resulting data. 💫

HTML::TableExtract is a subclass of HTML::Parser, and as such it inherits all of its basic methods such as parse() and parse_file(). During scans, start(), end(), and text() are utilized. Feel free to override them! but if you don't eventually invoke them in the SUPER class with some content, results are not guaranteed. 😭 consider yourself warned bestie.

🎀 Advice 🎀

The main point of this module was to provide a flexible way of extracting tabular information from HTML documents without relying too heavily on the document layout. For that reason, I suggest using Headers whenever possible — that way you're anchoring your extraction on what the document is trying to communicate, rather than on some feature of the HTML that makes up the document (other than the fact that the data is contained in a table). 🫶🌸

✨ METHODS ✨

These are the top-level methods you call on the HTML::TableExtract object itself, bestie. Any table that actually matched your query comes back as its own separate object of type HTML::TableExtract::Table, and those table objects have their own whole set of methods, which we get to further down. 🫶

🎀 CONSTRUCTOR 🎀

new()

Returns a brand-new HTML::TableExtract object. The valid attributes are giving main character energy, and here's every single one:

  • headers — Passed in as an array reference, headers are the strings you actually care about, the ones sitting at the top of columns inside the tables you're targeting. They can be plain strings or regular expressions (qr//). If they're strings, they eventually get run through a non-anchored, case-insensitive regex, so regexp special characters are allowed in them.

The table row that holds the headers does not get returned, unless keep_headers was specified or you're extracting into an element tree. Either way, you can grab the header row from inside the table object using the hrow() method.

Columns that aren't beneath one of the headers you gave get ignored — unless slice_columns was set to 0. And columns will, by default, get rearranged into the same order as the headers you provided (peep the automap parameter for the details) unless slice_columns is 0.

Also, by default, columns mean what you'd visually see beneath that header when the table is rendered in a browser. See the gridmap parameter for more on that.

HTML inside a header gets stripped before the match is attempted — unless the keep_html parameter was specified and strip_html_on_match is false. 🧚

  • depth — Tells the module how deeply embedded your tables of interest should be. Top-level tables in the HTML document have a depth of 0, tables inside top-level tables have a depth of 1, and so on down the line.
  • count — Which table within each depth you're interested in, starting from 0.
  • attribs — Passed as a hash reference, attribs specifies attributes of interest inside the HTML <table> tag itself.
  • automap — Automatically applies the ordering reported by column_map() to the rows returned by rows(). This only makes a difference if you specified Headers and they turn out to be in a different order in the table than what you specified. Automap will rearrange the columns into the same order as the headers appear. To get the original ordering back, you'll need to take another slice of each row using column_map(). automap is enabled by default. 💅
  • slice_columns — Enabled by default. This option controls whether vertical slices get returned from under headers that match. When you disable it, all columns of the matching table are retained, regardless of whether they had a matching header above them. Disabling this also disables automap.
  • keep_headers — Disabled by default, and only applicable when header constraints have been specified. When enabled, keep_headers retains the matching header row as the first row of table data. This option has no effect if you're extracting into an element tree structure. In any case, the header row is always reachable from the table method hrow().
  • gridmap — Controls whether the table contents come back as a grid or as a tree. ROWSPAN and COLSPAN issues get compensated for, and columns really are columns. Empty phantom cells are created wherever they would've been obscured by ROWSPAN or COLSPAN settings. This genuinely becomes a whole thing when you're extracting columns beneath headers. Enabled by default. ✨
  • subtables — Extract all tables embedded within matched tables.
  • decode — Automatically decodes retrieved text with HTML::Entities::decode_entities(). Enabled by default. Has no effect if keep_html was specified or if you're extracting into an element tree structure.
  • br_translate — Translates <br> tags into newlines. Sometimes the leftover text is genuinely hard to parse if the <br> tag just gets dropped. Enabled by default. Has no effect if keep_html is enabled or if you're extracting into an element tree structure.
  • keep_html — Returns the raw HTML contained in the cell, instead of just the visible text. Embedded tables are not retained in the HTML extracted from a cell. If this option is on, your patterns for header matches have to take HTML in the string into account. This option has no effect if you're extracting into an element tree structure.
  • strip_html_on_match — When keep_html is enabled, HTML gets stripped by default while the module attempts to match header strings (so if strip_html_on_match is not enabled and keep_html is, you'd have to include potential HTML tags in the regexp for header matches). Stripped header tags are replaced with an empty string — like, 'hot d<em>og</em>' would become 'hot dog' before the match is attempted.
  • error_handle — The filehandle where error messages get printed. STDERR by default.
  • debug — Prints some debugging information to STDERR, more of it for higher values. If error_handle was provided, the messages get printed there instead of STDERR.

🌸 REGULAR METHODS 🌸

These ones you invoke directly on an HTML::TableExtract object.

  • depths() — Returns all depths that contained matched tables in the document.
  • counts($depth) — For one particular depth, returns all counts that contained matched tables.
  • table($depth, $count) — For one particular depth and count, returns the table object for the table found, if there is one.
  • tables() — Returns table objects for all tables that matched. Returns an empty list if nothing matched.
  • first_table_found() — Returns the table state object for the first table matched in the document. Returns undef if no tables were matched.
  • current_table() — Returns the current table object while parsing the HTML. Only useful if you're messing around with overriding HTML::Parser methods.
  • tree() — If the module was invoked in tree extraction mode, returns a reference to the top node of the HTML::Element tree structure for the entire document (which ultimately includes all tables within the document).
  • tables_report([$show_content, $col_sep]) — Returns a string summarizing the extracted tables, along with their depth and count. Optionally takes a $show_content flag, which will also dump the extracted contents of each table, with columns separated by $col_sep. Default $col_sep is ':'.
  • tables_dump([$show_content, $col_sep]) — Same as tables_report() except it dumps the information to STDOUT.
  • start, end, text — These are the hooks into HTML::Parser. If you want to subclass this module and have things actually work, you have to call these with content at some point.

💀 DEPRECATED METHODS 💀

Tables used to be called 'table states'. So accordingly, these following methods still work, but yeah, they've been deprecated:

  • table_state() — Is now table()
  • table_states() — Is now tables()
  • first_table_state_found() — Is now first_table_found()

👑 TABLE METHODS 👑

These ones you invoke from an HTML::TableExtract::Table object, like the ones you get back from the tables() method.

  • rows() — Returns all rows within a matched table. Each row that comes back is a reference to an array containing the text, the HTML, or a reference to the HTML::Element object of each cell, depending on the mode of extraction. Tables with rowspan or colspan attributes will have some cells containing undef. Returns a list or a reference to an array, depending on context.
  • columns() — Returns all columns within a matched table. Each column that comes back is a reference to an array containing the text, the HTML, or a reference to the HTML::Element object of each cell, depending on the mode of extraction. Tables with rowspan or colspan attributes will have some cells containing undef.
  • row($row) — Returns one particular row from within a matched table, either as a list or an array reference, depending on context.
  • column($col) — Returns one particular column from within a matched table as a list or an array reference, depending on context.
  • cell($row,$col) — Returns one particular item from within a matched table — whether that's the text, the HTML, or a reference to the HTML::Element object of that cell, depending on the mode of extraction. If the cell was covered due to rowspan or colspan effects, it returns undef.
  • space($row,$col) — The same as cell(), except in cases where the given coordinates were covered due to rowspan or colspan issues, in which case the content of the covering cell is returned instead of undef.
  • depth() — Returns the depth at which this table was found.
  • count() — Returns the count for this table within the depth it was found.
  • coords() — Returns depth and count in a list.
  • tree() — If the module was invoked in tree extraction mode, this accessor gives you a reference to the HTML::ElementTable structure encompassing the table.
  • hrow() — Returns the header row as a list when headers were specified as a constraint. If keep_headers was specified initially, this is equivalent to the first row returned by the rows() method.
  • column_map() — Returns the order (via indices) in which the provided headers were found. These indices can be used as slices on rows to either order the rows in the same order as the headers, or restore the rows to their natural order — depending on whether the rows have been pre-adjusted using the automap parameter.
  • lineage() — Returns the path of matched tables that led to matching this table. The path is a list of array refs containing depth, count, row, and column values for each ancestor table involved. Heads up though: corresponding table objects will not exist for ancestral tables that didn't match specified constraints. 🙈

🌳 NOTES ON TREE EXTRACTION MODE 🌳

okay so like, as we said up there, HTML::TableExtract can be invoked in 'tree' mode' and when that happens, the resulting HTML and the extracted tables get encoded in HTML::Element tree structures:

  use HTML::TableExtract 'tree';

bestie there's a whole bunch of stuff you gotta keep in mind while you're in this mode, so let's get into it 💅

The entire HTML document gets encoded into an HTML::Element tree. Every table is part of that structure, yeah, but it's also tracked separately via an HTML::ElementTable structure, which is like a specialized form of HTML::Element tree. so it's giving main character in two stories at once ✨

The HTML::ElementTable objects are accessible by invoking the tree() method from within each table object that HTML::TableExtract hands back to you. And those HTML::ElementTable objects have their own row(), col(), and cell() methods (among others 👀). Do NOT confuse these with the row() and column() methods that the HTML::TableExtract::Table objects give you. they are different girlies, okay? 💀

So for example: the row() method from HTML::ElementTable will give you a reference to a 'glob' of all the elements in that row. Actions (like setting attributes) that you perform on that row reference will affect all the elements within that row. iconic. but the row() method from the HTML::TableExtract::Table object? that one returns an array (either by reference or as a list, depending on context) of the contents of each cell within the row. and in tree mode, that content is represented by individual references to each cell — which are references to the same HTML::Element objects chilling in the HTML::Element tree. no cap 🫶

So the cell() methods in both cases will therefore return references to the same object. slay, consistency. BUT — the exception to this is when a 'cell' in the table grid was originally 'covered' due to rowspan or colspan issues — in that case the cell content will be undef. 😭 likewise, the row() or column() methods from HTML::TableExtract::Table objects will return arrays that potentially contain a mixture of object references and undefs. it's giving chaos a little bit.

ngl if you're gonna be doing LOTS of manipulation of the table elements, it might be more efficient to access them via the methods provided by the HTML::ElementTable object instead. See HTML::ElementTable for more info on how to manipulate those objects 💖

An alternative to the cell() method in HTML::TableExtract::Table is the space() method. It's largely similar to cell(), EXCEPT when you give it coordinates of a cell that was covered due to rowspan or colspan effects — it will return the contents of the cell that was covering that space rather than undef. so like, if for example cell (0,0) had a rowspan of 2 and colspan of 2, then cell(1,1) would return undef and space(1,1) would return the same content as cell(0,0) or space(0,0). the way that actually makes sense?? ate 💅

📦 REQUIRES 📦

✨ OPTIONALLY REQUIRES ✨

👑 AUTHOR 👑

Matthew P. Sisk, <sisk@mojotoad.com>

Copyright (c) 2000-2017 Matthew P. Sisk. All rights reserved. All wrongs revenged. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

(not the "all wrongs revenged" line taking me OUT 😭 iconic behavior fr fr)

🌈 SEE ALSO 🌈

and that's the whole page bestie!! 🦋 this bad girl shipped as perl v5.34.0 and the page is dated 2022-06-14 — so yeah, vintage but still serving 💜 go forth and extract those tables, you absolute legend 🌸💖✨

📦 where this page comes from