htmlparse(3tcl)

htmlparse 💖

okay so bestie, welcome to the htmlparse manual page — it's from the Tcl crew, section 3, and the whole vibe is HTML Parser. we're gonna parse some HTML strings and it's gonna be iconic. grab a snack, this one's long 🍓

---

✨ NAME ✨

htmlparse — the procedures that parse HTML strings, periodt 💅

---

✨ SYNOPSIS ✨

first up, the prerequisites — you gotta require these before anything slays:

package require Tcl 8.5 9

package require struct::stack 1.3

package require cmdline 1.1

package require htmlparse ?1.2.3?

and now the actual procedures, one per line, no cap:

::htmlparse::parse  ?-cmd  cmd?  ?-vroot  tag?  ?-split n? ?-incvar var?
?-queue q? html

::htmlparse::debugCallback ?clientdata? tag slash param textBehindTheTag

::htmlparse::mapEscapes html

::htmlparse::2tree html tree

::htmlparse::removeVisualFluff tree

::htmlparse::removeFormDefs tree

note bestie: every single one of those ?question-mark? bits means the argument is optional — you can leave it out and it still works 🫶

---

part 1 done, more coming, don't go anywhere 🦋

✨ DESCRIPTION ✨

okay so the htmlparse package is literally a whole toolkit that lets libraries and applications parse HTML sitting in a string and turn it into whatever representation they vibe with. no cap. 🦋

Here's the lineup of commands you get:

---

🌸 ::htmlparse::parse 🌸

::htmlparse::parse ?-cmd cmd? ?-vroot tag? ?-split n? ?-incvar var? ?-queue q? html

This one is the basic parser for HTML, the main character fr. 💅 It takes an HTML string, parses it, and invokes a command prefix for every single tag it runs into.

Good news bestie: the HTML does not have to be valid for this parser to work. It's the job of the command that gets invoked for every tag to do that checking. That invoked command is also responsible for handling tag attributes and character entities (aka escaped characters). The parser hands the un-interpreted tag attributes over to the invoked command to help with the attribute situation, and the package at large ships a helper command, ::htmlparse::mapEscapes, to help out with the escape situation. 🫶

Also: the parser does ignore leading DOCTYPE declarations and all valid HTML comments it encounters. periodt.

Every bit of info beyond the HTML string itself gets specified via options, and those are explained down below. 👇

To help you actually understand the options, here's some more background on the parser.

It can detect incomplete tags in the HTML string you feed it. Normally that makes the parser throw an error, but if you use the -incvar option to name a global (or namespace) variable, the parser will stash the incomplete part of the input into that variable instead. This is a huge slay for handling incrementally arriving HTML, because the parser handles whatever it can and defers the incomplete part until more data shows up. ✨

Another feature: the parser has two possible modes of operation. Normal mode activates when the -queue option is not on the command line invoking the parser. If -queue is there, the parser goes into incremental mode instead.

The big difference: a parser in normal mode immediately invokes the command prefix for each tag it encounters. In incremental mode, though, the parser generates a bunch of scripts that invoke the command prefix for groups of tags in the HTML string, and then stores those scripts in the specified queue. After that, it's the caller's job (read: your job 😭) to make sure those scripts in the queue actually get executed.

Note: the queue object you hand to the parser has to provide the same interface as the queue defined in tcllib -> struct. So like, every queue created via that tcllib module can be used here immediately. But the queue doesn't have to come from tcllib -> struct — as long as it provides the same interface, you're golden. 👑

In both modes the parser returns an empty string to the caller.

The -split option can be given to a parser in incremental mode to specify the size of the groups it creates. In other words, -split 5 means each of the generated scripts will invoke the command prefix for 5 consecutive tags in the HTML string. A parser in normal mode will ignore this option and its value entirely.

The -vroot option specifies a virtual root tag. A parser in normal mode will invoke the command prefix for it immediately before and after it processes the tags in the HTML, which simulates the HTML string being enclosed in a <vroot> </vroot> combination. In incremental mode, though, the parser can't provide the closing virtual root because it never knows when the input is complete. In that case, the first script generated by each invocation of the parser will contain an invocation of the command prefix for the virtual root as its first command.

The following options are available:

• -cmd cmd — the command prefix to invoke for every tag in the HTML string. Defaults to ::htmlparse::debugCallback. 🎀
• -vroot tag — the virtual root tag to add around the HTML in normal mode. In incremental mode it's the first tag in each chunk processed by the parser, but there will be no closing tags. Defaults to hmstart.
• -split n — the size of the groups produced by an incremental mode parser. Ignored when in normal mode. Defaults to 10. Values <= 0 are not allowed. 🙅‍♀️
• -incvar var — the name of the variable where any incomplete HTML gets stored. This makes the most sense for incremental mode. The parser will throw an error if it sees incomplete HTML and has nowhere to store it — which makes sense for normal mode. Only incomplete tags are detected, not missing tags. Optional, defaults to no variable at all.

💫 Interface to the command prefix 💫

In normal mode, the parser invokes the command prefix with four arguments appended. Peep ::htmlparse::debugCallback for a description of those.

In incremental mode, though, the generated scripts invoke the command prefix with five arguments appended. The last four of those are the same ones mentioned above. The first is a placeholder string (@win@) for a clientdata value that gets supplied later, during the actual execution of the generated scripts. This could be a tk window path, for example. It lets you preprocess HTML strings without committing them to a specific window, object, or whatever during parsing — that connection can be made later. It also means you can cache preprocessed HTML. 🔥 And of course, nothing stops you from just replacing the placeholder with an empty string. do what you want bestie.

---

🦋 ::htmlparse::debugCallback 🦋

::htmlparse::debugCallback ?clientdata? tag slash param textBehindTheTag

This is the standard callback the parser uses in ::htmlparse::parse when nobody specified one. It just dumps its arguments to stdout. simple, iconic, no notes. 💖

This callback works for both normal and incremental mode of the calling parser — in other words, it accepts four or five arguments. The last four are described below. The optional fifth holds the clientdata value passed to the callback by a parser in incremental mode. Every callback has to follow the signature of this command in the last four arguments, and callbacks used in incremental parsing have to follow this signature in the last five arguments. that's the rule fr fr.

• The first argument, clientdata, is optional and only present if this command is invoked by a parser in incremental mode. It contains whatever the user of this package wants. 🙈
• The second argument, tag, contains the name of the tag currently being processed by the parser.
• The third argument, slash, is either empty or contains a slash character. It lets the callback tell opening tags (slash is empty) apart from closing tags (slash contains a slash character).
• The fourth argument, param, contains the un-interpreted list of parameters to the tag.
• The fifth and last argument, textBehindTheTag, contains the text the parser found behind the tag named in tag.

---

🧚 ::htmlparse::mapEscapes 🧚

::htmlparse::mapEscapes html

This command takes an HTML string, substitutes all escape sequences with their actual characters, and then returns the resulting string. HTML strings that don't contain any escape sequences come back unchanged. ✨

---

👑 ::htmlparse::2tree 👑

::htmlparse::2tree html tree

This command is a wrapper around ::htmlparse::parse. It takes an HTML string (in html) and converts it into a tree containing the logical structure of the parsed document. The name of the tree is given to the command as its second argument (tree).

Important: the command does not generate the tree by itself — it expects the caller to have provided it with an existing and empty tree. It also expects that the specified tree object follows the same interface as the tree object in tcllib -> struct. It doesn't have to be from tcllib -> struct, but it must provide the same interface.

The internal callback does some basic checking of HTML validity and tries to recover from the most basic errors. The command returns the contents of its second argument. Side effects: the creation and manipulation of a tree object. 🌈

Each node in the generated tree represents one tag in the input. The name of the tag is stored in the attribute type of the node. Any html attributes coming with the tag are stored unmodified in the attribute data of the tag. In other words, the command does not parse html attributes into their names and values. it's giving "you deal with that."

If a tag contains text, its node will have children of type PCDATA containing this text. The text gets stored in the attribute data of these children.

---

💜 ::htmlparse::removeVisualFluff 💜

::htmlparse::removeVisualFluff tree

This command walks a tree as generated by ::htmlparse::2tree and removes all the nodes representing visual tags and not structural ones. The whole point is to make the tree easier to navigate without getting bogged down in visual information that's not relevant to your search. Its only argument is the name of the tree to cut down. ✂️

---

🍓 ::htmlparse::removeFormDefs 🍓

::htmlparse::removeFormDefs tree

Like ::htmlparse::removeVisualFluff, this command is here to cut down the size of the tree as generated by ::htmlparse::2tree. It removes all nodes representing forms and form elements. Its only argument is the name of the tree to cut down. 🫶

💖 BUGS, IDEAS, FEEDBACK 💖

okay bestie, real talk time 🫶 this doc, and the whole package it's describing, will undoubtedly contain bugs and other problems. like, absolutely. no cap. nothing is perfect and this is giving humble honesty.

so if you find something cursed, please report it in the category htmlparse of the Tcllib Trackers at [http://core.tcl.tk/tcllib/reportlist]. 🐛✨

and listen — if you've got ideas for enhancements for either the package and/or the documentation, report those too! we want them fr fr. big brain ideas welcome 👑

When you're proposing code changes, please provide unified diffs, i.e. the output of diff -u. that's the vibe. that's what we need to actually look at your change 💅

also — note further that attachments are strongly preferred over inlined patches. to attach something, go to the Edit form of the ticket immediately after its creation, and then use the left-most button in the secondary navigation bar. yes it's a specific little path, but that's the way, iykyk 🎀

---

🌈 SEE ALSO 🌈

• struct::tree — go read this one too, it's the companion vibe ✨

---

🌸 KEYWORDS 🌸

html, parsing, queue, tree 🦋

---

👑 CATEGORY 👑

Text processing 💜

---

and that's a wrap on htmlparse(3tcl) bestie!! 🎀✨ this whole thing is from tcllib, version 1.2.3, dated htmlparse(3tcl) — periodt, go parse some HTML and slay 💅💖