MKDoc:...kenizer(3pm)

💖 MKDoc::XML::Tokenizer — the perl one, from User Contributed Perl Documentation 💖

okay bestie, gather round, because we're about to talk about tokenizing XML the REX way and honestly? it's giving genius. this little cutie lives in the MKDoc::XML::Tokenizer module and it is about to become your bestie. 🦋

---

✨ NAME ✨

MKDoc::XML::Tokenizer — Tokenize XML the REX way 💅

---

🌸 SYNOPSIS 🌸

here's the vibe, no cap:

my $tokens = MKDoc::XML::Tokenizer->process_data ($some_xml);
foreach my $token (@{$tokens})
{
    print "'" . $token->as_string() . "' is text\n" if (defined $token->text());
    print "'" . $token->as_string() . "' is a self closing tag\n" if (defined $token->tag_self_close());
    print "'" . $token->as_string() . "' is an opening tag\n" if (defined $token->tag_open());
    print "'" . $token->as_string() . "' is a closing tag\n" if (defined $token->tag_close());
    print "'" . $token->as_string() . "' is a processing instruction\n" if (defined $token->pi());
    print "'" . $token->as_string() . "' is a declaration\n" if (defined $token->declaration());
    print "'" . $token->as_string() . "' is a comment\n" if (defined $token->comment());
    print "'" . $token->as_string() . "' is a tag\n" if (defined $token->tag());
    print "'" . $token->as_string() . "' is a pseudo-tag (NOT text and NOT tag)\n" if (defined $token->pseudotag());
    print "'" . $token->as_string() . "' is a leaf token (NOT opening tag)\n" if (defined $token->leaf());
}

look at her go. 😭 she's so thorough.

---

🔥 SUMMARY 🔥

so MKDoc::XML::Tokenizer is the module that uses Robert D. Cameron's REX technique to parse XML. and when I say "uses a regex", I mean it uses a regex. like, one (1) regex. ignore the carriage returns btw, they're just line-wrap ✨aesthetic✨:

[^<]+|<(?:!(?:--(?:[^-]*-(?:[^-][^-]*-)*->?)?|\[CDATA\[(?:[^\]]*](?:[^\]]+])
*]+(?:[^\]>][^\]]*](?:[^\]]+])*]+)*>)?|DOCTYPE(?:[ \n\t\r]+(?:[A-Za-z_:]|[^\
x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*(?:[ \n\t\r]+(?:(?:[A-Za-z_:]|[^\
x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*|"[^"]*"|'[^']*'))*(?:[ \n\t\r]+)
?(?:\[(?:<(?:!(?:--[^-]*-(?:[^-][^-]*-)*->|[^-](?:[^\]"'><]+|"[^"]*"|'[^']*'
)*>)|\?(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*(?:\?>|[\
n\r\t ][^?]*\?+(?:[^>?][^?]*\?+)*>))|%(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0
-9_:.-]|[^\x00-\x7F])*;|[ \n\t\r]+)*](?:[ \n\t\r]+)?)?>?)?)?|\?(?:(?:[A-Za-z
_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*(?:\?>|[\n\r\t ][^?]*\?+(?
:[^>?][^?]*\?+)*>)?)?|/(?:(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x
00-\x7F])*(?:[ \n\t\r]+)?>?)?|(?:(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.
-]|[^\x00-\x7F])*(?:[ \n\t\r]+(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|
[^\x00-\x7F])*(?:[ \n\t\r]+)?=(?:[ \n\t\r]+)?(?:"[^<"]*"|'[^<']*'))*(?:[ \n\
t\r]+)?/?>?)?)

that's right bestie. one big regex. and it works rather well. 💅 periodt. she ate and left no crumbs.

---

😭 DISCLAIMER 😭

okay so warning bestie: this module does low level XML manipulation. she will somehow parse even broken XML and try to do something with it. 😳

do not use it unless you know what you're doing. fr fr. this is not the module to just vibe with. 🙈

---

👑 API 👑

my $tokens = MKDoc::XML::Tokenizer->process_data ($some_xml);

splits $some_xml into a whole list of MKDoc::XML::Token objects, and hands you back an array reference to that list of tokens. iconic behaviour honestly. ✨

my $tokens = MKDoc::XML::Tokenizer->process_file ('/some/file.xml');

literally the same as MKDoc::XML::Tokenizer->process_data ($some_xml) except it reads $some_xml from '/some/file.xml'. same energy, different source. 🫶

---

🎀 NOTES 🎀

MKDoc::XML::Tokenizer works with MKDoc::XML::Token, which is the move when you don't need to build a full tree. 🌸

but if you do need to build a tree? bestie, go look at MKDoc::XML::TreeBuilder instead. iykyk. 🦋

---

💜 AUTHOR 💜

Copyright 2003 - MKDoc Holdings Ltd.

Author: Jean-Michel Hiver

This module is free software and is distributed under the same license as Perl itself. Use it at your own risk. 💅

---

🧚 SEE ALSO 🧚

• MKDoc::XML::Token
• MKDoc::XML::TreeBuilder

---

and that's the whole page, bestie!! 🌈 this was perl v5.36.0, dated 2022-10-13, from the MKDoc:...kenizer(3pm) of it all — stan a regex queen. go tokenize something cute today. 💖✨🎀