JSON::Tokenize(3)

💖 NAME 💖

okay so welcome to the JSON::Tokenize(3) page, straight out of the User Contributed Perl Documentation aisle, bestie 🫶

JSON::Tokenize — tokenize JSON. that's the name, that's the vibe, no cap ✨

---

✨ SYNOPSIS ✨

bestie, here's the whole demo, and yes, we're printing tokens recursively like it's a little family tree 🌳

           use JSON::Tokenize ':all';

           my $input = '{"tuttie":["fruity", true, 100]}';
           my $token = tokenize_json ($input);
           print_tokens ($token, 0);

           sub print_tokens
           {
               my ($token, $depth) = @_;
               while ($token) {
                   my $start = tokenize_start ($token);
                   my $end = tokenize_end ($token);
                   my $type = tokenize_type ($token);
                   print "   " x $depth;
                   my $value = substr ($input, $start, $end - $start);
                   print "'$value' has type '$type'.\n";
                   my $child = tokenize_child ($token);
                   if ($child) {
                       print_tokens ($child, $depth+1);
                   }
                   my $next = tokenize_next ($token);
                   $token = $next;
               }
           }

and then you get this exact output, iconic:

           '{"tuttie":["fruity", true, 100]}' has type 'object'.
              '"tuttie"' has type 'string'.
              ':' has type 'colon'.
              '["fruity", true, 100]' has type 'array'.
                 '"fruity"' has type 'string'.
                 ',' has type 'comma'.
                 'true' has type 'literal'.
                 ',' has type 'comma'.
                 '100' has type 'number'.

---

🦋 VERSION 🦋

this page documents version 0.62 of JSON::Tokenize, and it corresponds to git commit d04630086f6c92fea720cba4568faa0cbbdde5a6 — the URL is https://github.com/benkasminbullock/JSON-Parse/commit/d04630086f6c92fea720cba4568faa0cbbdde5a6 — and it was released on Sat Jul 16 08:23:13 2022 +0900 ⭐

---

🌸 DESCRIPTION 🌸

okay so JSON::Tokenize is a module for tokenizing a JSON string. and “tokenizing” means breaking the string into individual tokens, without creating any Perl structures 💅

It uses the same underlying code as JSON::Parse. tokenizing can be used for tasks such as picking out or searching through parts of a large JSON structure without storing each part of the entire structure in memory 🧠✨

warning bestie 🚨 this module is an experimental part of JSON::Parse, and its interface is likely to change. the tokenizing functions are currently written in a very primitive way. so like, don't build your whole personality on the API staying put, okay? 💀

---

💅 FUNCTIONS 💅

🧚 tokenize_child

my $child = tokenize_child ($child);

walk the tree of tokens, bestie 🌳 this is how you get the child token.

🍓 tokenize_end

my $end = tokenize_end ($token);

grab the end of the token as a byte offset from the start of the string. and important: this is a byte offset, not a character offset 😭

🎀 tokenize_json

my $token = tokenize_json ($json);

this is the signature, and the original page literally doesn't add extra prose here — it's just the call, no cap 🫶

🦋 tokenize_next

my $next = tokenize_next ($token);

walk the tree of tokens — that's the note, and it's giving traversal energy 🌈

🧚 tokenize_start

my $start = tokenize_start ($token);

get the start of the token as a byte offset from the start of the string. again, byte offset, not character offset — don't forget it, bestie 💀

🌸 tokenize_text

my $text = tokenize_text ($json, $token);

given a token $token from this parsing and the JSON in $json, return the text which corresponds to the token. it's a convenience function written in Perl, and it uses tokenize_start, tokenize_end, and substr to get the string from $json ✨

💜 tokenize_type

my $type = tokenize_type ($token);

get the type of the token as a string. the possible return values are literally these:

• "array"
• "initial state"
• "invalid"
• "literal"
• "number"
• "object"
• "string"
• "unicode escape"

so yeah, those are the type strings you might see 👑

---

💌 AUTHOR 💌

Ben Bullock, <bkb@cpan.org> — that's the author, bestie 💖

---

🎀 COPYRIGHT & LICENCE 🎀

this package and associated files are copyright (C) 2016-2022 Ben Bullock.

you can use, copy, modify and redistribute this package and associated files under the Perl Artistic Licence or the GNU General Public Licence 🫶

---

and that's the page! shoutout to perl v5.42.0, the 2022-07-16 date, and JSON::Tokenize(3) — this bestie is signing off, mwah 💋