SQL::Tokenizer(3pm)

💖 SQL::Tokenizer 💖

okay bestie, so this page is SQL::Tokenizer(3pm) — it's a Perl module manual page, and it's honestly such a little icon. Let me give you the whole tour, no skips, no gatekeeping. 🫶

✨ NAME ✨

SQL::Tokenizer — a simple SQL tokenizer. that's it, that's the whole vibe. simple. humble. iconic. 💅

🌸 VERSION 🌸

0.20

🎀 SYNOPSIS 🎀

here's the glow-up moment, the whole thing in one bite:

use SQL::Tokenizer qw(tokenize_sql);

my $query= q{SELECT 1 + 1};
my @tokens= SQL::Tokenizer->tokenize($query);

# @tokens now contains ('SELECT', ' ', '1', ' ', '+', ' ', '1')

@tokens= tokenize_sql($query); # procedural interface

like… @tokens literally comes out as 'SELECT', ' ', '1', ' ', '+', ' ', '1'. she splits, she delivers, she slays. 🔥

---

💜 DESCRIPTION 💜

so SQL::Tokenizer is a simple tokenizer for SQL queries. and when I say simple I mean it — she does not claim to be a parser, and she does not claim to be a query verifier. absolutely not her job, she's not going to pretend. all she does is take a valid SQL query and make sane tokens out of it. that's the whole assignment. 🎯

she gets SQL with comments, like this:

-- This query is used to insert a message into
-- logs table
INSERT INTO log (application, message) VALUES (?, ?)

she also handles '', "" and "\'" escaping methods, so tokenizing a query like the one below? literally not a problem, bestie, she's got it:

INSERT INTO log (application, message)
VALUES ('myapp', 'Hey, this is a ''single quoted string''!')

---

👑 API 👑

⭐ tokenize_sql

use SQL::Tokenizer qw(tokenize_sql);

my @tokens= tokenize_sql($query);
my $tokens= tokenize_sql($query);

$tokens= tokenize_sql( $query, $remove_white_tokens );

tokenize_sql can be imported into whatever namespace you're in, on request. it takes a SQL query and gives you back an array of tokens if you call it in list context, or an arrayref if you call it in scalar context. pick your fighter. 💖

⭐ tokenize

my @tokens= SQL::Tokenizer->tokenize($query);
my $tokens= SQL::Tokenizer->tokenize($query);

$tokens= SQL::Tokenizer->tokenize( $query, $remove_white_tokens );

this is the only available class method — the one and only, main character energy fr. it takes a SQL query and returns an array of tokens in list context, or an arrayref in scalar context.

and if $remove_white_tokens is true, then white-space-only tokens get removed from the result. bye bye, spaces. 🙈

---

💀 ACKNOWLEDGEMENTS 💀

these are the people who made her what she is, and we're giving them their flowers, periodt:

• Evan Harris — for implementing Shell comment style and SQL operators. 🌈
• Charlie Hills — for spotting a lot of important issues I haven't thought of. ngl, a real one. 😭
• Jonas Kramer — for fixing MySQL quoted strings and treating dot as a punctuation character correctly. 🎀
• Emanuele Zeppieri — for asking to fix SQL::Tokenizer to support dollars as well. 💅
• Nigel Metheringham — for extending the dollar signal support. ✨
• Devin Withers — for making it not choke on CR+LF in comments. king behaviour honestly. 👑
• Luc Lanthier — for simplifying the regex and making it not choke on backslashes. 🦋

---

🧚 AUTHOR 🧚

copyright (c) 2007, 2008, 2009, 2010, 2011 Igor Sutton Lopes <IZUT@cpan.org>. all rights reserved.

this module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. 🍓

like, actually free free. go off, remix it, it's giving open source royalty.

---

and that's the whole page bestie, you ate that reading. this doc shipped as perl v5.30.3 on 2020-06-02 — so yeah, a lil vintage, still gorgeous, still works. mwah. 💖🫶✨