SQL::Sp...kenizer(3)

🌸 NAME 🌸

SQL::SplitStatement::Tokenizer — a simple SQL tokenizer. 💖 And yes bestie, this is the SQL::Sp...kenizer(3) page from User Contributed Perl Documentation.

---

✨ SYNOPSIS ✨

Okay so you import tokenize_sql like this:

use SQL::SplitStatement::Tokenizer qw(tokenize_sql);

my $query= q{SELECT 1 + 1};
my @tokens= tokenize_sql($query);

# @tokens now contains ('SELECT', ' ', '1', ' ', '+', ' ', '1')

That comment is literally the receipt for your tokens, no cap. 🧾✨

💖 DESCRIPTION 💖

SQL::SplitStatement::Tokenizer is a simple tokenizer for SQL queries. It does not claim to be a parser or query verifier — it just creates sane tokens from a valid SQL query. Periodt. 💅

It supports SQL with comments, like this:

-- This query is used to insert a message into
-- logs table
INSERT INTO log (application, message) VALUES (?, ?)

It also supports '', "" and "\'" escaping methods, so tokenizing queries like the one below should not be a problem:

INSERT INTO log (application, message)
VALUES ('myapp', 'Hey, this is a ''single quoted string''!')

🧚 API 🧚

🎀 tokenize_sql 🎀

use SQL::SplitStatement::Tokenizer qw(tokenize_sql);

my @tokens = tokenize_sql($query);
my $tokens = tokenize_sql($query);

$tokens = tokenize_sql( $query, $remove_white_tokens );

tokenize_sql can be imported to current namespace on request. It receives a SQL query, and returns an array of tokens if called in list context, or an arrayref if called in scalar context. 🫶

If $remove_white_tokens is true, white spaces only tokens will be removed from result. Bye-bye, space-only tokens. 👋

🫶 ACKNOWLEDGEMENTS 🫶

• Igor Sutton Lopes for writing SQL::Tokenizer, which this was forked from. 🧬
• Evan Harris, for implementing Shell comment style and SQL operators. 🐚
• Charlie Hills, for spotting a lot of important issues I haven't thought. 👀
• Jonas Kramer, for fixing MySQL quoted strings and treating dot as punctuation character correctly. 🐬
• Emanuele Zeppieri, for asking to fix SQL::Tokenizer to support dollars as well. 💵
• Nigel Metheringham, for extending the dollar signal support. 💲
• Devin Withers, for making it not choke on CR+LF in comments. 💻
• Luc Lanthier, for simplifying the regex and making it not choke on backslashes. 🪄

👑 AUTHOR 👑

Copyright (c) 2007, 2008, 2009, 2010, 2011 Igor Sutton Lopes "<IZUT@cpan.org>". All rights reserved.

Copyright (c) 2021 Veesh Goldman "<veesh@cpan.org>"

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

---

mwah 💋 — this SQL::Sp...kenizer(3) page comes from perl v5.42.0 and is dated 2021-06-06. slay and happy tokenizing! 🎀