DBD::SQ...ualTable(3)

📦
version 1.760.0 · 15 versions
, in opensuse-leap 16.0

🦋 DBD::SQLite::VirtualTable(3) 🦋

okay so bestie, welcome to theUser Contributed Perl Documentation section 3 moment of your life 💅 — this is where we explain how to build SQLite virtual tables in pure Perl. Buckle up, it's a long one.

🌸 NAME 🌸

DBD::SQLite::VirtualTable — SQLite virtual tables implemented in Perl ✨

✨ SYNOPSIS ✨

# register the virtual table module within sqlite
$dbh->sqlite_create_module(mod_name => "DBD::SQLite::VirtualTable::Subclass");

# create a virtual table
$dbh->do("CREATE VIRTUAL TABLE vtbl USING mod_name(arg1, arg2, ...)")

# use it as any regular table
my $sth = $dbh->prepare("SELECT * FROM vtbl WHERE ...");
🫶 Note: VirtualTable subclasses or instances are not called directly from Perl code. Everything happens indirectly through SQL statements within SQLite. iykyk — you don't poke the puppeteer, you talk to the puppet. 🎭

💖 DESCRIPTION 💖

This module is an abstract class for implementing SQLite virtual tables, written in Perl. Fr. These tables look like regular tables, and you access them through regular SQL instructions and the regular DBI API — but the actual implementation is done through hidden calls to a Perl class. 😭 It's giving… Perl's tied variables, but at the SQLite level. Obsessed.

The current abstract class can't be used directly (it's abstract, bestie, it's a blueprint not a house 🏠), so the synopsis up there is just to give a general idea. The concrete, usable classes bundled with this distribution are:

  • DBD::SQLite::VirtualTable::FileContent — implements a virtual column that exposes file contents. This is especially useful in conjunction with a fulltext index; see DBD::SQLite::Fulltext_search. 🔥
  • DBD::SQLite::VirtualTable::PerlData — binds to a Perl array within the Perl program. This can be used for simple import/export operations, for debugging purposes, for joining data from different sources, etc.

Other Perl virtual tables may also be published separately on CPAN. 👑

The following chapters document the structure of the abstract class and explain how to write new subclasses; this is meant for module authors, not for end users. If you just need to use a virtual table module, refer to that module's documentation instead. No gatekeeping, just signposting 🫶

🦋 ARCHITECTURE 🦋

Classes

A virtual table module for SQLite is implemented through a pair of classes:

  • the table class implements methods for creating or connecting a virtual table, for destroying it, for opening new searches, etc.
  • the cursor class implements methods for performing a specific SQL statement

Methods

Most methods in both classes are not called directly from Perl code — instead, they're callbacks, called from the sqlite kernel. 💀 Following common Perl conventions, such methods have names in UPPERCASE. Yes, all caps. It's shouting, but make it ✨intentional✨.

👑 TABLE METHODS 👑

Class methods for registering the module

CREATE_MODULE

$class->CREATE_MODULE($sqlite_module_name);

Called when the client code invokes

$dbh->sqlite_create_module($sqlite_module_name => $class);

The default implementation is empty. (Yeah, it does nothing. That's the vibe. 🫧)

DESTROY_MODULE

$class->DESTROY_MODULE();

Called automatically when the database handle is disconnected. The default implementation is empty.

Class methods for creating a vtable instance

CREATE

$class->CREATE($dbh_ref, $module_name, $db_name, $vtab_name, @args);

Called when sqlite receives a statement

CREATE VIRTUAL TABLE $db_name.$vtab_name USING $module_name(@args)

The default implementation just calls "NEW".

CONNECT

$class->CONNECT($dbh_ref, $module_name, $db_name, $vtab_name, @args);

Called when attempting to access a virtual table that had been created during a previous database connection. The creation arguments were stored within the sqlite database and are passed again to the CONNECT method. 🕰️ (yes, it remembered. it's thoughtful like that.)

The default implementation just calls "NEW".

_PREPARE_SELF

okay this one has a leading underscore, so you know it's internal business 🙈

$class->_PREPARE_SELF($dbh_ref, $module_name, $db_name, $vtab_name, @args);

Prepares the datastructure for a virtual table instance. @args is just the collection of strings (comma-separated) that were given within the "CREATE VIRTUAL TABLE" statement; each subclass should decide what to do with this information.

The method parses @args to differentiate between options (strings of shape $key=$value or $key="$value", stored in "$self->{options}"), and columns (other @args, stored in "$self->{columns}"). It creates a hashref with the following fields:

  • "dbh_ref" — a weak reference to the $dbh database handle (see Scalar::Util for an explanation of weak references). 🪄
  • "module_name" — name of the module as declared to sqlite (not to be confounded with the Perl class name).
  • "db_name" — name of the database (usuallly 'main' or 'temp'), but it may also be an attached database.
  • "vtab_name" — name of the virtual table
  • "columns" — arrayref of column declarations
  • "options" — hashref of option declarations

This method should not be redefined, since it performs general work which is supposed to be useful for all subclasses. Instead, subclasses may override the "NEW" method. 💡 (don't touch the plumbing, just decorate the bathroom, bestie.)

NEW

$class->NEW($dbh_ref, $module_name, $db_name, $vtab_name, @args);

Instantiates a virtual table. 🐣

Instance methods called from the sqlite kernel

DROP

Called whenever a virtual table is destroyed from the database through the "DROP TABLE" SQL instruction.

Just after the DROP() call, the Perl instance will be destroyed (and will therefore automatically call the DESTROY() method if such a method is present). 💀

The default implementation for DROP is empty.

Note: this corresponds to the "xDestroy" method in the SQLite documentation; here it was not named "DESTROY", to avoid any confusion with the standard Perl method "DESTROY" for object destruction. naming crisis averted 🙏

DISCONNECT

Called for every virtual table just before the database handle is disconnected.

Just after the DISCONNECT() call, the Perl instance will be destroyed (and will therefore automatically call the DESTROY() method if such a method is present).

The default implementation for DISCONNECT is empty.

VTAB_TO_DECLARE

This method is called automatically just after "CREATE" or "CONNECT", to register the columns of the virtual table within the sqlite kernel. The method should return a string containing a SQL "CREATE TABLE" statement; but only the column declaration parts will be considered. Columns may be declared with the special keyword "HIDDEN", which means that they are used internally for the virtual table implementation, and are not visible to users — see <http://sqlite.org/c3ref/declare_vtab.html> and <http://www.sqlite.org/vtab.html#hiddencol> for detailed explanations. 🫣 (hidden columns are the backstage crew of this whole production.)

The default implementation returns:

CREATE TABLE $self->{vtab_name}(@{$self->{columns}})

BEST_INDEX

my $index_info = $vtab->BEST_INDEX($constraints, $order_by)

This is the most complex method to redefined in subclasses. 😮‍💨 Main character energy, honestly. This method will be called at the beginning of a new query on the virtual table; the job of the method is to assemble some information that will be used

  1. by the sqlite kernel to decide about the best search strategy
  2. by the cursor "FILTER" method to produce the desired subset of rows from the virtual table.

By calling this method, the SQLite core is saying to the virtual table that it needs to access some subset of the rows in the virtual table and it wants to know the most efficient way to do that access. The "BEST_INDEX" method replies with information that the SQLite core can then use to conduct an efficient search of the virtual table. 🧠✨

The method takes as input a list of $constraints and a list of $order_by instructions. It returns a hashref of indexing properties, described below; furthermore, the method also adds supplementary information within the input $constraints. Detailed explanations are given in <http://sqlite.org/vtab.html#xbestindex>.

Input constraints

Elements of the $constraints arrayref correspond to specific clauses of the "WHERE ..." part of the SQL query. Each constraint is a hashref with keys:

  • "col" — the integer index of the column on the left-hand side of the constraint
  • "op" — the comparison operator, expressed as string containing '=', '>', '>=', '<', '<=' or 'MATCH'.
  • "usable" — a boolean indicating if that constraint is usable; some constraints might not be usable because of the way tables are ordered in a join.

The $constraints arrayref is used both for input and for output. While iterating over the array, the method should add the following keys into usable constraints:

  • "argvIndex" — An index into the @values array that will be passed to the cursor's "FILTER" method. In other words, if the current constraint corresponds to the SQL fragment "WHERE ... AND foo < 123 ...", and the corresponding "argvIndex" takes value 5, this means that the "FILTER" method will receive 123 in $values[5]. 🎯
  • "omit" — A boolean telling to the sqlite core that it can safely omit to double check that constraint before returning the resultset to the calling program; this means that the FILTER method has fulfilled the filtering job on that constraint and there is no need to do any further checking.

The "BEST_INDEX" method will not necessarily receive all constraints from the SQL "WHERE" clause: for example a constraint like "col1 < col2 + col3" cannot be handled at this level. Furthemore, the "BEST_INDEX" might decide to ignore some of the received constraints. This is why a second pass over the results will be performed by the sqlite core. 🔁 (round two, fight!)

order_by input information

The $order_by arrayref corresponds to the "ORDER BY" clauses in the SQL query. Each entry is a hashref with keys:

  • "col" — the integer index of the column being ordered
  • "desc" — a boolean telling of the ordering is DESCending or ascending

This information could be used by some subclasses for optimizing the query strategfy; but usually the sqlite core will perform another sorting pass once all results are gathered. 🗂️

Hashref information returned by BEST_INDEX

The method should return a hashref with the following keys:

  • "idxNum" — An arbitrary integer associated with that index; this information will be passed back to "FILTER".
  • "idxStr" — An arbitrary str associated with that index; this information will be passed back to "FILTER".
  • "orderByConsumed" — A boolean telling the sqlite core if the $order_by information has been taken into account or not.
  • "estimatedCost" — A float that should be set to the estimated number of disk access operations required to execute this query against the virtual table. The SQLite core will often call BEST_INDEX multiple times with different constraints, obtain multiple cost estimates, then choose the query plan that gives the lowest estimate. 💸 (cheapest plan wins, capitalism but make it query optimisation.)
  • "estimatedRows" — An integer giving the estimated number of rows returned by that query.

OPEN

…and that's where part 1 runs out, bestie — OPEN is about to get its moment in part 2 🫶

Called to instantiate a new cursor. The default implementation appends ::Cursor to the current classname and calls NEW() within that cursor class.

Like, that's the whole job. Slap ::Cursor on the end of the classname, call NEW() in there, boom, cursor. 💅

_SQLITE_UPDATE

This is the dispatch method that implements the xUpdate() callback for virtual tables. And bestie, the default implementation already does the whole algorithm from <http://sqlite.org/vtab.html#xupdate> to figure out whether to call INSERT, DELETE or UPDATE — so there is literally no reason to override this method in subclasses. None. Don't do it. ✨

INSERT

  my $rowid = $vtab->INSERT($new_rowid, @values);

Override this one in your subclasses to implement insertion of a new row into the virtual table. The size of the @values array matches the number of columns you declared through VTAB_TO_DECLARE. The $new_rowid might be handed to you explicitly, or it might be undef — and in that case your method has to compute a new id and return it as the result of the method call. You got this 🫶

DELETE

  $vtab->INSERT($old_rowid);

Override this in your subclasses to implement deletion of a row from the virtual table.

okay yes, the original really does show INSERT in the DELETE synopsis — we are faithfully reproducing the manpage exactly as it is, no cap, no edits, no "fixing" it. 💀

UPDATE

  $vtab->UPDATE($old_rowid, $new_rowid, @values);

Override this in your subclasses to implement a row update within the virtual table. Usually $old_rowid equals $new_rowid, which is just a regular update — but the rowid can change, from a SQL statement like

  UPDATE table SET rowid=rowid+1 WHERE ...;

so yeah, keep that in mind 👑

FIND_FUNCTION

  $vtab->FIND_FUNCTION($num_args, $func_name);

When a function uses a column from a virtual table as its first argument, this method gets called to see if the virtual table wants to overload that function. The parameters are the number of arguments to the function, and the name of the function. If you don't want to overload anything, return false. If you do want to overload it, return a coderef to the function implementation.

Each virtual table keeps a cache of results from FIND_FUNCTION calls, so the method will only be called once for each pair ($num_args, $func_name). Efficient queen 🌸

BEGIN_TRANSACTION

Called to begin a transaction on the virtual table.

SYNC_TRANSACTION

Called to signal the start of a two-phase commit on the virtual table.

SYNC_TRANSACTION

Called to commit a virtual table transaction.

and yes, SYNC_TRANSACTION shows up twice in the original with two different descriptions. We're not merging them, we're not questioning them, we're just reporting the facts 🙈

ROLLBACK_TRANSACTION

Called to rollback a virtual table transaction.

RENAME

  $vtab->RENAME($new_name)

Called to rename a virtual table.

SAVEPOINT

  $vtab->SAVEPOINT($savepoint)

Called to signal the virtual table to save its current state at savepoint $savepoint (an integer).

ROLLBACK_TO

  $vtab->ROLLBACK_TO($savepoint)

Called to signal the virtual table to return to the state $savepoint. Heads up: this will invalidate all savepoints with values greater than $savepoint. 🫧

RELEASE

  $vtab->RELEASE($savepoint)

Called to invalidate all savepoints with values greater or equal to $savepoint.

🧰 Utility instance methods

Quick note, bestie: the methods in this section are all lower case, and that's on purpose — they are not called directly from the sqlite kernel. They're utility methods, meant to be called from the other methods described above. iykyk ✨

dbh

This method returns the database handle ($dbh) associated with the current virtual table.

🦋 CURSOR METHODS 🦋

Class methods

NEW

  my $cursor = $cursor_class->NEW($vtable, @args)

Instantiates a new cursor. The default implementation just returns a blessed hashref with keys vtable and args. Simple, iconic, does the job 💖

Instance methods

FILTER

  $cursor->FILTER($idxNum, $idxStr, @values);

This method begins a search of a virtual table.

The $idxNum and $idxStr arguments correspond to values returned by BEST_INDEX for the chosen index. The specific meanings of those values are unimportant to SQLite, as long as BEST_INDEX and FILTER agree on what that meaning is. (Communication is key, besties 🤝)

The BEST_INDEX method may have requested the values of certain expressions using the argvIndex values of the $constraints list. Those values are passed to FILTER through the @values array.

If the virtual table contains one or more rows that match the search criteria, then the cursor must be left pointing at the first row. Subsequent calls to EOF must return false. If there are no matching rows, then the cursor must be left in a state that will cause EOF to return true. The SQLite engine will use the COLUMN and ROWID methods to access that row content, and the NEXT method will be used to advance to the next row.

EOF

This method must return false if the cursor currently points to a valid row of data, or true otherwise. This method is called by the SQL engine immediately after each FILTER and NEXT invocation. Like, immediately immediately 👀

NEXT

This method advances the cursor to the next row of a result set initiated by FILTER. If the cursor is already pointing at the last row when this method is called, then the cursor no longer points to valid data, and a subsequent call to the EOF method must return true. If the cursor is successfully advanced to another row of content, then subsequent calls to EOF must return false.

COLUMN

  my $value = $cursor->COLUMN($idxCol);

The SQLite core invokes this method in order to find the value for the N-th column of the current row. N is zero-based, so the first column is numbered 0. Classic programmer brain 🧠✨

ROWID

  my $value = $cursor->ROWID;

Returns the rowid of the row that the cursor is currently pointing at.

👀 SEE ALSO

SQLite::VirtualTable is another module for virtual tables written in Perl, but it's designed for the reverse use case: instead of starting a Perl program and embedding the SQLite library into it, the intended use is to start an sqlite program and embed the Perl interpreter into it. Flip the script, fr fr 🦋

👑 AUTHOR

Laurent Dami <dami@cpan.org>

Copyright Laurent Dami, 2014.

Parts of the code are borrowed from SQLite::VirtualTable, copyright (C) 2006, 2009 by Qindel Formacion y Servicios, S. L.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

That's a wrap on the whole page, bestie — huge thank you to Laurent Dami (2014) and to Qindel Formacion y Servicios, S. L. for the borrowed bits 💖 This one's running on perl v5.42.0, dated 2024-10-19, and it's the DBD::SQLite::VirtualTable(3) page — go build something iconic with it 🌈⭐💅

📦 where this page comes from