KiokuDB(3pm)

📦
version 0.57 · 3 versions
, in debian bookworm, debian trixie, ubuntu noble, ubuntu questing

🫶 KiokuDB(3pm) 🫶

okay bestie so this is the manpage for KiokuDB, from the User Contributed Perl Documentation, and it is a whole saga, so grab a snack 🍓

✨ NAME ✨

KiokuDB — Object Graph storage engine

🌸 VERSION 🌸

version 0.57

🎀 SYNOPSIS 🎀

use KiokuDB;

# use a DSN
my $d = KiokuDB->connect( $dsn, %args );

# or manually instantiate a backend
my $d = KiokuDB->new(
    backend => KiokuDB::Backend::Files->new(
        dir        => "/tmp/foo",
        serializer => "yaml",
    ),
);

# create a scope object
my $s = $d->new_scope;

# takes a snapshot of $some_object
my $uuid = $d->store($some_object);

# or with a custom ID:
$d->store( $id => $some_object ); # $id can be any string

# retrieve by ID
my $some_object = $d->lookup($uuid);

# some backends (like DBI) support simple searches
$d->search({ name => "foo" });

# others use GIN queries (DBI supports both)
$d->search($gin_query);

💜 DESCRIPTION 💜

KiokuDB is a Moose based frontend to various data stores, and honestly it lives somewhere in between Tangram and Pixie. it's giving "the middle child but make it iconic" ✨

Its whole purpose? Providing persistence for "regular" objects with as little effort as possible, without sacrificing control over how persistence is actually done — especially for the harder-to-serialize objects. lowkey a flex 💅

KiokuDB is also non-invasive: it does not use ties, "AUTOLOAD", proxy objects, "sv_magic" or any other type of trickery. no cap, it just behaves 🫶

Many features important for proper Perl space semantics are supported, including shared data, circular structures, weak references, tied structures, etc. all the chaotic little things 💖

KiokuDB is meant to solve two related persistence problems:

  • Transparent persistence — Store arbitrary objects without changing their class definitions or worrying about schema details, and without needing to conform to the limitations of a relational model. no schema drama, periodt.
  • Interoperability — Persisting arbitrary objects in a way that is compatible with existing data/code (for example interoperating with another app using CouchDB with JSPON semantics).

🌈 TUTORIAL 🌈

If you're new to KiokuDB check out KiokuDB::Tutorial. fr fr, start there bestie 🧚

👑 FUNDAMENTAL CONCEPTS 👑

In order to use any persistence framework it is important to understand what it does and how it does it. like, actually understand it, not just vibe with it 😌

Systems like Tangram or DBIx::Class generally require explicit meta data and use a schema, which makes them fairly predictable. bless them.

When using transparent systems like KiokuDB or Pixie it is more important to understand what's going on behind the scenes in order to avoid surprises and limitations. 👀

An architectural overview is available on the website: http://www.iinteractive.com/kiokudb/arch.html

The process is explained here and in the various component documentation in more detail.

🦋 Collapsing 🦋

When an object is stored using KiokuDB it's collapsed into an KiokuDB::Entry.

An entry is a simplified representation of the object, allowing the data to be saved in formats as simple as JSON. cute and portable, we love that ✨

References to other objects are converted to symbolic references in the entry, so objects can be saved independently of each other. independent queens 💅

The entries are given to the KiokuDB::Backend for actual storage.

Collapsing is explained in detail in KiokuDB::Collapser. The way an entry is created varies with the object's class.

🦋 Linking 🦋

When objects are loaded, entries are retrieved from the backend using their UIDs.

When a UID is already loaded (in the live object set of a KiokuDB instance, see KiokuDB::LiveObjects) the live object is used. This way references to shared objects are shared in memory regardless of the order the objects were stored or loaded. same object, same energy, no duplicates 🫶

This process is explained in detail in KiokuDB::Linker.

🌸 ROOT SET MEMBERSHIP 🌸

Any object that is passed to "store" or "insert" directly is implicitly considered a member of the root set.

This flag implies that the object is an identified resource and should not be garbage collected with any of the proposed garbage collection schemes. it's protected, your honor 👑

The root flag may be modified explicitly:

$kiokudb->set_root(@objects); # or unset_root

$kiokudb->update(@objects);

Lastly, root set membership may also be specified explicitly by the typemap.

A root set member must be explicitly removed using "delete" or by removing it from the root set. Only non-members of the root set will be purged with any garbage collection scheme. ⚠️ so like, if you thought GC was gonna sweep your roots away, no it will not. you have to do it yourself, bestie.

💅 TRANSACTIONS 💅

On supporting backends the "txn_do" method will execute a block and commit the transaction at its end.

Nesting of "txn_do" blocks is always supported, though rolling back a nested transaction may produce different results on different backends. ymmv depending on who you're talking to 😭

If the backend does not support transactions "txn_do" simply executes the code block normally. it just does the thing, no ceremony 💖

🔥 CONCURRENCY 🔥

Most transactional backends are also concurrent.

KiokuDB::Backend::BDB and KiokuDB::Backend::CouchDB default to serializable transaction isolation and do not suffer from deadlocks, but serialization errors may occur, aborting the transaction (in which case the transaction should be tried again). so like, just retry, it's fine, breathe 🧘

KiokuDB::Backend::Files provides good concurrency support but will only detect deadlocks on platforms which return "EDEADLK" from "flock". Directory::Transactional may provide alternative mechanisms in the future. future plans, we love to see it ✨

Concurrency support in KiokuDB::Backend::DBI depends on the database. SQLite defaults to serializable transaction isolation out of the box, whereas MySQL and PostgreSQL default to read committed.

Depending on your application read committed isolation may be sufficient, but due to the graph structure nature of the data repeatable reads or serializable level isolation is highly recommended. Read committed isolation generally works well when each row in the database is more or less independent of others, and various constraints ensure integrity. Unfortunately this is not the case with the graph layout. 💀 like, the graph said "no ❤️"

To enable stronger isolation guarantees see "Transactions" in KiokuDB::Backend::DBI for per-database pointers.

🌟 ATTRIBUTES 🌟

KiokuDB uses a number of delegates which do the actual work. they're the ones actually clocking in 💅

Of these only "backend" is required, the rest have default definitions.

Additional attributes that are not commonly used are listed in "INTERNAL ATTRIBUTES".

🌼 backend 🌼

This attribute is required. (yes, required required.)

This must be an object that does KiokuDB::Backend.

The backend handles storage and retrieval of entries. it's the one holding all the receipts 🎀

🌼 typemap 🌼

This is an instance KiokuDB::TypeMap.

The typemap contains entries which control how KiokuDB::Collapser and KiokuDB::Linker handle different types of objects.

🌼 allow_classes 🌼

An array references of extra classes to allow.

Objects blessed into these classes will be collapsed using KiokuDB::TypeMap::Entry::Naive.

🌼 allow_bases 🌼

An array references of extra base classes to allow.

Objects derived from these classes will be collapsed using KiokuDB::TypeMap::Entry::Naive.

🌼 allow_class_builders 🌼

If true adds KiokuDB::TypeMap::ClassBuilders to the merged typemap.

It's possible to provide a hash reference of options to give to "new" in KiokuDB::TypeMap::ClassBuilders.

🌼 check_class_versions 🌼

Controls whether or not the class versions of objects are checked on load.

Defaults to true.

🌼 class_version_table 🌼

A table of classes and versions that is passed to the default typemap entry for Moose/Class::MOP objects.

When a class version has changed between the time that an object was stored and the time it's being retrieved, the data must be converted. glow-up required ✨

See KiokuDB::TypeMap::Entry::MOP for more details.

(part 2 is coming, bestie — we are not done here 💅)

💖 METHODS 💖

connect $dsn, %args

okay so this is the DWIM wrapper for new — like, it just knows what you meant bestie 🫶

$dsn is some kind of backend, kinda the same way DBI dsns map to DBDs. iykyk.

A lil example DSN for u:

my $dir = KiokuDB->connect("bdb:dir=path/to/data/");

The backend moniker name gets extracted by splitting on the colon. Everything after that gets passed to new_from_dsn, which is documented in way more detail in KiokuDB::Backend. go read it, i'll wait 💅

Typically DSN arguments are separated by ;, and = separates keys from values. Arguments with no value are assumed to mean boolean truth — so like, jspon:dir=foo;pretty means dir => "foo", pretty => 1. BUT ⚠️ a backend may override the default parsing, so this is not guaranteed. no cap.

Extra arguments get passed to both the backend constructor and the KiokuDB constructor.

heads up bestie: if you need a typemap you still gotta pass it in yourself: KiokuDB->connect( $dsn, typemap => $typemap );

The DSN can also be a valid JSON string, taking one of these two forms:

dsn => '["dbi:SQLite:foo",{"schema":"MyApp::DB"}]'

dsn => '{"dsn":"dbi:SQLite:foo","schema":"MyApp::DB"}'

This is how you get more complicated arguments specified accurately, or arbitrary options specified when the backend has nonstandard DSN parsing. For instance, KiokuDB::Backend::DBI simply passes the string to DBI, so this is literally necessary in order to specify options on the command line. ✨

configure $config_file, %args

TODO 🙈 (yes really, that's the whole entry, she's a work in progress)

new %args

Creates a new directory object. 💫

See "ATTRIBUTES" for the deets.

new_scope

Creates a new object scope. Handled by live_objects.

The object scope artificially bumps up the reference count of objects to make sure they live at least as long as the scope does.

Why do we care?? Because this ensures weak references aren't deleted prematurely, and the object graph doesn't get corrupted — all without needing to create circular structures and clean up leaks by hand. iconic behavior honestly 👑

lookup @ids

Fetches the objects for the specified IDs from the live object set or from storage. 🦋

store @objects

store %objects

store_nonroot @objects

store_nonroot %objects

Recursively collapses @objects and inserts or updates the entries.

This performs a full update of every reachable object from @objects, snapshotting everything. the whole vibe, captured 📸

Strings found in the object list are assumed to be IDs for the following objects.

The nonroot variant will not mark the objects as members of the root set — which means they will be subject to garbage collection. 🗑️

update @objects

Performs a shallow update of @objects (referents are not updated).

it is an error to update an object not in the database. fr fr.

deep_update @objects

Updates @objects and all of the objects they reference. Every referenced object must already be in the database.

insert @objects

insert %objects

insert_nonroot @objects

insert_nonroot %objects

Inserts objects to the database. 🍓

it is an error to insert objects that are already in the database. All elements of @objects must be new — but their referents don't have to be.

@objects will be collapsed recursively, but the collapsing stops at known objects, which will not be updated.

The nonroot variant will not mark the objects as members of the root set — so yeah, they'll be subject to garbage collection. same energy as above 💀

delete @objects_or_ids

Deletes the specified objects from the store.

note: this can cause lookup errors if the object you're deleting is referred to by another object, because that link will be broken. 🫠

set_root @objects

unset_root @objects

Modify the root flag on the associated entries.

update must be called for the change to take effect. don't forget, bestie 💅

txn_do $code, %args

txn_do %args

scoped_txn $code

Executes $code within the scope of a transaction.

This requires that the backend supports transactions (KiokuDB::Backend::Role::TXN).

If the backend does not support transactions, the code block will simply be invoked. no drama, it just runs. 😌

Transactions may be nested. ✨

If the scope argument is true, an implicit call to new_scope will be made, keeping the scope for the duration of the transaction.

The return value is propagated from the code block, with handling of list/scalar/void context.

scoped_txn is like txn_do but sets scope to true.

txn_begin

txn_commit

txn_rollback

These methods simply call the corresponding methods on the backend.

Like txn_do, these methods are no-ops if the backend does not support transactions.

search \%proto

search @args

Searching requires a backend that supports querying. 🔍

The \%proto form is currently unspecified, but in the future it should provide a simple but consistent way of looking up objects by attributes.

The second form is backend-specific querying — for instance, Search::GIN::Query objects passed to KiokuDB::Backend::BDB::GIN, or the generic GIN backend wrapper KiokuDB::GIN.

Returns a Data::Stream::Bulk of the results. 🌈

root_set

Returns a Data::Stream::Bulk of all the root objects in the database. 👑

all_objects

Returns a Data::Stream::Bulk of all the objects in the database. literally everything 🌸

grep $filter

Returns a Data::Stream::Bulk of the objects in root_set filtered by $filter.

scan $callback

Iterates the root set, calling $callback for each object. 🦋

object_to_id

objects_to_ids

id_to_object

ids_to_objects

Delegates to KiokuDB::LiveObjects.

directory

Returns $self.

This is used when setting up KiokuDB::Role::API delegation chains. Calling directory on any level of delegator will always return the real KiokuDB instance, no matter how deep. 💅 deep delegation? handled. periodt.

✨ GLOBALS ✨

$SERIAL_IDS

If set at compile time, the default UUID generation role will use serial IDs instead of UUIDs.

This is useful for testing, since the same IDs will be issued each run — but it is utterly broken in the face of concurrency. 😭 do not do this in prod, i am begging you.

🧚 INTERNAL ATTRIBUTES 🧚

These attributes are documented for completeness and should typically not be needed. (but here they are anyway, because we're thorough like that 💜)

collapser

KiokuDB::Collapser

The collapser prepares objects for storage by creating KiokuDB::Entry objects to pass to the backend.

linker

KiokuDB::Linker

The linker links entries into functioning instances, loading necessary dependencies from the backend.

live_objects

KiokuDB::LiveObjects

The live object set keeps track of objects and entries for the linker and the resolver.

It also creates scope objects that help ensure objects don't garbage collect too early (new_scope in KiokuDB::LiveObjects, KiokuDB::LiveObjects::Scope), and transaction scope objects used by txn_do (KiokuDB::LiveObjects::TXNScope). 🫶

typemap_resolver

An instance of KiokuDB::TypeMap::Resolver. Handles actual lookup and compilation of typemap entries, using the user typemap.

🌈 SEE ALSO 🌈

Prior Art on the CPAN

  • Pixie
  • DBM::Deep
  • OOPS
  • Tangram
  • DBIx::Class
    • Polymorphic retrieval is possible with DBIx::Class::DynamicSubclass
  • Fey::ORM
  • MooseX::Storage

👑 VERSION CONTROL 👑

KiokuDB is maintained using Git. Information about the repository is available on <http://www.iinteractive.com/kiokudb/> 🌸

💖 AUTHOR 💖

Yuval Kogman <nothingmuch@woobling.org>

This software is copyright (c) 2014 by Yuval Kogman, Infinity Interactive.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

okay bestie that's the whole page 💅 this was KiokuDB(3pm), served up by perl v5.34.0, dated 2022-05-23. go forth and store some objects, you absolute icon 🎀✨🦋

📦 where this page comes from