CDB_File(3pm)

📦

version 1.05 · 8 versions

, in ubuntu stonking

💖 CDB_File(3pm) — bestie's guide to the CDB_File Perl extension 💖

(that's from User Contributed Perl Documentation, btw 💅)

✨ NAME ✨

CDB_File — a Perl extension for access to cdb databases. that's it, that's the whole vibe 🫶

🌸 SYNOPSIS 🌸

okay so here's the whole menu, no cap:

use CDB_File;
$c = tie(%h, 'CDB_File', 'file.cdb') or die "tie failed: $!\n";

# If accessing a utf8 stored CDB_File
$c = tie(%h, 'CDB_File', 'file.cdb', utf8 => 1) or die "tie failed: $!\n";

$fh = $c->handle;
sysseek $fh, $c->datapos, 0 or die ...;
sysread $fh, $x, $c->datalen;
undef $c;
untie %h;

$t = CDB_File->new('t.cdb', "t.$$") or die ...;
$t->insert('key', 'value');
$t->finish;

CDB_File::create %t, $file, "$file.$$";
=head1 DESCRIPTION

or… ✨

use CDB_File 'create';
create %t, $file, "$file.$$";

# If you want to store the data in utf8 mode.
create %t, $file, "$file.$$", utf8 => 1;

🦋 DESCRIPTION 🦋

So CDB_File is the module that gives Perl a whole interface to Dan Bernstein's cdb package. literally the bridge, the liaison, the main character connector 🔥

cdb is a fast, reliable, lightweight package for creating and reading constant databases.

fast. reliable. lightweight. constant. it's giving excellence, honestly.

📖 Reading from a cdb

So after the tie we did up in the synopsis, every time you poke at %h it's actually talking to the cdb file file.cdb, exactly how "tie" in perlfunc describes it. 🎀

Now for the ✨low level✨ stuff: you get three methods, handle, datapos, and datalen. and here's the thing bestie — to use them you have to remember the CDB_File object that the tie call handed back — that's $c in the example up top. The datapos and datalen methods give you the file offset position and the length, respectively, of the most recently visited key (like, say, when you did an exists). 💅

⚠️ heads up, this one matters: if you make an extra reference to the CDB_File object (yes, like $c in the example!), you must destroy it with undef before you call untie on the hash. that's what makes the object's DESTROY method actually fire. good news: perl -w will check this for you, so let the vibes-watcher watch 👀 — check perltie for the full lore.

🛠️ Creating a cdb

Making a cdb file is a three-step arc, like any good story 💖

call new CDB_File ($final, $tmp) — where $final is the name of the database you wanna create, and $tmp is the name of a temporary file that can be atomically renamed to $final.
call the insert method once for every single (key, value) pair.
call the finish method to complete the creation and the renaming of the cdb file. slay, done ✨

Alternative route, and it's faster: you can call the insert() method with multiple key/value pairs. why is that better? because there's significantly less crossing over the bridge from perl to C code, and every bridge crossing costs you. 🫠 One simple way to do it is to just pass in an entire hash:

$cdbmaker->insert(%hash);

And there's an even simpler interface for cdb file creation, courtesy of CDB_File::create %t, $final, $tmp. This makes a cdb file named $final holding all the contents of %t. Same deal as before: $tmp must name a temporary file that can be atomically renamed to $final. and yes, CDB_File::create can be imported. 🎀

🌈 UTF8 support.

okay story time, because this one's a whole saga 😭

When CDB_File was created back in 1997 (before even Perl 5.6, which is ancient), Perl SVs didn't really deal with UTF8 at all. To properly store mixed bytes and utf8 data in the file, you'd normally need to store a bit for each string that clarifies the encoding of the key / values. That bit would genuinely be useful, since Perl hash keys get downgraded to bytes whenever possible — that's how it normalizes hash key access regardless of encoding.

But here's the plot twist: the CDB_File format is used outside of Perl too, so it has to maintain file format compatibility with those other systems. 🫶 So instead, this module gives you a utf8 mode which has to be enabled at database generation and then again later at read. Keys will always be stored as UTF8 strings, which is the opposite of how Perl stores strings. That approach had to be taken to make sure no data corruption happened from accidentally downgraded SVs before they get stored, or on retrieval.

You enable utf8 mode by passing utf8 => 1 to new, tie, or create. Every returned SV while you're in this mode will be encoded in utf8. This feature is not available below 5.14 due to lack of Perl macro support, so if you're on something older… no utf8 mode for you, bestie 💀

NOTE: read/write of databases that aren't stored in utf8 mode will often be incompatible with any non-ascii data. ⚠️ not a drill.

💜 EXAMPLES 💜

these are all complete programs, so you can literally copy-paste and go 🏃♀️💨

1. Convert a Berkeley DB (B-tree) database to cdb format. 🌸

use CDB_File;
use DB_File;

tie %h, DB_File, $ARGV[0], O_RDONLY, undef, $DB_BTREE or
        die "$0: can't tie to $ARGV[0]: $!\n";

CDB_File::create %h, $ARGV[1], "$ARGV[1].$$" or
        die "$0: can't create cdb: $!\n";

2. Convert a flat file to cdb format. 🎀

In this example, the flat file consists of one key per line, separated by a colon from the value. Blank lines and lines beginning with # are skipped. iconic behaviour.

use CDB_File;

$cdb = new CDB_File("data.cdb", "data.$$") or
        die "$0: new CDB_File failed: $!\n";
while (<>) {
        next if /^$/ or /^#/;
        chop;
        ($k, $v) = split /:/, $_, 2;
        if (defined $v) {
                $cdb->insert($k, $v);
        } else {
                warn "bogus line: $_\n";
        }
}
$cdb->finish or die "$0: CDB_File finish failed: $!\n";

3. Perl version of cdbdump. 🦋

use CDB_File;

tie %data, 'CDB_File', $ARGV[0] or
        die "$0: can't tie to $ARGV[0]: $!\n";
while (($k, $v) = each %data) {
        print '+', length $k, ',', length $v, ":$k->$v\n";
}
print "\n";

4. For really enormous data values 🐘

you can use handle, datapos, and datalen, in combination with sysseek and sysread, to avoid reading the values into memory at all. Here is the script bun-x.pl, which can extract uncompressed files and directories from a bun file. it's giving efficient.

use CDB_File;

sub unnetstrings {
    my($netstrings) = @_;
    my @result;
    while ($netstrings =~ s/^([0-9]+)://) {
            push @result, substr($netstrings, 0, $1, '');
            $netstrings =~ s/^,//;
    }
    return @result;
}

my $chunk = 8192;

sub extract {
    my($file, $t, $b) = @_;
    my $head = $$b{"H$file"};
    my ($code, $type) = $head =~ m/^([0-9]+)(.)/;
    if ($type eq "/") {
            mkdir $file, 0777;
    } elsif ($type eq "_") {
            my ($total, $now, $got, $x);
            open OUT, ">$file" or die "open for output: $!\n";
            exists $$b{"D$code"} or die "corrupt bun file\n";
            my $fh = $t->handle;
            sysseek $fh, $t->datapos, 0;
            $total = $t->datalen;
            while ($total) {
                    $now = ($total > $chunk) ? $chunk : $total;
                    $got = sysread $fh, $x, $now;
                    if (not $got) { die "read error\n"; }
                    $total -= $got;
                    print OUT $x;
            }
            close OUT;
    } else {
            print STDERR "warning: skipping unknown file type\n";
    }
}

die "usage\n" if @ARGV != 1;

my (%b, $t);
$t = tie %b, 'CDB_File', $ARGV[0] or die "tie: $!\n";
map { extract $_, $t, \%b } unnetstrings $b{""};

5. Simulating an update 🫶

a cdb file is constant — that's the whole point — but you can simulate updating it in Perl. just know it's an expensive operation, because you have to create a brand new database, and copy into it everything that's unchanged from the old database. (compensation though: the update does not affect database readers. the old database stays available for them, right up till the moment the new one is finished. so thoughtful 💖)

use CDB_File;

$file = 'data.cdb';
$new = new CDB_File($file, "$file.$$") or
        die "$0: new CDB_File failed: $!\n";

# Add the new values; remember which keys we've seen.
while (<>) {
        chop;
        ($k, $v) = split;
        $new->insert($k, $v);
        $seen{$k} = 1;
}

# Add any old values that haven't been replaced.
tie %old, 'CDB_File', $file or die "$0: can't tie to $file: $!\n";
while (($k, $v) = each %old) {
        $new->insert($k, $v) unless $seen{$k};
}

$new->finish or die "$0: CDB_File finish failed: $!\n";

🔁 REPEATED KEYS 🔁

most users can ignore this section, fr fr. but if you're curious, stay.

A cdb file can contain repeated keys. If the insert method gets called more than once with the same key while you're creating a cdb file, that key will be repeated. 💀

Here's an example:

$cdb = new CDB_File ("$file.cdb", "$file.$$") or die ...;
$cdb->insert('cat', 'gato');
$cdb->insert('cat', 'chat');
$cdb->finish;

Normally, any attempt to access a key retrieves the first value stored under that key. So this code snippet always prints gato:

$catref = tie %catalogue, CDB_File, "$file.cdb" or die ...;
print "$catalogue{cat}";

However — and this is the slay part — all the usual ways of iterating over a hash (keys, values, and each) do the Right Thing, even when repeated keys are in the mix. This code snippet prints cat cat gato chat:

print join(' ', keys %catalogue, values %catalogue);

And these two both print cat:gato cat:chat, although the second one is more efficient. 👑

foreach $key (keys %catalogue) {
        print "$key:$catalogue{$key} ";
}

while (($key, $val) = each %catalogue) {
        print "$key:$val ";
}

The multi_get method retrieves all the values associated with a key. it returns a reference to an array containing all the values. this code prints gato chat:

print "@{$catref->multi_get('cat')}";

multi_get always returns an array reference. If the key wasn't found in the database, it'll be a reference to an empty array. So to test whether the key was found, you have to test the array, not the reference. this trips people up, do not be people 🙈

$x = $catref->multiget($key);
warn "$key not found\n" unless $x; # WRONG; message never printed
warn "$key not found\n" unless @$x; # Correct

The fetch_all method returns a hashref of all keys with the first value in the cdb. This is useful for quickly loading a cdb file where there's a 1:1 key mapping. In practice it proved to be about 400% faster then iterating a tied hash. four hundred percent. 🔥

# Slow
my %copy = %tied_cdb;

# Much Faster
my $copy_hashref = $catref->fetch_all();

🎀 RETURN VALUES 🎀

the routines tie, new, and finish return undef if the attempted operation failed — and $! holds the reason for the failure. so always check, bestie, or you'll be debugging in the dark 🌈

💀 DIAGNOSTICS 💀

okay bestie, listen — these are the fatal errors that can just happen to you. they will kill your program dead. if you wanna catch them instead of, like, dying, go read "eval" in perlfunc. that's the move. 🫶

Modification of a CDB_File attempted — you literally tried to modify a hash that's tied to a CDB_File. bestie. it's read-only energy. it said no. 😭
CDB database too large — you tried to make a cdb file bigger than 4 gigabytes. that's the limit, it's not personal, it's just math. ✋
[ Write to | Read of | Seek in ] CDB_File failed: <error string> — something went wrong at the OS level. now, if that error string is literally Protocol error, it means you did use CDB_File on something that is not a cdb file (awkward 💀). otherwise it's a genuinely serious OS problem — like, you ran out of disk space, kind of serious. go check your disks, queen. 🍓

🔥 PERFORMANCE 🔥

so sometimes you just need MAXIMUM performance out of a library, right? and the rumour on the street is that perl's tie() interface is slow. it's giving sluggish. and like — to get around that, you can use CDB_File in an object oriented fashion instead of going through tie(). no cap. 👑

  my $cdb = CDB_File->TIEHASH('/path/to/cdbfile.cdb');

  if ($cdb->EXISTS('key')) {
      print "Key is: ", $cdb->FETCH('key'), "\n";
  }

and if you want the whole tea on what methods tied hashes even have, go read perltie. it's all there, i promise. 💅

🧚 THE ALGORITHM 🧚

this algorithm is described over at http://cr.yp.to/cdb/cdb.txt — and it's small enough that it's literally included inline right here, just in case the internet loses the page. which. iconic of it, honestly. 🌈

🎀 A structure for constant databases 🎀

a cdb is an associative array, okay? it maps strings (keys) to strings (data).

a cdb contains 256 pointers to linearly probed open hash tables. those hash tables contain pointers to (key,data) pairs. and a cdb is stored in a single file on disk, like so:

    +----------------+---------+-------+-------+-----+---------+
    | p0 p1 ... p255 | records | hash0 | hash1 | ... | hash255 |
    +----------------+---------+-------+-------+-----+---------+

each of those 256 initial pointers states a position and a length. the position is the starting byte position of the hash table. the length is how many slots are in that hash table. simple. ✨

records are stored sequentially, with no special alignment. a record states a key length, a data length, the key, and the data. that's the whole vibe.

each hash table slot states a hash value and a byte position. if the byte position is 0, the slot is empty (it's on its own, respectfully). otherwise, that slot points to a record whose key has that hash value.

positions, lengths, and hash values are all 32-bit quantities, stored in little-endian form in 4 bytes. which means a cdb must fit into 4 gigabytes. that's just how it is. 🫠

a record is located like this, step by step:

Compute the hash value of the key in the record.
The hash value modulo 256 is the number of a hash table.
The hash value divided by 256, modulo the length of that table, is a slot number.
Probe that slot, then the next higher slot, and so on, until you find the record or run into an empty slot.

and the cdb hash function is h = ((h << 5) + h) ^ c, with a starting hash of 5381. memorise it. or don't. i'm not your mom. 💖

🙈 BUGS 🙈

the create() interface could be done with "TIEHASH". that's it. that's the note. someone's been thinking about this. 📝

👑 SEE ALSO 👑

cdb(3)

💜 AUTHOR 💜

Tim Goodwin, <tjg@star.le.ac.uk>. CDB_File began on 1997-01-08. like. the lore goes back that far. 🕰️

work provided through 2008 by Matt Sergeant, <matt@sergeant.org>

now maintained by Todd Rinaldo, <toddr@cpan.org>

and that's a wrap, bestie 🫶 this came at you from perl v5.42.3, dated 2026-08-20 — and yes, that's the CDB_File(3pm) page, all of it, every last crumb. go forth and cdb. 💅✨

📦 where this page comes from

ubuntu stonking: libcdb-file-perl 1.05-2build6 · Artistic-1.0-Perl OR GPL-1.0-or-later
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms