cdb(3)

📦
version 0.81 · 4 versions
, in arch rolling, debian trixie, nixpkgs unstable, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 cdb(3) — Constant DataBase library 💖

🎀 NAME 🎀

cdb — Constant DataBase library. periodt, that's the name.

✨ SYNOPSYS ✨

#include <cdb.h>
cc ... -lcdb

💖 DESCRIPTION 💖

okay so bestie, cdb is a library to create and access Constant DataBase files. A file stores (key,value) pairs and is used to quickly find a value based on a given key. Cdb files are create-once files, that is, once created, file cannot be updated but recreated from scratch — this is why database is called constant. Cdb file is optimized for quick access. Format of such file described in cdb(5) manpage. This manual page corresponds to version 0.81 of tinycdb package.

Library defines two non-interlaced interfaces: for querying existing cdb file data (read-only mode) and for creating such a file (almost write-only). Strictly speaking, those modes allows very limited set of opposite operation as well (i.e. in query mode, it is possible to update key's value).

All routines in this library are thread-safe as no global data used, except of errno variable for error indication. No cap.

cdb datafiles may be moved between systems safely, since format does not depend on architecture. It's giving portable, fr fr.

🔍 QUERY MODE 🔍

There are two query modes available. First uses a structure that represents a cdb database, just like FILE structure in stdio library, and another works with plain filedescriptor. First mode is more sophisticated and flexible, and usually somewhat faster. It uses mmap(2) internally. This mode may look more "natural" or object-oriented compared to second one.

The following routines works with any mode:

  • unsigned cdb_unpack(buf)
    • const unsigned char buf[4];
    • helper routine to convert 32-bit integer from internal representation to machine format. May be used to handle application integers in a portable way. There is no error return.

🌸 Query Mode 1 🌸

All query operations in first mode deal with common data structure, struct cdb, associated with an open file descriptor. This structure is opaque to application. iykyk, it's the main character struct.

The following routines exists for accessing cdb database:

  • int cdb_init(cdbp, fd)
    • struct cdb *cdbp;
    • int fd;
    • okay bestie, this is your entry point. initializes structure given by cdbp pointer and associates it with opened file descriptor fd. Memory allocation for structure itself if needed and file open operation should be done by application. File fd should be opened at least read-only, and should be seekable. Routine returns 0 on success or negative value on error.
  • void cdb_free(cdbp)
    • struct cdb *cdbp;
    • frees internal resources held by structure. Note that this routine does not closes a file.
  • int cdb_fileno(cdbp)
    • const struct cdb *cdbp;
    • returns filedescriptor associated with cdb (as was passed to cdb_init()).
  • int cdb_read(cdbp, buf, len, pos) int cdb_readdata(cdbp, buf, len, pos) int cdb_readkey(cdbp, buf, len, pos)
    • const struct cdb *cdbp;
    • void *buf;
    • unsigned len;
    • unsigned pos;
    • reads a data from cdb file, starting at position pos of length len, placing result to buf. This routine may be used to get actual value found by cdb_find() or other routines that returns position and length of a data. Returns 0 on success or negative value on error. Routines cdb_readdata() and cdb_readkey() are shorthands to read current (after e.g. cdb_find()) data and key respectively, using cdb_read().
  • const void *cdb_get(cdbp, len, pos) const void *cdb_getdata(cdbp) const void *cdb_getkey(cdbp)
    • const struct cdb *cdbp;
    • unsigned len;
    • unsigned pos;
    • Internally, cdb library uses memory-mmaped region to access the on-disk database. cdb_get() allows one to access internal memory in a way similar to cdb_read() but without extra copying and buffer allocation. Returns pointer to actual data on success or NULL on error (position points to outside of the database). Routines cdb_getdata() and cdb_getkey() are shorthands to access current (after e.g. cdb_find()) data and key respectively, using cdb_get().
  • int cdb_find(cdbp, key, klen) unsigned cdb_datapos(cdbp) unsigned cdb_datalen(cdbp) unsigned cdb_keypos(cdbp) unsigned cdb_keylen(cdbp)
    • struct cdb *cdbp;
    • const void *key;
    • unsigned klen;
    • attempts to find a key given by (key,klen) parameters. If key exists in database, routine returns 1 and places position and length of value associated with this key to internal fields inside cdbp structure, to be accessible by cdb_datapos(cdbp) and cdb_datalen(cdbp) routines. If key is not in database, cdb_find() returns 0. On error, negative value is returned. Data pointers (available via cdb_datapos() and cdb_datalen()) gets updated only in case of successful search. Note that using cdb_find() it is possible to lookup only first record with a given key.
  • int cdb_findinit(cdbfp, cdbp, key, klen) int cdb_findnext(cdbfp)
    • struct cdb_find *cdbfp;
    • const struct cdb *cdbp;
    • const void *key;
    • unsigned klen;
    • sequential-find routines that used separate structure. It is possible to have more than one record with the same key in a database, and these routines allows one to enumerate all them. cdb_findinit() initializes search structure pointed to by cdbfp. It will return negative value on error or non-negative value on success. cdb_findnext() attempts to find next (first when called right after cdb_findinit()) matching key, setting value position and length in cdbfp structure. It will return positive value if given key was found, 0 if there is no more such key(s), or negative value on error. To access value position and length after successful call to cdb_findnext() (when it returned positive result), use cdb_datapos(cdbp) and cdb_datalen(cdbp) routines. It is error to continue using cdb_findnext() after it returned 0 or error condition (cdb_findinit() should be called again). Current data pointers (available via cdb_datapos() and cdb_datalen()) gets updated only on successful search.
  • void cdb_seqinit(cptr, cdbp) int cdb_seqnext(cptr, cdbp)
    • unsigned *cptr;
    • struct cdb *cdbp;
    • sequential enumeration of all records stored in cdb file. cdb_seqinit() initializes access current data pointer cptr to point before first record in a cdb file. cdb_seqnext() updates data pointers in cdbp to point to the next record and updates cptr, returning positive value on success, 0 on end of data condition and negative value on error. Current record will be available after successful operation using cdb_datapos(cdbp) and cdb_datalen(cdbp) (for the data) and cdb_keypos(cdbp) and cdb_keylen(cdbp) (for the key of the record). Data pointers gets updated only in case of successful operation.

🦋 Query Mode 2 🦋

In this mode, one need to open a cdb file using one of standard system calls (such as open(2)) to obtain a filedescriptor, and then pass that filedescriptor to cdb routines. Available methods to query a cdb database using only a filedescriptor include:

  • int cdb_seek(fd, key, klen, dlenp)
    • int fd;
    • const void *key;
    • unsigned klen;
    • unsigned *dlenp;
    • searches a cdb database (as pointed to by fd filedescriptor) for a key given by (key, klen), and positions file pointer to start of data associated with that key if found, so that next read operation from this filedescriptor will read that value, and places length of value, in bytes, to variable pointed to by dlenp. Returns positive value if operation was successful, 0 if key was not found, or negative value on error. To read the data from a cdb file, cdb_bread() routine below can be used.
  • int cdb_bread(fd, buf, len)
    • int fd;
    • void *buf;
    • int len;
    • reads data from a file (as pointed to by fd filedescriptor) and places len bytes from this file to a buffer pointed to by buf. Returns 0 if exactly len bytes was read, or a negative value in case of error or end-of-file. This routine ignores interrupt errors (EINTR). Sets errno variable to EIO in case of end-of-file condition (when there is less than len bytes available to read).

📝 Notes 📝

Note that value of any given key may be updated in place by another value of the same size, by writing to file at position found by cdb_find() or cdb_seek(). However one should be very careful when doing so, since write operation may not succeed in case of e.g. power failure, thus leaving corrupted data. When database is (re)created, one can guarantee that no incorrect data will be written to database, but not with inplace update. Note also that it is not possible to update any key or to change length of value. so like, respect the constant database vibe, bestie.

✨ CREATING MODE ✨

okay bestie so here's the tea on making a cdb file 🫖 a cdb database file should usually be created in two steps: first, a temporary file gets created and written to disk, and second, that temporary file gets renamed into its permanent place. the Unix rename(2) call is an atomic operation — it removes the destination file if there is one AND renames another file in, all in one single step 🎀 this way it is guaranteed that readers will not see an incomplete database. and to prevent multiple simultaneous updates, locking may also be used 🔒

All routines used to create a cdb database work with a struct cdb_make object that is opaque to the application. Your app may assume that struct cdb_make has at least the same member(s) as published in struct cdb above 👑

🌸 cdb_make_start

int cdb_make_start(cdbmp, fd)
   struct cdb_make *cdbmp;
   int fd;

initializes the structure to create a database 💅 file fd should be opened read-write and should be seekable. returns 0 on success or a negative value on error ✨

🌸 cdb_make_add

int cdb_make_add(cdbmp, key, klen, val, vlen)
   struct cdb_make *cdbmp;
   const void *key, *val;
   unsigned klen, vlen;

adds a record with key (key,klen) and value (val,vlen) to the database 💖 returns 0 on success or a negative value on error. heads up bestie: this routine does not check if the given key already exists, but cdb_find() will not see a second record with the same key. and ngl, it is not possible to continue building a database if cdb_make_add() returned an error indicator 😭

🌸 cdb_make_finish

int cdb_make_finish(cdbmp)
   struct cdb_make *cdbmp;

finalizes the database file, constructing all the needed indexes, and frees the memory structures 🧚 it does not close the filedescriptor. returns 0 on success or a negative value on error ✨

🌸 cdb_make_exists

int cdb_make_exists(cdbmp, key, klen)
   struct cdb_make *cdbmp;
   const void *key;
   unsigned klen;

this routine attempts to find the key given by (key,klen) in a not-yet-complete database 🔍 it may significantly slow down the whole process, and currently it flushes the internal buffer to disk on every call with a key whose hash value already exists in the db 😮‍💨 returns 0 if such a key doesn't exist, 1 if it does, or a negative value on error. also note: the database file should be opened read-write (not write-only) to use this routine 👀 and if cdb_make_exists() returned an error, it may be not possible to continue constructing the database 💀

🌸 cdb_make_find

int cdb_make_find(cdbmp, key, klen, mode)
   struct cdb_make *cdbmp;
   const void *key;
   unsigned klen;
   int mode;

this routine attempts to find the key given by (key,klen) in the database being created 🔎 if the given key already exists, then an action specified by mode will be performed:

  • CDB_FIND — checks whenever the given record is already in the database ✅
  • CDB_FIND_REMOVE — removes all matching records by re-writing the database file accordingly 🗑️
  • CDB_FIND_FILL0 — fills all matching records with zeros and removes them from the index so that the records in question will not be findable with cdb_find() 🌫️ this is faster than CDB_FIND_REMOVE, but it leaves zero "gaps" in the database. lastly inserted records, if matched, are always removed 👑

If no matching keys were found, the routine returns 0. in case at least one record has been found/removed, a positive value will be returned. on error, a negative value will be returned and errno will be set appropriately. and when cdb_make_find() returned a negative value in case of error, it is not possible to continue constructing the database 😭

cdb_make_exists() is the same as calling cdb_make_find() with mode set to CDB_FIND — iykyk 💅

🌸 cdb_make_put

int cdb_make_put(cdbmp, key, klen, val, vlen, mode)
   struct cdb_make *cdbmp;
   const void *key, *val;
   unsigned klen, vlen;
   int mode;

this is a somewhat combined cdb_make_exists() and cdb_make_add() situation 🫶 the mode argument controls how repeated (already existing) keys will be treated:

  • CDB_PUT_ADD — no duplicate checking will be performed. this mode is the same as what the cdb_make_add() routine does 🍓
  • CDB_PUT_REPLACE — if the key already exists, it will be removed from the database before adding the new key,value pair. this requires moving data in the file, and can be quite slow if the file is large 🐌 all matching old records will be removed this way. this is the same as calling cdb_make_find() with the CDB_FIND_REMOVE mode argument followed by calling cdb_make_add() 💖
  • CDB_PUT_REPLACE0 — if the key already exists and it isn't the last record in the file, the old record will be zeroed out before adding the new key,value pair. this is a lot faster than CDB_PUT_REPLACE, but some extra data will still be present in the file 👀 the data — that old record — will not be accessible by normal searches, but will appear in sequential database traversal. this is the same as calling cdb_make_find() with the CDB_FIND_FILL0 mode argument followed by cdb_make_add() ✨
  • CDB_PUT_INSERT — add the key,value pair only if such a key does not exist in the database. note that since a query (see query mode above) will find the first added record, this mode is somewhat useless (but it allows one to reduce database size in case of repeated keys) 🙈 this is the same as calling cdb_make_exists(), followed by cdb_make_add() if the key was not found 💅
  • CDB_PUT_WARN — add the key,value pair unconditionally, but also check if this key already exists. this is equivalent to cdb_make_exists() to check existence of the given key, unconditionally followed by cdb_make_add() ⭐

If any error occurred during operations, the routine will return a negative integer and will set the global variable errno to indicate the reason of failure. in case of a successful operation and no duplicates found, the routine will return 0. if any duplicates have been found or removed (which, in case of CDB_PUT_INSERT mode, indicates that the new record was not added), the routine will return a positive value. and if an error occurred and cdb_make_put() returned a negative error, it is not possible to continue the database construction process 💀

as with cdb_make_exists() and cdb_make_find(), usage of this routine with any mode but CDB_PUT_ADD can significantly slow down database creation, especially when mode is equal to CDB_PUT_REPLACE0 😮‍💨

🌸 cdb_pack

void cdb_pack(num, buf)
   unsigned num;
   unsigned char buf[4];

a helper routine that's used internally to convert the machine integer n to the internal form that gets stored in the datafile 📦 a 32-bit integer is stored in 4 bytes in network byte order. it may be used to handle application data. there is no error return ✨

🌸 cdb_hash

unsigned cdb_hash(buf, len)
   const void *buf;
   unsigned len;

a helper routine that calculates the cdb hash value of the given bytes 🔐 the CDB hash function is

hash[n] = (hash[n-1] + (hash[n-1] << 5)) ^ buf[n]

starting with

hash[-1] = 5381

💀 ERRORS 💀

the cdb library may set errno to the following on error:

  • EPROTO — database file is corrupted in some way 😭
  • EINVAL — the same as EPROTO above if the system lacks the EPROTO constant 🙃
  • EINVAL — flag argument for cdb_make_put() is invalid 🚫
  • EEXIST — flag argument for cdb_make_put() is CDB_PUT_INSERT, and the key already exists 👀
  • ENOMEM — not enough memory to complete the operation (cdb_make_finish and cdb_make_add) 🧠
  • EIO — set by cdb_bread and cdb_seek if a cdb file is shorter than expected or corrupted in some other way 💥

🌸 EXAMPLES 🌸

quick note bestie: in all the examples down here, error checking is not shown for brewity. like, we're keeping it cute and short. your real code should be checking stuff, fr fr.

💖 Query Mode

int fd;
struct cdb cdb;
char *key, *data;
unsigned keylen, datalen;

/* opening the database */
fd = open(filename, O_RDONLY);
cdb_init(&cdb, fd);
/* initialize key and keylen here */

/* single-record search. */
if (cdb_find(&cdb, key, keylen) > 0) {
  datalen = cdb_datalen(&cdb);
  data = malloc(datalen + 1);
  cdb_read(&cdb, data, datalen, cdb_datapos(&cdb));
  data[datalen] = '\0';
  printf("key=%s data=%s\n", key, data);
  free(data);
}
else
  printf("key=%s not found\n", key);

/* multiple record search */
struct cdb_find cdbf;
int n;
cdb_findinit(&cdbf, &cdb, key, keylen);
n = 0;
while(cdb_findnext(&cdbf) > 0) {
  datalen = cdb_datalen(&cdb);
  data = malloc(datalen + 1);
  cdb_read(&cdb, data, datalen, cdb_datapos(&cdb));
  data[datalen] = '\0';
  printf("key=%s data=%s\n", key, data);
  free(data);
  ++n;
}
printf("key=%s %d records found\n", n);

/* sequential database access */
unsigned pos;
int n;
cdb_seqinit(&pos, &cdb);
n = 0;
while(cdb_seqnext(&pos, &cdb) > 0) {
  keylen = cdb_keylen(&cdb);
  key = malloc(keylen + 1);
  cdb_read(&cdb, key, keylen, cdb_keypos(&cdb));
  key[keylen] = '\0';
  datalen = cdb_datalen(&cdb);
  data = malloc(datalen + 1);
  cdb_read(&cdb, data, datalen, cdb_datapos(&cdb));
  data[datalen] = '\0';
  ++n;
  printf("record %n: key=%s data=%s\n", n, key, data);
  free(data); free(key);
}
printf("total records found: %d\n", n);

/* close the database */
cdb_free(&cdb);
close(fd);

/* simplistic query mode */
fd = open(filename, O_RDONLY);
if (cdb_seek(fd, key, keylen, &datalen) > 0) {
  data = malloc(datalen + 1);
  cdb_bread(fd, data, datalen);
  data[datalen] = '\0';
  printf("key=%s data=%s\n", key, data);
}
else
  printf("key=%s not found\n", key);
close(fd);

okay so what's happening here: you got your single-record search (the classic one-and-done cdb_find vibe), then the multiple record search where cdb_findinit + cdb_findnext just keep handing you records like a bestie handing you snacks, then sequential database access where cdb_seqinit and cdb_seqnext walk the whole thing top to bottom, and finally the simplistic query mode — the low-effort, high-reward cdb_seek + cdb_bread combo that skips the whole struct cdb situation entirely. pick your fighter ✨

🎀 Create Mode

int fd;
struct cdb_make cdbm;
char *key, *data;
unsigned keylen, datalen;

/* initialize the database */
fd = open(filename, O_RDWR|O_CREAT|O_TRUNC, 0644);
cdb_make_start(&cdbm, fd);

while(have_more_data()) {
  /* initialize key and data */
  if (cdb_make_exists(&cdbm, key, keylen) == 0)
    cdb_make_add(&cdbm, key, keylen, data, datalen);
  /* or use cdb_make_put() with appropriate flags */
}

/* finalize and close the database */
cdb_make_finish(&cdbm);
close(fd);

the whole arc is: open the file with O_RDWR|O_CREAT|O_TRUNC and mode 0644 💅, kick it off with cdb_make_start, dump your records in while you still have data (peep the cdb_make_exists check so you don't double-add, or just use cdb_make_put() with the flags you want), then cdb_make_finish and close(fd) — and boom, database built. 🦋

👀 SEE ALSO 👀

💜 AUTHOR 💜

the tinycdb package was written by Michael Tokarev <mjt+cdb@corpit.ru>, and it's based on ideas and shares file format with the original cdb library by Dan Bernstein. giving credit where credit is due, periodt. 👑

⭐ LICENSE ⭐

tinycdb is licensed under the MIT license. slay, go build something. 🌈

okay that's the whole page bestie, we did it 🥳 this was from Jun 2006 and it's the cdb(3) page from the tinycdb package — go forth and query responsibly, mwah 💋🫶

📦 where this page comes from