dirfile-encoding(5)

📦 version 0.11.0, in debian bookworm, debian trixie, fedora 43, fedora 44, nixos 26.05, nixpkgs unstable, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 dirfile-encoding 💖 — the manual, but make it make sense

okay bestie so this is dirfile-encoding(5), from the DATA FORMATS crew, and we're about to talk encoding schemes for your dirfile database. sit down, hydrate, let's go 🫶

✨ NAME ✨

dirfile-encoding — dirfile database encoding schemes, aka the glow-up your binary files can get 💅

🌸 DESCRIPTION 🌸

so here's the tea, fr fr. The Dirfile Standards say that RAW fields defined in the database come with binary files that hold the field data in whatever simple data type was specified. that's just the baseline, the default fit, no notes.

but sometimes it's lowkey advantageous to convert those binary files in the database into a more convenient form. and that, bestie, is what we call encoding the binary file into the alternate form. the most common use-case for encoding a binary file? compressing it to save disk space, obviously. iconic behaviour.

important, don't skip this: only data is modified by an encoding scheme. database metadata is never encoded. periodt.

🦋 the whole optional thing

support for encoding schemes is optional. an implementation doesn't have to support any particular encoding scheme, and it might only support certain operations with it. but — and this is the vibe check — it should expect to encounter unknown encoding schemes and fail gracefully in those situations. no crashing, no drama, just a polite little "don't know her" 💀

🎀 interoperability, iykyk

also, how a particular encoding is implemented is not specified by the Dirfile Standards. but for the sake of interoperability, all dirfile implementations are encouraged to support the encoding implementation used by the GetData dirfile reference implementation, which gets elaborated below. so like, you can do your own thing, but the group chat will be happier if you match ✨

💜 fragments are local, actually

an encoding scheme is local to the particular format specification fragment in which it's indicated. which is actually so slay, because it means a single dirfile can have binary files stored using multiple encodings — you just define them in multiple fragments. one dirfile, many looks. main character energy 👑

🌈 the fine print

the rest of this manual page discusses the specifics of the encoding framework implemented in the GetData library, and — say it with me — does not constitute part of the Dirfile Standards. so don't come for us about compliance, this part is just the GetData lore 🙈

💖 THE GETDATA ENCODING FRAMEWORK 💖

okay so bestie, listen — the GetData library ships a whole encoding framework that abstracts binary file I/O, so you get generic support for a huge variety of encoding schemes. it's giving plug-and-play, no cap. the functions that may make use of this encoding framework are:

Most of the encodings GetData supports are actually implemented through external libraries that handle the real file I/O and data translation. and here's the tea: all such libraries are optional. if you build the library without one of them, you just don't get support for that associated encoding scheme. GetData will still properly identify the encoding scheme (she's observant like that 💅), but any attempt to actually use GetData for file I/O through that encoding will fail with the GD_E_UNSUPPORTED error code. rude, but fair.

GetData figures out the encoding scheme of a particular RAW field by peeking at the filename extension of the files associated with that field. binary files that form an unencoded dirfile have no file extension. the extensions used by all the other encodings are listed below. discovery goes like this: it searches for files with the known list of file extensions (in an unspecified order) and stops the moment it gets the first successful match.

⚠️ because of that first-match-wins thing, when a field has multiple data files with different, supported file extensions that could legitimately be associated with it, the encoding scheme discovered by GetData is not well defined. so like… don't do that.

on top of raw (unencoded) data, GetData supports nine other encoding schemes: text encoding, bzip2 encoding, flac encoding, gzip encoding, lzma encoding, sie (sample-index encoding), slim encoding, zzip encoding, and zzslim encoding. all of them are discussed below, don't worry 🫶.

and a lil bonus: the text encoding and the sample-index encoding are implemented by GetData natively and need no external library at all. so they're always present in the library. iconic behavior honestly.

🌀 Out-of-place writes

Some of the encodings below only support writing via out-of-place writes — meaning raw files get written to a temporary location and only moved into place when they're closed. the consequence? writing to these encodings requires making a copy of the whole binary data file. 😭

a further side effect: if a third-party is trying to concurrently read a Dirfile while it's being written to using one of these encodings, it usually doesn't work.

within GetData, reading from a field so encoded after writing to it will cause the writing to the temporary file to be finished, and then the file moved into place, before the read happens — which may take some time to do. the encodings that do out-of-place writes are: bzip2, flac, gzip, and lzma.

🎀 BZip2 Encoding

The BZip2 Encoding reads compressed raw binary files using the Burrows-Wheeler block sorting text compression algorithm and Huffman coding, as implemented in the bzip2 format. GetData's BZip2 Encoding scheme is implemented through the bzip2 compression library written by Julian Seward. every operation is supported by the BZip2 Encoding, but writing occurs out-of-place — see the Out-of-place writes section above for the details.

GetData caches an uncompressed megabyte of data at a time to speed up access times. a call to gd_nframes(3) requires decompressing the entire binary file just to determine its uncompressed size, so it may take a hot minute to complete. the file extension of the BZip2 Encoding is .bz2.

🌸 FLAC Encoding

The FLAC Encoding compresses raw binary files using the Free Lossless Audio Codec. GetData's FLAC Encoding scheme is implemented through the libFLAC reference implementation developed by Josh Coalson and the Xiph.Org Foundation. all operations are supported by the FLAC Encoding, but writing occurs out-of-place — see Out-of-place writes above for details.

here's the tricky bit: the FLAC format only permits samples up to 32-bits, but the libFLAC reference codec can only handle samples up to 24-bits. GetData gets around this by slicing data that's wider than 16-bits into multiple channels (2, 4, or 8, depending on width). for big-ended data, the most-significant 16-bits are in channel 0, the second 16-bits in channel 1, &c. for little-ended data, this is reversed, with the least significant word in channel 0.

the sample rate specified in the FLAC header is ignored and may be any valid value. FLAC files written by GetData use a sample rate of 1 Hz. the file extension of the FLAC Encoding is .flac. the Ogg container format is not supported.

🍓 GZip Encoding

The GZip Encoding compresses raw binary files using Lempel-Ziv coding (LZ77) as implemented in the gzip format. GetData's GZip Encoding scheme is implemented through the zlib compression library written by Jean-loup Gailly and Mark Adler. all operations are supported by the GZip Encoding, but writing occurs out-of-place — see Out-of-place writes above for details.

to speed up gd_nframes(3), the GZip Encoding takes the uncompressed size of the file from the gzip footer, which contains the file's uncompressed size in bytes, modulo 2**32.

⚠️ so if you use a field with an (uncompressed) binary file size larger than 4 GiB as the reference field, you'll get the wrong number of frames reported. fr fr, watch out for that one.

the file extension of the GZip Encoding is .gz.

🦋 LZMA Encoding

The LZMA Encoding reads compressed raw binary files using the Lempel-Ziv Markov Chain Algorithm (LZMA) as implemented in the xz container format. GetData's LZMA Encoding scheme is implemented through the lzma library, part of the XZ Utils suite written by Lasse Collin, Ville Koskinen, and Igor Pavlov. all operations are supported by the LZMA Encoding, but writing occurs out-of-place — see Out-of-place writes above for details. writing is supported only for the .xz container format, and not for the obsolete .lzma format — though that one can still be read. 🙈

GetData caches an uncompressed megabyte of data at a time to speed access times. a call to gd_nframes(3) requires decompression of the entire binary file to determine its uncompressed size, and may take some time to complete. the file extension of the LZMA Encoding is .xz, or .lzma.

💜 Sample-Index Encoding

The Sample-Index Encoding (SIE) compresses raw binary data by replacing runs of repeated data — similar to run-length encoding, iykyk. SIE files contain binary records consisting of a 64-bit sample number followed by a datum (the size and format of which is determined by the RAW field's data type in the format metadata). the sample number indicates the last sample of the field which has the specified value. the first sample with the value is the sample immediately following the data in the previous record, or sample number zero, for the first record. sample numbers are relative to any /FRAMEOFFSET specified in the Dirfile metadata. all operations are supported by the Sample-Index Encoding. the file extension of the Sample-Index Encoding is .sie.

✨ Slim Encoding

The Slim Encoding reads compressed raw binary files using the slimlib compression library written by Joseph Fowler. the slimlib library was developed at Princeton University to compress dirfile-like data. GetData's Slim Encoding framework currently lacks write capabilities; as a result, the Slim Encoding does not support functions which modify binary files. the file extension of the Slim Encoding is .slm.

👀 heads up: using the Slim Encoding with GetData may result in unexpected, but manageable, memory usage. see the gd_getdata(3) manual page for details.

🎀 Text Encoding

The Text Encoding replaces the binary data files with 7-bit ASCII files containing a decimal text encoding of the data, one sample per line. all operations are supported by the Text Encoding. the file extension of the Text Encoding is .txt.

🧚 ZZip Encoding

The ZZip Encoding reads compressed raw binary files using the DEFLATE algorithm as implemented in the PKWARE ZIP archive container format. GetData's ZZip Encoding scheme is implemented through the zzip library written by Tomi Ollila and Guido Draheim. the ZZip Encoding framework currently lacks write capabilities; as a result the ZZip Encoding does not support functions which modify binary data.

unlike most encoding schemes, the ZZip encoding merges all binary data files defined in a given fragment into a single ZIP archive. the name of this archive is raw.zip by default, but a different name may be specified using the second parameter to the /ENCODING directive. for example,

/ENCODING zzip archive

indicates that the ZIP archive is called archive.zip. the file extension of the ZZip Encoding is .zip.

🌈 ZZSlim Encoding

The ZZSlim Encoding is a convolution of the Slim Encoding and the ZZip Encoding. to create ZZSlim Encoded files, first the raw data are compressed using the slim library, and then these slim-compressed files are archived (and compressed again) into a ZIP archive. as with the ZZip Encoding, the ZIP archive is raw.zip by default, but a different name may be specified with the /ENCODING directive.

notably, since the archives have the same name as ZZip Encoded data, automatic encoding detection on ZZSlim Encoded data always fails: they are incorrectly identified as simply ZZip Encoded. 💀 as a result, an /ENCODING directive in the format file or else a GD_ZZSLIM_ENCODED flag passed to gd_open(3) is required to read ZZSlim encoded data. the file extension of the ZZSlim Encoding is .zip.

👀 same warning as Slim, bestie: using the ZZSlim Encoding with GetData may result in unexpected, but manageable, memory usage. see the gd_getdata(3) manual page for details.

👑 AUTHOR

This manual page was written by D. V. Wiebe <dvw@ketiltrout.net>.

💅 SEE ALSO

bzip2(1), flac(1), gzip(1), xz(1), zlib(3), dirfile(5), dirfile-format(5)

and that's a wrap, bestie! 🫶 this page is Standards Version 9, dated 15 October 2015, and you just read dirfile-encoding(5). go forth and encode responsibly, mwah 💖✨

📦 where this page comes from