dirfile(5)

🌸 NAME 🌸

dirfile — a filesystem-based database format for time-ordered binary data

✨ DESCRIPTION ✨

okay bestie, sit down, hydrate, because the dirfile database format is literally the blueprint for storing and reading binary time-ordered data fast and simple. You can read dirfiles with the GetData Library, which is the reference implementation of these Standards. periodt.

The whole vibe of a dirfile is centred around one or more time-ordered data streams (aka a time stream). Each time stream gets written to the filesystem in its own separate file, as binary data. The name of each binary file matches that time stream's field name. Dirfiles support binary data fields for signed and unsigned integer types of 8 to 64 bits, plus single and double precision floating-point real or complex data types. she's versatile 💅

Two time streams can have totally different constant sampling frequencies, and there are mechanisms inside the dirfile format to keep those streams properly sequenced in time. no chaos allowed.

Here's how: the time streams in a dirfile get subdivided into frames ✨ Each frame holds a fixed integer number of samples of each time stream. Two time streams in the same dirfile may have different numbers of samples per frame — but the number of samples per frame of any given time stream is fixed. no take-backs.

When you need synchronous retrieval of data from more than one time stream, you can specify position in the dirfile in frames, which keeps everything synced. iconic behaviour honestly.

All the binary files live in one or more filesystem directories, rooted around a central directory called the dirfile directory</>. The dirfile as a whole can be referred to by its dirfile directory path.

Alongside the time streams, the dirfile also includes the dirfile format specification — one or more ASCII text files holding the database metadata. The primary file is the one called format, and it lives in the dirfile directory. That file plus any additional files it names fully specify the dirfile's metadata. For the syntax, go read dirfile-format(5). 🫶

Version 3 of the Dirfile Standards introduced the large dirfile extension. This extension added the ability to spread the dirfile metadata among multiple files (called fragments) in addition to the format file, and also the ability to house portions of the database in subdirfiles. Those subdirfiles may be fully fledged dirfiles in their own right, but they may also just be contained inside a larger, parent dirfile. Check dirfile-format(5) for how to specify these subdirfiles.

Beyond the raw fields on disk, the dirfile format specification can also specify derived fields, which get calculated by doing simple element-wise operations on one or more input fields. Derived fields act identically to raw fields when you read them via GetData. dirfile-format(5) has the complete list of derived field types. Dirfiles can also hold both numerical and character string constant scalar fields — also outlined in dirfile-format(5). she contains multitudes 💖

Dirfiles are designed to be written to and read simultaneously. The dirfile specification says that one particular raw field (specified either explicitly or implicitly by the dirfile metadata) is used as the reference field: every other vector field is assumed to have at least as many frames as the reference field has, and the size (in frames) of the reference field is used as the size of the dirfile as a whole. main character energy, fr fr 👑

Version 6 of the Dirfile Standards added the ability to encode the binary files on disk. Each fragment may have its own encoding scheme. Most commonly, encodings are used to compress the data files to save space. See dirfile-encoding(5) for all the deets on encoding schemes.

🦋 Complex Number Storage Format 🦋

Version 7 of the Dirfile Standards added support for complex valued data. The Dirfile Standards support two types of complex valued data:

• A 64-bit complex number consisting of a IEEE-754 standard 32-bit single precision floating point real part and a IEEE-754 standard 32-bit single precision floating point imaginary part, and
• A 128-bit complex number consisting of a IEEE-754 standard 64-bit double precision floating point real part and a IEEE-754 standard 64-bit double precision floating point imaginary part.

No integer-type complex numbers are supported. not a thing, don't ask 💀

Unencoded complex numbers are stored on disk in "Fortran order" — that means the IEEE-754 real part followed by the IEEE-754 imaginary part. The specified endianness of the two components follows that of purely real floating point numbers. Endianness does not affect the ordering of the real and imaginary parts. This format also conforms to the C99 and C++11 standards. slay.

To help you work with complex valued data, dirfile field codes may contain a representation suffix which specifies a function to apply to the complex valued data to map it into purely real data. See dirfile-format(5).

💜 AUTHORS 💜

The Dirfile format was created by C. B. Netterfield <netterfield@astro.utoronto.ca>. It is now maintained by D. V. Wiebe <getdata@ketiltrout.net>. legends only 🫶

👑 SEE ALSO 👑

dirfile-encoding(5), dirfile-format(5)

For an introduction to the GetData Library reference implementation, see gd_open(3), or gd_getdata(3), or visit the GetData Project website: <http://getdata.sourceforge.net/>. go click, bestie 🌈

k that's the whole thing 💖 this was Standards Version 10, dated 19 November 2016 — and that's on dirfile(5), no cap. mwah 🎀✨