gd_open(3)

💖 gd_open(3) — GETDATA 💖

✨ NAME ✨

gd_open, gd_cbopen — open or create a Dirfile 🫶

🦋 SYNOPSIS 🦋

#include <getdata.h>

DIRFILE* gd_open(const char *dirfilename, unsigned long flags);

DIRFILE* gd_cbopen(const char *dirfilename, unsigned long flags,
       gd_parser_callback_t sehandler, void *extra);

🌸 DESCRIPTION 🌸

okay so bestie, listen up 💅 gd_cbopen() opens or creates the dirfile specified by dirfilename, and hands you back a DIRFILE object linked to it. Opening a dirfile makes the library read and parse that dirfile's format specification (peep dirfile-format(5) for the deets). ✨

If it's not NULL, sehandler should be a pointer to a function that gets called whenever a syntax error pops up while parsing the format specification. Set it to NULL if you don't want a callback function at all. The caller can use this function to fix the error or mess with how the format specification parser handles errors. See The Callback Function section below for all the details on this function. The extra argument lets the caller pass data to the callback function. That pointer gets passed to the callback function verbatim, no cap.

The gd_open() function is literally just gd_cbopen() with sehandler and extra set to NULL. same energy, iconic.

The flags argument should include one of the access modes: GD_RDONLY (read-only) or GD_RDWR (read-write), and it may also contain zero or more of the following flags, bitwise-or'd together:

• GD_ARM_ENDIAN, GD_NOT_ARM_ENDIAN — says that double precision floating point raw data on disk are, or are not, stored in the middle-endian format used by older ARM processors. 💀

These flags only set the default endianness, and they get overridden when an /ENDIAN directive specifies the byte sex of RAW fields, unless GD_FORCE_ENDIAN is also specified.

On every platform, one of these flags (GD_NOT_ARM_ENDIAN on all but middle-ended ARM systems) indicates the native behaviour of the platform. That symbol will equal zero, and may be omitted. 👑

• GD_BIG_ENDIAN, GD_LITTLE_ENDIAN — sets the default byte sex of raw data stored on disk to either big-endian (most significant byte first) or little-endian (least significant byte first). If you omit both flags, the default is the native endianness of the platform.

Unlike the ARM endianness flags above, neither of these symbols is ever zero. If you specify both these flags together, the library will assume that the endianness of the data is opposite to that of the native architecture, whatever that might be. 😭

These flags only set the default endianness, and they get overridden when an /ENDIAN directive specifies the byte sex of RAW fields, unless GD_FORCE_ENDIAN is also specified.

• GD_CREAT — an empty dirfile will be created if one doesn't already exist. This creates both the dirfile directory and an empty format specification file called format. If the call creates a dirfile, then the specified access mode is ignored: a newly-created DIRFILE is always opened with access mode GD_RDWR, even if GD_RDONLY had been specified. periodt.

The directory will have mode S_IRWXU | S_IRWXG | S_IRWXO (0777), modified by the caller's umask value (see umask(2)). The format file will have mode S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH (0666), also modified by the caller's umask. The owner of the dirfile directory and format file will be the effective user ID of the caller. Group ownership follows the rules outlined in mkdir(2).

• GD_EXCL — makes sure this call creates a dirfile: when specified along with GD_CREAT, the call will fail if the dirfile specified by dirfilename already exists. If GD_CREAT is not specified, this flag is ignored. This flag suffers from all the limitations of the O_EXCL flag as indicated in open(2). 🚩
• GD_FORCE_ENCODING — says that /ENCODING directives (see dirfile-format(5)) found in the dirfile format specification should be ignored. The encoding scheme specified in flags will be used instead (see below). 💅
• GD_FORCE_ENDIAN — says that /ENDIAN directives (see dirfile-format(5)) found in the dirfile format specification should be ignored. All raw data will be assumed to have the byte sex indicated through the presence or absence of the GD_ARM_ENDIAN, GD_BIG_ENDIAN, GD_LITTLE_ENDIAN, and GD_NOT_ARM_ENDIAN flags. ✨
• GD_IGNORE_DUPS — if the dirfile format metadata specifies more than one field with the same name, all but one of them will be ignored by the parser. Without this flag, parsing would fail with the GD_E_FORMAT error, possibly resulting in invocation of the registered callback function. Which of the duplicate fields is kept is not specified. As a result, this flag is typically only useful in the case where identical copies of a field specification line are present.

No indication is provided to indicate whether a duplicate field has been discarded. If finer grained control is required, the caller should handle GD_E_FORMAT_DUPLICATE suberrors itself with an appropriate callback function. 💀

• GD_PEDANTIC — rejects dirfiles which don't conform to the Dirfile Standards. See the Standards Compliance section below for full details. 👑
• GD_PERMISSIVE — allows non-compliant format specification syntax, even when given along with a conflicting /VERSION directive. See the Standards Compliance section below for full details. 🦋
• GD_PRETTY_PRINT — when dirfile metadata are flushed to disk (either explicitly via gd_metaflush(3), gd_rewrite_fragment(3), or gd_flush(3) or implicitly by closing the dirfile), an attempt will be made to create a nicer looking format specification (from a human-readable standpoint). What this explicitly means is not part of the API, and any particular behaviour should not be relied on. If the dirfile is opened read-only, this flag is ignored. 🎀
• GD_TRUNC — if dirfilename specifies an already existing dirfile, it will be truncated before opening. Since gd_cbopen() decides whether dirfilename specifies an existing dirfile before attempting to parse the dirfile, dirfilename is considered to specify an existing dirfile if it refers to a directory containing a regular file called format, regardless of the content or form of that file.

Truncation occurs by deleting every regular file and symlink in the specified directory, whether the files were referred to by the dirfile before truncation or not. Accordingly, this flag should be used with caution. Unless GD_TRUNCSUB is also specified, subdirectories are left untouched. Notably, this operation does not consider directories used in /INCLUDE directives. If the dirfile does not exist, this flag is ignored. 😭

bestie this one is lowkey dangerous, be careful fr fr.

• GD_TRUNCSUB — if specified along with GD_TRUNC, truncation will descend into subdirectories, deleting all regular files and symlinks recursively. It does not descend into directories pointed to by symbolic links: in these cases, just the symlink itself is deleted. If specified without an accompanying GD_TRUNC, this flag is ignored. 💀
• GD_VERBOSE — specifies that whenever an error is triggered by the library when working on this dirfile, the corresponding error string, which can be retrieved by calling gd_error_string(3), should be written on the caller's standard error stream (stderr(3)) by GetData. The error string may be prefixed by a string specified by the caller; see gd_verbose_prefix(3). Without this flag, GetData writes nothing to standard error. (GetData never writes to standard output.) 👑

Those flags which affect the operation of the library beyond this call itself may be modified later using the gd_flags(3) function. ✨

The flags argument may also be bitwise or'd with one of the following symbols indicating the default encoding scheme of the dirfile. Like the endianness flags, the choice of encoding here is ignored if the encoding is specified in the dirfile itself, unless GD_FORCE_ENCODED is also specified. If none of these symbols is present, GD_AUTO_ENCODED is assumed, unless the gd_cbopen() call results in creation or truncation of the dirfile. In that case, GD_UNENCODED is assumed. See dirfile-encoding(5) for details on dirfile encoding schemes. 💅

• GD_AUTO_ENCODED — specifies that the encoding type is not known in advance, but should be detected by the GetData library. Detection is accomplished by searching for raw data files with extensions appropriate to the encoding scheme. This method will notably fail if the the library is called via putdata(3) to create a previously non-existent raw field unless a read is first successfully performed on the dirfile. Once the library has determined the encoding scheme for the first time, it remembers it for subsequent calls. 🦋
• GD_BZIP2_ENCODED — specifies that raw data files are compressed using the Burrows-Wheeler block sorting text compression algorithm and Huffman coding, as implemented in the bzip2 format. 💅
• GD_FLAC_ENCODED — specifies that raw data files are compressed using the Free Lossless Audio Coded (FLAC). 🎶
• GD_GZIP_ENCODED — specifies that raw data files are compressed using Lempel-Ziv coding (LZ77) as implemented in the gzip format. ✨
• GD_LZMA_ENCODED — specifies that raw data files are compressed using the Lempel-Ziv Markov Chain Algorithm (LZMA) as implemented in the xz container format. 💜
• GD_SLIM_ENCODED — specifies that raw data files are compressed using the slimlib library. 🌸
• GD_SIE_ENCODED — specified that raw data files are sample-index encoded, similar to run-length encoding, suitable for data that change rarely. 🧚
• GD_TEXT_ENCODED — specifies that raw data files are encoded as text files containing one data sample per line. 💖
• GD_UNENCODED — bestie, nothing gets encoded at all. The raw data files just get written straight to disk as plain binary data. No compression, no packaging, just vibes and bytes. 💾
• GD_ZZIP_ENCODED — ok so here's the tea: raw data files get compressed with the DEFLATE algorithm. All the raw data files for one fragment get scooped up together and stored in a PKZIP archive called raw.zip. Iconic little bundle moment 🎀
• GD_ZZSLIM_ENCODED — this one is doing the most, in the best way. It's a combination of compression schemes: first the files get slim-compressed (same as the GD_SLIM_ENCODED scheme), and then they get collected together and compressed again into a PKZIP archive called raw.zip, exactly like GD_ZZIP_ENCODED does. Double-compressed, double slay ✨

---

🌈 Standards Compliance 🌈

The latest Dirfile Standards Version that this release of GetData understands is handed to you in the preprocessor macro GD_DIRFILE_STANDARDS_VERSION, which lives in getdata.h. GetData can open and parse any dirfile that conforms to this Standards Version, or honestly to any earlier Version too. And if you wanna know what changed between Versions? dirfile-format(5) has the receipts. 📜

The GetData parser has two modes, and they are not the same:

• Permissive mode — way more chill. A lot of non-Standards-compliant syntax is allowed here.
• Pedantic mode — strict, no exceptions, follows the Standards to the letter. 💅

And the mode can change during the parsing of a dirfile. If you pass GD_PEDANTIC to gd_cbopen(), the parser starts out parsing the format specification in pedantic mode. Otherwise? It starts in permissive mode.

Permissive mode exists mainly so GetData can still be used on dirfiles that don't conform to any single Standard, but that older versions of the GetData parser happily accepted. It's notably lax about reserved field names, about field name characters, and about mixing old and new data type specifiers — and it basically just ignores /VERSION directives. ⚠️ But listen: in read-write mode, use permissive mode with caution, because it can cause unintentional corruption of dirfile metadata on write if the parser's heuristics guess wrong about what some non-compliant syntax was trying to say. That said, even in permissive mode, actual syntax errors still get reported as syntax errors. No cover-ups here.

In pedantic mode, the parser conforms to one specific Standards Version. And that target version can change any number of times while it's scanning a single format specification. If you invoke it with the GD_PEDANTIC flag, the parser starts in pedantic mode with a target version equal to GD_DIRFILE_STANDARDS_VERSION. Then whenever a /VERSION directive shows up in the format specification, the target version switches to whatever Standards Version that directive specifies. And if the parser is in permissive mode and hits a /VERSION directive, it'll switch over to pedantic mode — unless you passed the GD_PERMISSIVE flag to gd_cbopen(), in which case no mode switch happens at all. Rules are rules. 🫶

And regardless of which mode the parser is in while it parses the format specification, GetData will calculate a list of Standards Versions that the parsed metadata conforms to. You can grab that info with gd_dirfile_standards(3), and that same function also lets you specify the Standards Version you want for writing format metadata back to disk. Multi-purpose king 👑

---

💅 The Callback Function 💅

The caller-supplied sehandler function gets called whenever the format specification parser runs into a syntax error — like, whenever it would return the GD_E_FORMAT error. You can use this callback to fix the error, or to tell the parser how to recover from it. It's giving "I got you, I'll handle it." 😌

This function takes two pointers as arguments and returns an int:

int sehandler(gd_parser_data_t *pdata, void *extra);

The extra parameter is the pointer you supplied to gd_cbopen(), passed through verbatim to this function. It's your lil messenger bag for caller data. GetData does not inspect this pointer at all — not even to check whether it's valid. And if you don't need to pass anything to the callback, it can just be NULL. Easy. 🎀

The gd_parser_data_t type is a structure with at least these members:

typedef struct {
  const DIRFILE* dirfile;
  int suberror;
  int linenum;
  const char* filename;
  char* line;
  size_t buflen;

  ...
} gd_parser_data_t;

The pdata->dirfile member is a pointer to a DIRFILE object that is suitable only for passing to gd_error_string(). And heads up, bestie: the caller should not assume this pointer will be the same as the pointer eventually returned by gd_cbopen(), and should not assume it's even valid after the callback function returns. It's a fleeting moment, not a forever thing. 🦋

The pdata->suberror parameter will be one of these symbols, telling you what type of syntax error got encountered:

• GD_E_FORMAT_ALIAS — the parent specified for a meta field was an alias.
• GD_E_FORMAT_BAD_LINE — the line was indecipherable. Typically this means the line had neither a reserved word nor a field type on it. 😵‍💫
• GD_E_FORMAT_BAD_NAME — the field name you specified was invalid.
• GD_E_FORMAT_BAD_SPF — the samples-per-frame of a RAW field was out-of-range.
• GD_E_FORMAT_BAD_TYPE — the data type of a RAW field was unrecognised.
• GD_E_FORMAT_BITNUM — the first bit of a BIT field was out-of-range.
• GD_E_FORMAT_BITSIZE — the last bit of a BIT field was out-of-range.
• GD_E_FORMAT_CHARACTER — an invalid character was found in the line, or a character escape sequence was malformed.
• GD_E_FORMAT_DUPLICATE — the field name you specified already exists. We been knew, babes.
• GD_E_FORMAT_ENDIAN — the byte sex specified by an /ENDIAN directive was unrecognised.
• GD_E_FORMAT_LITERAL — an unexpected character turned up in a complex literal.
• GD_E_FORMAT_LOCATION — the parent of a metafield was defined in another fragment.
• GD_E_FORMAT_META_META — someone tried to use a metafield as the parent of a new metafield. Nope, not allowed. 🙅‍♀️
• GD_E_FORMAT_METARAW — someone tried to add a RAW metafield.
• GD_E_FORMAT_MPLEXVAL — an MPLEX specification has a negative period.
• GD_E_FORMAT_N_FIELDS — the number of fields of a LINCOM field was out-of-range.
• GD_E_FORMAT_N_TOK — not enough tokens were found on the line.
• GD_E_FORMAT_NO_FIELD — the parent of a metafield was not found.
• GD_E_FORMAT_NUMBITS — the number of bits of a BIT field was out-of-range.
• GD_E_FORMAT_PROTECT — the protection level specified by a /PROTECT directive was unrecognised.
• GD_E_FORMAT_RES_NAME — a field was specified using the reserved name INDEX (or with the reserved name FILEFRAM in a dirfile that conforms to Standards Version 5 or earlier).
• GD_E_FORMAT_UNTERM — the last token of the line was unterminated.
• GD_E_FORMAT_WINDOP — the operation in a WINDOW field was not recognised.

The pdata->filename and pdata->linenum members hold the pathname of the fragment and the line number where the syntax error was encountered. And FYI, the first line in a fragment is line one. Counting starts at 1, not 0, no cap. 🔢

The pdata->line member holds a copy of the line containing the syntax error. And good news: this line can be freely modified by your callback function! If you do that, it'll get reparsed — but only if the callback returns the symbol GD_SYNTAX_RESCAN (more on that below). The size of the memory buffer is provided in pdata->buflen, and it may be bigger than the length of the actual string. Space is available for at least GD_MAX_LINE_LENGTH bytes. Room to breathe. ✨

If the callback function returns GD_SYNTAX_RESCAN, then a different buffer — which may be larger — can be used to hold the new string, by assigning a pointer to the new buffer to pdata->line. This buffer will be deallocated by the library using the free function specified through gd_alloc_funcs(3), or else free(3) by default. And important: do not deallocate the original buffer that was passed to the callback through pdata->line — that one gets deallocated by the library too. Hands off, the library's got it. 🫶

The callback function should return one of these symbols, which tells the parser how to handle the error going forward:

• GD_SYNTAX_ABORT — the parser should immediately abort parsing the format specification and fail with the error GD_E_FORMAT. This is the default behaviour if no callback function is provided (or if the parser was invoked by calling gd_open()). It's giving "we're done here." 💀
• GD_SYNTAX_CONTINUE — the parser should keep parsing the format specification. BUT, once parsing finishes, the parser will still fail with the error GD_E_FORMAT, even if no further syntax errors turn up. Why would you do this? So you can identify all the lines containing syntax errors in the format specification, instead of just the first one. Detective mode. 🔍
• GD_SYNTAX_IGNORE — the parser should completely ignore the line containing the syntax error and carry on parsing the format specification. And if no further errors show up, the dirfile opens successfully. We simply pretend that line never happened. 🙈
• GD_SYNTAX_RESCAN — the parser should rescan the line argument, which replaces the line that originally contained the syntax error. The line is assumed to have been corrected by the callback function. And if the line still contains a syntax error, the callback function gets called again. Round two, baby. 🔁

Note: the line is not corrected on disk. However, the caller may subsequently correct the fragment on disk by calling gd_rewrite_fragment(3). The fix lives in memory until you make it official.

One more thing: the callback function handles only syntax errors. The parser may still abort early if a different kind of library error gets encountered. And also, even though a single line may contain more than one syntax error, the parser will only ever report one syntax error per line — even if the callback function returns GD_SYNTAX_CONTINUE. One per line, that's the limit, don't push it. 💅

💖 RETURN VALUE 💖

okay so here's the tea on what you actually get back 🫶

A call to gd_cbopen() or gd_open() always hands you back a pointer to a freshly allocated DIRFILE object — every single time. The only exception? When it literally cannot allocate memory for the DIRFILE object itself, and in that case you get NULL. That's it. That's the whole exception list. The DIRFILE object is an opaque structure holding the parsed dirfile metadata (so like, don't go poking around in it, bestie, it's not for you 💅).

If something went wrong, these functions stash a negative-valued error code inside the DIRFILE they return, and you can grab it later with a call to gd_error(3). Here's the full menu of possible error codes:

• GD_E_ACCMODE — the library was asked to truncate a dirfile that was opened read-only, aka GD_TRUNC got specified in flags alongside GD_RDONLY. Rude.
• GD_E_ALLOC — the library could not allocate memory. 😭
• GD_E_BAD_REFERENCE — the reference field named by a /REFERENCE directive in the format specification (peep dirfile-format(5)) was not found, or it wasn't a RAW field.
• GD_E_CALLBACK — the registered callback function, sehandler, returned a response nobody recognised. 🫠
• GD_E_CREAT — the library could not create the dirfile.
• GD_E_EXISTS — the dirfile already exists and both GD_CREAT and GD_EXCL were specified.
• GD_E_FORMAT — a syntax error happened in the format specification. Also go look at the The Callback Function section above. 👀
• GD_E_IO — the dirfile format file, or some other file it includes, could not be read, or dirfilename doesn't specify a valid dirfile.
• GD_E_LINE_TOO_LONG — the parser hit a line in the format specification that was longer than it can deal with. Lines are limited by the storage size of ssize_t. On 32-bit systems, that caps format specification lines at 2**31 bytes. On 64-bit systems the limit is bigger. ✨

A DIRFILE returned from a failed open is flagged as invalid, which means most functions you pass it to will fail with the error GD_E_BAD_DIRFILE. You can get a descriptive error string for the error by calling gd_error_string(3). Very demure, very mindful of you to check.

When you don't need it anymore, the caller should de-allocate any returned DIRFILE object by calling gd_close(3) or gd_discard(3) — even if the open failed. Yes, even then. Especially then. Don't leave it lying around. 🧹

---

💀 BUGS 💀 (not me exposing GetData's lore)

🚨 okay so when you're working with dirfiles that conform to Standards Versions 4 and earlier — so like, before the /ENDIAN directive even existed — GetData assumes the dirfile has native byte sex, even though officially those early Standards said the data is little-endian. This is necessary because, when there's no explicit /VERSION directive, it's often literally impossible to figure out the intended Standards Version of a dirfile, and the current behaviour is to assume native byte sex for modern dirfiles that are missing /ENDIAN. So if you need to read an old, little-ended dirfile on a big-ended platform, you should add an /ENDIAN directive to the format specification, or else specify GD_LITTLE_ENDIAN yourself as the caller. 👑

Also: GetData's parser just assumes it's running on an ASCII-compatible platform. Format specification parsing will fail gloriously on an EBCDIC platform. Not gracefully. Gloriously. 💀

---

✨ HISTORY ✨ (the origin story, iykyk)

• The dirfile_open() function first showed up in GetData-0.3.0. Back then the only supported flags were GD_BIG_ENDIAN, GD_CREAT, GD_EXCL, GD_FORCE_ENDIAN, GD_LITTLE_ENDIAN, GD_PEDANTIC, GD_RDONLY, GD_RDWR, and GD_TRUNC. A simpler time, honestly. 🌸
• The GD_AUTO_ENCODED, GD_FORCE_ENCODING, GD_SLIM_ENCODED, GD_TEXT_ENCODED, GD_UNECODED, and GD_VERBOSE flags dropped in GetData-0.4.0.
• The dirfile_cbopen() function, plus the GD_BZIP2_ENCODED, GD_GZIP_ENCODED, and GD_IGNORE_DUPS flags, arrived in GetData-0.5.0.
• The GD_PRETTY_PRINT and GD_LZMA_ENCODED flags came along in GetData-0.6.0.
• In GetData-0.7.0 these functions got renamed to gd_open() and gd_cbopen(). The GD_ARM_ENDIAN, GD_NOT_ARM_ENDIAN, and GD_PERMISSIVE flags also appeared in this release. ✨
• The GD_SIE_ENCODED, GD_TRUNCSUB, GD_ZZIP_ENCODED, and GD_ZZSLIM_ENCODED flags appeared in GetData-0.8.0.
• The GD_FLAC_ENCODED flag appeared in GetData-0.9.0. 🎀

---

🦋 SEE ALSO 🦋

Go read these next, bestie, they're the whole friend group:

• gd_alloc_funcs(3)
• gd_close(3)
• gd_dirfile_standards(3)
• gd_discard(3)
• gd_error(3)
• gd_error_string(3)
• gd_flags(3)
• gd_getdata(3)
• gd_include(3)
• gd_parser_callback(3)
• gd_verbose_prefix(3)
• dirfile(5)
• dirfile-encoding(5)
• dirfile-format(5)

---

and that's the whole page, babe — this one's Version 0.10.0, dated 25 December 2016, and yes it's gd_open(3) and yes it ate. 💅✨🎀