gd_include(3)

💖 gd_include(3) — GETDATA manual, bestie edition 💖

okay so this is the gd_include(3) page from the GETDATA manual, and it's a whole saga so we're doing it in two parts. sit down, grab a strawberry 🍓, and let's get into it.

---

✨ NAME ✨

gd_include, gd_include_affix, gd_include_ns — the trio that adds a format specification fragment to a Dirfile. slay 💅

---

🌸 SYNOPSIS 🌸

#include <getdata.h>

int gd_include(DIRFILE *dirfile, const char *include_file, int
       parent_fragment, unsigned long flags);

int gd_include_affix(DIRFILE *dirfile, const char *include_file, int
       parent_fragment, const char *prefix, const char *suffix, unsigned
       long flags);

int gd_include_ns(DIRFILE *dirfile, const char *include_file, int
       parent_fragment, const char *namespace, unsigned long flags);

---

🦋 DESCRIPTION 🦋

okay so the main character here is gd_include_affix(). what it does is take the format specification fragment sitting at the path include_file and add it to the dirfile you're pointing at — and it might even create that fragment, using the affixes you gave it. it's giving "i can do both, i contain multitudes" ✨

this all happens as if, inside the existing fragment indexed by parent_fragment, this directive were just sitting there being its little self:

/INCLUDE <include_file> <prefix> <suffix>

(peep dirfile-format(5) for the lore on that directive, iykyk)

now, about that prefix — it may include a namespace, and that namespace is separated from the rest of the prefix by a dot (.). also the prefix can totally be the empty string, no judgement. 🎀

and if a parser callback function was specified back when the dirfile was opened using gd_cbopen(3), or added later with gd_parser_callback(3), that callback function will get called if a syntax error pops up while parsing the included fragment. so like, heads up.

passing NULL as prefix or suffix is exactly the same as using the empty string (ie. the corresponding affix is empty). lowkey the same vibe either way 🫶

• gd_include() — equivalent to calling gd_include_affix() with both prefix and suffix equal to NULL. it's the low-effort version, and honestly? valid.
• gd_include_ns() — equivalent to calling gd_include_affix() with suffix equal to NULL and prefix equal to namespace concatenated with a trailing dot. periodt.

---

🌈 the flags argument

the flags argument should be a bitwise-or'd collection of zero or more of the following flags. yes, zero is allowed, no you don't have to pick one, it's not a vibe check you can fail 💜

• GD_ARM_ENDIAN, GD_NOT_ARM_ENDIAN — specifies that double precision floating point raw data on disk are, or are not, stored in the middle-endian format used by older ARM processors. (iconic legacy behaviour honestly)

these flags only set the default endianness, and they will be overridden when an /ENDIAN directive specifies the byte sex of RAW fields — unless GD_FORCE_ENDIAN is also specified. so don't come crying if your /ENDIAN wins, it's literally supposed to.

on every platform, one of these flags (GD_NOT_ARM_ENDIAN on all but middle-ended ARM systems) indicates the native behaviour of the platform. that symbol will equal zero, and may be omitted. so like, the free one, basically. ✨

• GD_BIG_ENDIAN, GD_LITTLE_ENDIAN — specifies the default byte sex of raw data stored on disk to be either big-endian (most significant byte first) or little-endian (least significant byte first). omitting both flags means the default should be the native endianness of the platform. no cap.

unlike the ARM endianness flags above, neither of these symbols is ever zero. specifying both these flags together will cause the library to assume that the endianness of the data is opposite to that of the native architecture, whatever that might be. yes that's real, yes it's chaotic, and yes it's exactly what the manual says 💀

these flags only set the default endianness, and will be overridden when an /ENDIAN directive specifies the byte sex of RAW fields, unless GD_FORCE_ENDIAN is also specified.

• GD_CREAT — an empty fragment will be created, if one does not already exist. the fragment will have mode S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH (0666), modified by the caller's umask value (see umask(2)). new fragment just dropped 🎀
• GD_EXCL — ensure that this call creates a new fragment: when specified along with GD_CREAT, the call will fail if the file specified by include_file already exists. if GD_CREAT is not specified, this flag is ignord. (yes the manpage really says ignord, we love a typo, we stan a typo, we are not fixing it 💅)

this flag suffers from all the limitations of the O_EXCL flag as indicated in open(2). so, baggage. inherited baggage. 🙈

• GD_FORCE_ENCODING — specifies that /ENCODING directives (see dirfile-format(5)) found in the fragment should be ignored. the encoding scheme specified in flags will be used instead (see below).
• GD_FORCE_ENDIAN — specifies that /ENDIAN directives (see dirfile-format(5)) found in the fragment should be ignored. when specified with one of GD_BIG_ENDIAN or GD_LITTLE_ENDIAN, the indicated endianness will be assumed. if this flag is specified with neither of those flags, the fragment will be assumed to have the endianness of the native architecture. it's giving "trust me bro, i know what endian this is" 👑
• GD_IGNORE_DUPS — if the fragment specifies more than one field with the same name, or a field with the same name as an existing field, all but one of them will be ignored by the parser. without this flag, parsing would fail with the GD_E_FORMAT error, possibly resulting in invocation of the registered callback function. 😭

which of the duplicate fields is kept is not specified, nor whether an existing field takes precedence over a new one or not. as a result, this flag is typically only useful in the case where identical copies of a field specification line are present. so like... same line twice? fine. anything sneakier? bestie, you're on your own.

no indication is provided to indicate whether a duplicate field has been discarded. if finer grained control is required, the caller should handle GD_E_FORMAT_DUPLICATE suberrors itself with an appropriate callback function. ✨

• GD_IGNORE_REFS — if the dirfile currently has a reference field (either because one was specified explicitly, or else because the first RAW field was used), /REFERENCE directives in the included fragment will be ignored. otherwise, a /REFERENCE directive in the included fragment will replace the current reference field in the dirfile. swap or skip, pick your fighter 💜
• GD_PEDANTIC — specifies that unrecognised lines found during the parsing of the fragment should always cause a fatal error. without this flag, if a VERSION directive (see dirfile-format(5)) indicates that the fragment being opened conforms Standards Version newer than the version understood by the library, unrecognised lines will be silently ignored. so this is the "i want to know EVERYTHING" mode. main character energy fr fr 👑
• GD_TRUNC — if include_file already exists, it will be truncated before opening. if the file does not exist, this flag is ignored. bye bye old contents 👋

---

🧚 the encoding flags (a whole second buffet)

the flags argument may also be bitwise or'd with one of the following symbols indicating the default encoding scheme of the fragment. like the endianness flags, the choice of encoding here is ignored if the encoding is specified in the fragment itself, unless GD_FORCE_ENCODED is also specified. 🫶

if none of these symbols is present, GD_AUTO_ENCODED is assumed — unless this call results in creation or truncation of the fragment. in that case, GD_UNENCODED is assumed. see dirfile-encoding(5) for details on dirfile encoding schemes.

• GD_AUTO_ENCODED — specifies that the encoding type is not known in advance, but should be detected by the GetData library. detection is accomplished by searching for raw data files with extensions appropriate to the encoding scheme. this method will notably fail if the the library is called via gd_putdata(3) to create a previously non-existent raw field — unless a read is first successfully performed on the dirfile. once the library has determined the encoding scheme for the first time, it remembers it for subsequent calls. it's giving "i'll figure it out" and honestly, it usually does 🦋
• GD_BZIP2_ENCODED — specifies that raw data files are compressed using the Burrows-Wheeler block sorting text compression algorithm and Huffman coding, as implemented in the bzip2 format.
• GD_FLAC_ENCODED — specifies that raw data files are compressed using the Free Lossless Audio Coded (FLAC). yes, the audio one. for your data. we love a crossover episode 🎧
• GD_GZIP_ENCODED — specifies that raw data files are compressed using Lempel-Ziv coding (LZ77) as implemented in the gzip format.
• GD_LZMA_ENCODED — specifies that raw data files are compressed using the Lempel-Ziv Markov Chain Algorithm (LZMA) as implemented in the xz container format.
• GD_SIE_ENCODED — specified that raw data files are sample-index encoded, similar to run-length encoding, suitable for data that change rarely. (yes it says "specified", not "specifies", we're keeping it, it's part of the charm 💅)
• GD_SLIM_ENCODED — specifies that raw data files are compressed using the slimlib library.
• GD_TEXT_ENCODED — specifies that raw data files are encoded as text files containing one data sample per line. cute, readable, yours truly approves ✨
• GD_UNENCODED — specifies that raw data files are not encoded, but written verbatim to disk. raw data staying raw, periodt.
• GD_ZZIP_ENCODED — specifies that raw data files are compressed using the DEFLATE algorithm. all raw data files for a given fragment are collected together and stored in a PKZIP archive called raw.zip. one zip to hold them all 👑
• GD_ZZSLIM_ENCODED — specifies that raw data files are compressed using a combinations of compression schemes: first files are slim-compressed, as with the GD_SLIM_ENCODED scheme, and then they are collected together and compressed (again) into a PKZIP archive called raw.zip, as in the GD_ZZIP_ENCODED scheme. double compression, double the slay 🌈

---

and that's the end of part 1, bestie — there's more coming, so don't go anywhere 💜

✨ RETURN VALUE ✨

Okay bestie, here's the tea on what you get back 🫶

• On success — these functions hand you the format specification index of the freshly-added fragment. Cute, right? That's your little receipt 🧾
• On error — you get a negative-valued error code. Not a vibe. Possible error codes are:
• GD_E_ACCMODE — the dirfile you handed over was opened in read-only mode 🙅♀️ no writing today.
• GD_E_ALLOC — the library could not allocate memory. It's giving out-of-ram energy 😭
• GD_E_BAD_DIRFILE — the dirfile you supplied was invalid. Not a real dirfile, bestie 💀
• GD_E_BAD_INDEX — the parent fragment index you supplied was out of range. Too far, babes 📏
• GD_E_BAD_REFERENCE — the reference field specified by a /REFERENCE directive in the fragment (peep dirfile-format(5)) was not found, or was not a RAW field. And here's the important bit: in this case, the included fragment will still be added to the dirfile, but the /REFERENCE directive will be ignored. So like... half-slay? 🤔
• GD_E_CALLBACK — the registered callback function returned an unrecognised response. Rude.
• GD_E_FORMAT — a syntax error occurred in the fragment. Grammar check failed, periodt ✍️
• GD_E_LINE_TOO_LONG — the parser hit a line in the format specification that was longer than it could deal with. Lines are limited by the storage size of ssize_t. On 32-bit systems, that caps format specification lines at 2**31 characters. The limit is bigger on 64-bit systems. So like, it's very long, but not infinite 💅
• GD_E_IO — an I/O error occured while trying to read or create the fragment. The disk said no 🫠
• GD_E_PROTECTED — the metadata of the parent fragment was protected from change. Hands off, it's guarded 🔒

The error code is also stashed in the DIRFILE object, so you can go grab it after this function returns by calling gd_error(3). And if you want a whole descriptive error string — like, an actual sentence about what went wrong — call gd_error_string(3) 🌸

---

📜 HISTORY 📜

• The dirfile_include() function appeared in GetData-0.4.0.
• In GetData-0.7.0, this function was renamed to gd_include().
• The gd_include_affix() function appeared in GetData-0.8.0.
• In GetData-0.10.0, the error return from these functions changed from -1 to a negative-valued error code. The gd_include_ns() function also appeared in this release. Glow-up era ✨
• See gd_open(3) for history of the flags. 👀

---

👀 SEE ALSO 👀

gd_alter_affixes(3), gd_error(3), gd_error_string(3), gd_fragmentname(3), gd_nfragments(3), gd_open(3), gd_parser_callback(3), gd_reference(3), gd_uninclude(3), dirfile(5), dirfile-encoding(5), dirfile-format(5)

---

and that's the whole page, bestie!! 💖 this one's Version 0.10.0, dated 25 December 2016, and it lives at gd_include(3) 🎀 go forth and include your fragments responsibly, ily 🦋💅