gd_getdata(3)

okay bestie, welcome to the gd_getdata(3) page, straight from the GETDATA section of manual-land. This function is literally the data-fetching main character for a Dirfile database, and she is iconic. 🦋

✨ NAME ✨

gd_getdata — retrieve data from a Dirfile database. That's the whole name vibe: she gets data. 💅

---

🎀 SYNOPSIS 🎀

#include <getdata.h>

size_t gd_getdata(DIRFILE *dirfile, const char *field_code, off_t
       first_frame, off_t first_sample, size_t num_frames, size_t
       num_samples, gd_type_t return_type, void *data_out);

🌸 DESCRIPTION 🌸

The gd_getdata() function queries a dirfile(5) database specified by dirfile for the field field_code. It fetches num_frames frames plus num_samples samples from this field, starting first_sample samples past frame first_frame. The data is converted to the data type specified by return_type, and stored in the user-supplied buffer data_out. 💖

The field_code may contain one of the representation suffixes listed in dirfile-format(5). If it does, gd_getdata() will compute the appropriate complex norm before returning the data. No cap. ✨

The dirfile argument must point to a valid DIRFILE object previously created by a call to gd_open(3). The argument data_out must point to a valid memory location of sufficient size to hold all data requested. do not play with that buffer size, bestie. 🫶

Unless using GD_HERE (see below), the first sample returned will be

first_frame * samples_per_frame + first_sample

as measured from the start of the dirfile, where samples_per_frame is the number of samples per frame as returned by gd_spf(3). The number of samples fetched is, similarly,

num_frames * samples_per_frame + num_samples.

Although calling gd_getdata() using both samples and frames is possible, the function is typically called with either num_samples and first_sample, or num_frames and first_frames, equal to zero. (Yes, first_frames. That's what the page says. 👀)

Instead of explicitly specifying the origin of the read, the caller may pass the special symbol GD_HERE as first_frame. This will result in the read occurring at the current position of the I/O pointer for the field (see GetData I/O Pointers below for a discussion of field I/O pointers). In this case, the value of first_sample is ignored. 💀

When reading a SINDIR field, return_type must be GD_STRING. For all other field types, the return_type argument should be one of the following symbols, which indicates the desired return type of the data:

• GD_UINT8 — unsigned 8-bit integer. periodt.
• GD_INT8 — signed (two's complement) 8-bit integer. yes.
• GD_UINT16 — unsigned 16-bit integer. slay.
• GD_INT16 — signed (two's complement) 16-bit integer. okay.
• GD_UINT32 — unsigned 32-bit integer. cute.
• GD_INT32 — signed (two's complement) 32-bit integer. noted.
• GD_UINT64 — unsigned 64-bit integer. big unsigned energy.
• GD_INT64 — signed (two's complement) 64-bit integer. big signed energy.
• GD_FLOAT32 — IEEE-754 standard 32-bit single precision floating point number. fancy.
• GD_FLOAT64 — IEEE-754 standard 64-bit double precision floating point number. even fancy.
• GD_COMPLEX64 — C99-conformant 64-bit single precision complex number. complex baddie.
• GD_COMPLEX128 — C99-conformant 128-bit double precision complex number. double complex baddie.
• GD_NULL — the null type: the database is queried as usual, but no data is returned. In this case, data_out is ignored and may be NULL. iconic.

The return type of the data need not be the same as the type of the data stored in the database. Type conversion will be performed as necessary to return the requested type. If the field_code does not indicate a representation, but conversion from a complex value to a purely real one is required, only the real portion of the requested vector will be returned. 🧚

Upon successful completion, the I/O pointer of the field will be on the sample immediately following the last sample returned, if possible. On error, the position of the I/O pointer is not specified, and may not even be well defined. so like, error handling? yes. 😭

🌈 Behaviour While Reading Specific Field Types 🌈

🔥 MPLEX 🔥

Reading an MPLEX field typically requires GetData to read data before the range returned in order to determine the value of the first sample returned. This can become expensive if the encoding of the underlying RAW data does not support seeking backwards (which is true of most compression encodings). How much preceding data GetData searches for the initial value of the returned data can be adjusted, or the lookback disabled completely, using gd_mplex_lookback(3). If the initial value of the field is not found in the data searched, GetData will fill the returned vector, up to the next available sample of the mulitplexed field, with zero for integer return types, or IEEE-754-conforming NaN (not-a-number) for floating point return types, as it does when providing data before the beginning-of-field. 💅

GetData caches the value of the last sample from every MPLEX it reads so that a subsequent read of the field starting from the following sample (either through an explicit starting sample given by the caller or else implicitly using GD_HERE) will not need to scan the field backwards. This cache is invalidated if a different return type is used, or if an intervening operation moves the field's I/O pointer. cache girlie is sensitive. ✨

🦋 SINDIR 🦋

The only allowed return_type when reading SINDIR data is GD_STRING. The data argument should be of type const char **, and be large enough to hold one pointer for each sample requested. It will be filled with pointers to read-only string data. The caller should not free the returned string pointers. For convenience when allocating buffers, the GD_STRING constant has the property: GD_SIZE(GD_STRING) == sizeof(const char *). On samples where the index vector is out of range of the SARRAY, and also on samples before the index vector's frame offset, the value stored in data will be the NULL pointer. 🫶

🌸 PHASE 🌸

A forward-shifted PHASE field will always encounter the end-of-field marker before its input field does. This has ramifications when reading streaming data with gd_getdata() and using gd_nframes(3) to gauge field lengths (that is: a forward-shifted PHASE field always has less data in it than gd_nframes(3) implies that it does). As with any other field, gd_getdata() will return a short count whenever a read from a PHASE field encounters the end-of-field marker. 😭

Backward-shifted PHASE fields do not suffer from this problem, since gd_getdata() pads reads past the beginning-of-field marker with NaN or zero as appropriate. Database creators who wish to use the PHASE field type with streaming data are encouraged to work around this limitation by only using backward-shifted PHASE fields, by writing RAW data at the maximal frame lag, and then back-shifting all data which should have been written earlier. Another possible work-around is to write systematically less data to the reference RAW field in proportion to the maximal forward phase shift. This method will work with applications which respect the database size reported by gd_nframes(3) resulting in these applications effectively ignoring all frames past the frame containing the maximally forward-shifted PHASE field's end-of-field marker. slay workaround energy. 👑

💜 WINDOW 💜

The samples of a WINDOW for which the field conditional is false will be filled with either zero for integer return types, or IEEE-754-conforming NaN (not-a-number) for floating point return types. 🎀

💖 RETURN VALUE 💖

okay so here's the vibe: no matter what happens, gd_getdata() hands you back the number of samples it actually read from the database. samples, bestie, not bytes — don't get it twisted. 🫶

and if it hits the end-of-field before it's grabbed all the samples you asked for? you just get a short count. that's not an error, that's just the data running out. periodt.

now — if you ask for data from before the beginning-of-field marker (which, heads up, might have been shifted off frame zero by a PHASE field or the /FRAMEOFFSET directive), the library pads the front with NaN or zero. which one? depends on whether your return type is floating point or integral. iykyk. ✨

on error tho, this function returns zero and tucks a negative-valued error code into the DIRFILE object, which you can grab later with gd_error(3). the possible error codes are:

• GD_E_ALLOC — the library literally could not allocate memory. 💀
• GD_E_BAD_CODE — the field named by field_code, or one of the fields it uses for input, was not found in the database. ghosted fr.
• GD_E_BAD_DIRFILE — you handed over an invalid dirfile. not it.
• GD_E_BAD_SCALAR — a scalar field used in the field's definition wasn't found, or wasn't actually scalar type.
• GD_E_BAD_TYPE — the return_type you specified is invalid. nope.
• GD_E_DIMENSION — the field_code you gave referred to a CONST, CARRAY, or STRING field. bestie, you want gd_get_constant(3) or gd_get_string(3) for those. (or: a scalar field showed up where a vector field was expected in the definition of field_code or one of its inputs.)
• GD_E_DOMAIN — an immediate read was attempted using GD_HERE, but the field's I/O pointer wasn't well defined because two or more of the field's inputs disagreed about where the I/O pointer is. messy. 😭
• GD_E_INTERNAL_ERROR — something broke inside the library while doing the thing. this is a bug in the library, no cap. please report the incident to the maintainer. 🙈
• GD_E_IO — an error happened while trying to open or read from a file on disk containing a raw field or LINTERP table.
• GD_E_LUT — a LINTERP table was malformed. cursed.
• GD_E_RANGE — an attempt was made to read data outside the addressable Dirfile range (more than 2**63 samples past the start of the dirfile). that's a lot of samples, and also, no.
• GD_E_RECURSE_LEVEL — too many levels of recursion while trying to resolve field_code. usually this means there's a circular dependency in the field specification in the dirfile. 🌀
• GD_E_UNKNOWN_ENCODING — the encoding scheme of a RAW field could not be determined. this may also mean the binary file associated with the RAW field couldn't be found.
• GD_E_UNSUPPORTED — reading from dirfiles with the encoding scheme of the specified dirfile isn't supported by the library. see dirfile-encoding(5) for the deets on dirfile encoding schemes.

and if you want an actual descriptive error string? call gd_error_string(3). 💅

---

🌸 NOTES 🌸

okay important stuff, pay attention:

to save memory, gd_getdata() uses the memory pointed to by data_out as scratch space while it computes derived fields. so if an error pops off during that computation, the contents of that memory buffer are unspecified — and may have already been modified by this call — even though gd_getdata() will report zero samples returned on error. so like… don't trust that buffer after a fail. 🙈

reading slim-compressed data (see defile-encoding(5)) may cause unexpected memory usage. why? because slimlib internally caches open decompressed files as they're read, and GetData doesn't close data files between gd_getdata() calls — for efficiency's sake, bestie. the memory used by that internal slimlib buffer can be reclaimed by calling gd_raw_close(3) on fields when you're done reading them. 🧹

also, when you're on a platform whose size_t is N-bytes wide, a single call of gd_getdata() will never return more than (2**(N-1) - 1) samples. the request gets truncated at (2**(N-M) - 1) samples, where M is the size, in bytes, of the largest data type used to calculate the returned field. if you ask for more than that, you'll get less data than requested, and it will not raise an error. and this limit applies even when return_type is GD_NULL or when reading from the INDEX field (i.e. even when no actual I/O or calculation happens). in all cases, the actual amount of data is returned. so check your counts. 👑

---

🧚 GETDATA I/O POINTERS 🧚

this bit is a general discussion of field I/O pointers in the GetData library, and it contains info that's not directly applicable to gd_getdata(). still eating tho. 🍓

every RAW field in an open Dirfile has an I/O pointer which indicates the library's current read and write position in the field. these I/O pointers are useful when you're doing sequential reads or writes on Dirfile fields (see GD_HERE in the description above). the value of a field's I/O pointer is reported by gd_tell(3). ✨

derived fields have virtual I/O pointers, arising from the I/O pointers of their input fields. these virtual I/O pointers may be valid (when all input fields agree on their position in the dirfile) or invalid (when the input fields are not in agreement). the I/O pointer of some derived fields is always invalid. the usual reason? the derived field is simultaneously reading from two different places in the same RAW field. for example, given this Dirfile metadata specification:

a RAW UINT8 1
b PHASE a 1
c LINCOM 2 a 1 0 b 1 0

the derived field c never has a valid I/O pointer, since any particular sample of c ultimately involves reading from more than one place in the RAW field a. attempting sequential reads or writes (with GD_HERE) on a derived field when its I/O pointer is invalid will result in an error — specifically, GD_E_DOMAIN. 😭

the implicit INDEX field has an effective I/O pointer that mostly behaves like a true RAW field I/O pointer, except that it permits simultaneous reads from multiple locations. so, given this metadata specification:

d PHASE INDEX 1
e LINCOM 2 INDEX 1 0 d 1 0

the I/O pointer of the derived field e will always be valid, unlike the similarly defined c above. iconic. 👑

the virtual I/O pointer of a derived field changes in response to movement of the RAW I/O pointers underlying the derived field's inputs, and vice versa: moving the I/O pointer of a derived field will move the I/O pointer of the RAW fields from which it ultimately derives. as a result, the I/O pointer of any particular field may move in unexpected ways if multiple fields are manipulated at the same time. be aware. 🌀

when a Dirfile is first opened, the I/O pointer of every RAW field is set to the beginning-of-frame (the value returned by gd_bof(3)), as is the I/O pointer of any newly-created RAW field.

the following library calls cause I/O pointers to move:

• gd_getdata() and gd_putdata(3) — these functions move the I/O pointer of affected fields to the sample immediately following the last sample read or written, both when performed at an absolutely specified position and when called for a sequential read or write using GD_HERE. when reading a derived field which simultaneously reads from more than one place in a RAW field (such as c above), the position of that RAW field's I/O pointer is unspecified (that is: it is not specified which input field is read first).
• gd_seek(3) — this function is used to manipulate I/O pointers directly.
• gd_flush(3) and gd_raw_close(3) — these functions set the I/O pointer of any RAW field which is closed back to the beginning-of-field.
• calls which result in modifications to raw data files — this may happen when calling any of: gd_alter_encoding(3), gd_alter_endianness(3), gd_alter_frameoffset(3), gd_alter_entry(3), gd_alter_raw(3), gd_alter_spec(3), gd_malter_spec(3), gd_move(3), or gd_rename(3); these functions close affected RAW fields before making changes to the raw data files, and so reset the corresponding I/O pointers to the beginning-of-field.

in general, when these calls fail, the I/O pointers of affected fields may be anything — even out-of-bounds or invalid. after an error, the caller should issue an explicit gd_seek(3) to reposition I/O pointers before attempting further sequential operations. fr fr, don't skip this. 🫶

---

⭐ HISTORY ⭐

• the function getdata() appeared in GetData-0.3.0. ✨
• the GD_COMPLEX64 and GD_COMPLEX128 data types appeared in GetData-0.6.0. 💅
• in GetData-0.7.0, this function was renamed to gd_getdata(). glow-up.
• the GD_HERE symbol used for sequential reads appeared in GetData-0.8.0. 🦋
• the GD_STRING data type appeared in GetData-0.10.0. 👑

---

👑 SEE ALSO 👑

GD_SIZE(3), gd_error(3), gd_error_string(3), gd_get_constant(3), gd_get_string(3), gd_mplex_lookback(3), gd_nframes(3), gd_open(3), gd_raw_close(3), gd_seek(3), gd_spf(3), gd_putdata(3), dirfile(5), dirfile-encoding(5)

---

and that's the whole page, bestie 💖 this was Version 0.10.0, dated 25 December 2016, and the manpage itself is gd_getdata(3). go read some data, you absolute icon. 🎀✨