MTBL_FILESET(3)

🎀 NAME 🎀

mtbl_fileset — automatic multiple MTBL data file merger

okay so hi bestie 💖 you're reading the manual page for mtbl_fileset(3), and honestly? she's a baddie. she merges a whole little squad of MTBL data files for you, automatically, no cap.

---

✨ SYNOPSIS ✨

#include <mtbl.h>

Fileset objects:

struct mtbl_fileset *
mtbl_fileset_init(const char *fname, const struct mtbl_fileset_options *fopt);

struct mtbl_fileset *
mtbl_fileset_dup(struct mtbl_fileset *orig, const struct mtbl_fileset_options *fopt);

void
mtbl_fileset_destroy(struct mtbl_fileset **f);

void
mtbl_fileset_reload(struct mtbl_fileset *f);

void
mtbl_fileset_reload_now(struct mtbl_fileset *f);

const struct mtbl_source *
mtbl_fileset_source(struct mtbl_fileset *f);

void
mtbl_fileset_partition(struct mtbl_fileset *f,
                mtbl_filename_filter_func cb,
                void *clos,
                struct mtbl_merger **m1,
                struct mtbl_merger **m2);

Fileset options:

struct mtbl_fileset_options *
mtbl_fileset_options_init(void);

void
mtbl_fileset_options_destroy(struct mtbl_fileset_options **fopt);

void
mtbl_fileset_options_set_merge_func(
        struct mtbl_fileset_options *fopt,
        mtbl_merge_func fp,
        void *clos);

void
mtbl_fileset_options_set_dupsort_func(
        struct mtbl_fileset_options *fopt,
        mtbl_dupsort_func fp,
        void *clos);

void
mtbl_fileset_options_set_filename_filter_func(
        struct mtbl_fileset_options *fopt,
        mtbl_filename_filter_func fp,
        void *clos);

void
mtbl_fileset_options_set_reader_filter_func(
        struct mtbl_fileset_options *fopt,
        mtbl_reader_filter_func fp,
        void *clos);

void
mtbl_fileset_options_set_reload_interval(
        struct mtbl_fileset_options *fopt,
        uint32_t reload_interval);

---

🌸 DESCRIPTION 🌸

so the mtbl_fileset is basically the convenience interface that keeps a merged view of a whole set of MTBL data files, automatically, like a little bestie doing your homework for you. 🫶 and you get to eat those merged entries through the mtbl_source(3) and mtbl_iter(3) interfaces. slay.

mtbl_fileset objects get initialized from a "setfile", which is literally just a list of filenames of MTBL data files, one per line. and here's the cute part, bestie: for each filename, if it is not an absolute path, then the directory name of the fileset file's pathname gets prepended to it. examples, because i know you love examples:

• if the setfile path is "/export/dnstable/mtbl/dns.fileset", and a line in it is of the form "dns.2017.Y.mtbl" then dnstable will try to open "/export/dnstable/mtbl/dns.2017.Y.mtbl".
• if the setfile path is "/export/dnstable/mtbl/dns.fileset", and a line in it is of the form "/tmp/dns.2017.Y.mtbl" then dnstable will try to open "/tmp/dns.2017.Y.mtbl".

internally, an mtbl_reader object is initialized from each filename and added to an mtbl_merger object. and the setfile? watched for changes 👀 so when filenames get added to or removed from the setfile, the corresponding mtbl_reader objects get added or removed too. it's giving automatic. it's giving low maintenance.

now, because the MTBL format does not allow duplicate keys, the caller must provide a function that will accept a key and two conflicting values for that key and return a replacement value. and ngl, this function may be called multiple times for the same key if the same key is inserted more than twice. so like, don't be weird about it. see mtbl_merger(3) for further details about the merge function.

mtbl_fileset objects are created with the mtbl_fileset_init() function, which requires the path to a "setfile", fname, and a non-NULL fopt argument which has been configured with a merge function fp. then you should call mtbl_fileset_source() in order to consume output via the mtbl_source(3) interface. that's the whole ritual bestie, you got this. 💅

the mtbl_fileset_dup() function, which requires a mtbl_fileset object and a non-NULL opt argument, creates a duplicate mtbl_fileset object that shares the underlying fileset, but with different fileset options, as specified. two filesets, one fileset-file. iconic.

accesses via the mtbl_source(3) interface will implicitly check for updates to the setfile. but sometimes you gotta be extra and explicitly call mtbl_fileset_reload() in order to check for updates — especially if files are being removed from the setfile and the mtbl_source is infrequently accessed. lowkey a trap, so watch out.

the mtbl_fileset_reload() function avoids checking for updates more frequently than every reload_interval seconds. and if reload_interval is set to MTBL_FILESET_RELOAD_INTERVAL_NEVER, then mtbl_fileset_reload() will only load the fileset once. meanwhile mtbl_fileset_reload_now() can be called to bypass the reload_interval check. no waiting, no patience, main character energy. ⭐

the mtbl_fileset_partition() function yields two struct mtbl_merger objects that are split based on the output of a callback. the caller is responsible for calling mtbl_merger_destroy() on each of these mergers. and here's the part where i hold your hand: calls to mtbl_source_*() on the fileset's source object, and calls to mtbl_fileset_reload() and mtbl_fileset_reload_now(), including those implicitly performed by operations on the fileset source, may leave these mergers in an inconsistent state. 😭 for this reason, mtbl_fileset_partition() use is deprecated in favor of mtbl_fileset_dup() with the fname_filter_func option set. periodt. use the new one, bestie.

---

💫 Fileset options 💫

• merge_func — see mtbl_merger(3). an mtbl_merger object is used internally for the external sort.
• dupsort_func — see mtbl_merger(3). used to sort the entries with duplicate keys during the merge process based on their data.
• fname_filter_func — used to filter specific files by name from a fileset. if the function returns false, the file's data will not be included in the results returned by any iterators on the fileset. bye bestie ✋
• reader_filter_func — used to filter specific readers from a fileset. if the function returns false, the reader's data will not be included in the results returned by any iterators on the fileset.
• reload_interval — specifies the interval between checks for updates to the setfile, in seconds. defaults to 60 seconds. MTBL_FILESET_RELOAD_INTERVAL_NEVER is a special value that indicates to never reload the fileset. 🦋

---

okay that's the whole page bestie, you ate that 💖 this one's dated 11/28/2022, straight out of the mtbl manual — go forth and merge, you gorgeous little sysadmin. 🎀✨