FIFOLOG(1)

🦋 FIFOLOG(1) 🦋

okay so bestie, this is the FIFOLOG(1) page from the General Commands Manual, and we’re about to make fifologs make sense. No cap, it’s giving “compact round-robin circular storage” energy already. 💖

✨ NAME ✨

fifolog_create, fifolog_writer, fifolog_reader — initialize, write, seek and extract data from a fifolog.

📜 SYNOPSIS 📜

       fifolog_create [-l record-size] [-r record-count] [-s size] file
       fifolog_reader [-t] [-b tstart] [-B Tstart] [-e tend] [-E Tend]
                      [-o ofile] [-R regexp] [-T timefmt] file
       fifolog_writer [-w write-rate] [-s sync-rate] [-z compression] file

💖 DESCRIPTION 💖

Fifologs provide a compact round-robin circular storage for recording text and binary information to permanent storage in a bounded and predictable fashion, time and space wise. literally bounded and predictable, she said what she said. ✨

A fifolog can be stored either directly on a disk partition or in a regular file. iconic, flexible, lowkey versatile. 🦋

The input data stream is encoded, compressed and marked up with timestamps before it is written to storage, such that it is possible to seek out a particular time interval in the stored data, without having to decompress the entire logfile. that’s the whole flex, bestie. 🔥

🛠️ fifolog_create — the one-time setup girlie

The fifolog_create utility is used to initialize the first sector of a disk device or file system file to make it a fifolog and should be called only once. yes, only once — it’s not a “run it again for fun” situation. 🙈

Running fifolog_create on an existing fifolog will reset it so that fifolog_reader and fifolog_writer will not see the previous contents. (The previous contents are not physically erased, and with a bit of hand-work all but the first record can be easily recovered.) so yeah, it’s a reset vibe — keep that in mind 💀

If the file does not already exist, fifolog_create will attempt to create and ftruncate(2) it to the specified size, defaulting to 86400 records of 512 bytes if the -r, -l or -s options do not specify otherwise.

The flags on deck are:

• -l record-size — the record-size knob from the synopsis; it’s part of the size/default story.
• -r record-count — the record-count knob from the synopsis; same default clause applies.
• -s size — the size knob from the synopsis; same default clause applies.
• file — the disk device or file system file that becomes the fifolog.

✍️ fifolog_writer — the main character writing the tea ☕

The fifolog_writer utility will read standard input and write it to the end of the fifolog according to the parameters given. it takes file plus these flags:

• -w write-rate — the first and faster timer. It forces available data to be written but does not flush and reset the compression dictionary. This timer is intended to minimize the amount of logdata lost in RAM in case of a crash, and by default it fires 10 seconds after the previous write.
• -s sync-rate — the second and slower timer. It forces a full flush and reset of the compression engine and causes the next record written to be a synchronization point with an uncompressed timestamp, making it possible to start reading the logfile from that record. By default this timer fires a minute after the previous sync.
• -z compression — controls the zlib(3) compression level; legal values are zero to nine which is the default.

Writes happen whenever the output buffer is filled with compressed data or when either of two timers expire, forcing a partially filled buffer to be written. buffer said “I’m full” or timer said “bestie, it’s time.” 💅

🔎 fifolog_reader — the seek-and-extract bestie

The fifolog_reader utility will retrieve records from the fifolog according to the specified parameters and write them either to standard output or the file specified with -o. it takes file plus these flags:

• -t — forces timestamps to be formatted as YYYYMMDDhhmmss instead of as time_t.
• -T timefmt — allows the specification of an strftime(3) formatting string.
• -b tstart — lower-case start time; takes a time_t value.
• -B Tstart — upper-case start time; takes a human-readable specification such as “1 hour ago”.
• -e tend — lower-case end time; takes a time_t value.
• -E Tend — upper-case end time; takes a human-readable specification.
• -o ofile — writes to the file specified with -o instead of standard output.
• -R regexp — filters so that only records matching the (REG_BASIC) regular expression specified with -R are output.

It is possible to specify a start and end time to limit the amount of data fifolog_reader will report. the lower-case variants -b and -e take a time_t value, whereas the upper-case variants -B and -E take human-readable specifications such as “1 hour ago”. 🕰️

🧠 IMPLEMENTATION NOTES 🧠

The data stored in the fifolog consists of three layers, an outer layer that allows searches to synchronization points based on timestamps without having to decompress and decode the actual contents, a compression layer implemented with zlib(3), and an inner serialization and timestamping layer. the exact encoding is described in the fifolog.h file. three layers, one iconic little log format. ✨

Fifolog is particularly well suited for use on Flash based media, where it results in much lower write-wear, than a file system with regular log files rotated with newsyslog(8) etc. 🌸

🌈 EXAMPLES 🌈

Create a fifolog with 1024*1024 records of 512 bytes:

fifolog_create -r 10m /tmp/fifolog

Write a single record to this file:

date | fifolog_writer /tmp/fifolog

Read it back with human readable timestamps:

fifolog_reader -t /tmp/fifolog

One particular useful use of fifolog_writer is with syslogd(8) using a line such as this in syslog.conf(5):

*.* |fifolog_writer /var/log/syslog_fifolog

🕰️ HISTORY 🕰️

The fifolog tools have been liberated from an open source SCADA applications called “measured”, which monitors and controls remote radio navigation transmitters for the Danish Air Traffic Control system. lore drop: air traffic control radio navigation realness. 🛫

👑 AUTHORS 👑

The fifolog tools were written by Poul-Henning Kamp. bow down, bestie. 👑

---

💖 okay bestie, that’s the whole FIFOLOG(1) page — brought to you by the Debian source/version and dated February 9, 2008, so take that info and slay your fifolog journey. mwah 💅🎀