slim(1)

📦 version 2.7.1, in fedora 43, fedora 44

slim(1) — PHYSICS DATA COMPRESSION 💖

okay bestie, buckle up, because this is the manual page for slim, unslim and slimcat, and honestly? it's giving very niche but very iconic. 💅

✨ NAME ✨

slim, unslim, slimcat — reversible compression of noisy physics data. 🦋

🌸 SYNOPSIS 🌸

slim [options] filename ...
unslim [options] filename ...
slimcat [options] filename ...

💜 DESCRIPTION 💜

slim does lossless compression on binary data files. no cap. and the whole vibe is that it's built to go very rapidly and to beat general-purpose tools like gzip and bzip2 when it comes to noisy physics data. it's not trying to be a universal compressor, it's trying to be the compressor for one specific kind of file, and it ate. 🔥

real talk though: slim only works well on very specific kinds of data. it needs a file that's made of integers in their native 16- or 32-bit binary format, arranged in a regular repetitive pattern. that's the whole requirement, that's the assignment. specifically, slim was designed for the kind of enormous files produced copiously by the Atacama Cosmology Telescope. 🛰️ if it happens to work well on other types of files from other experiments, then like… so much the better, we love a bonus. 💖

ngl, slim is not now appropriate for self-describing files that carry their own meta-data (thinking of things like FITS or NetCDF files). 🙈 probably the slim library could be adapted to those files, if someone wrote a front end to parse the meta-data and pull out the details about the structure of the main data. so it's not a never, it's a "not today, bestie."

okay here's the thing that trips people up: even though slim can operate on any file — in the sense that the "slimfile" will correctly expand back into the original file, like it always round-trips — getting good compression performance, or honestly any compression at all, requires that slim actually knows specific details about what's inside that file. 👑 it's not psychic, it needs the tea. check out the Data description options for how to describe the file contents, and RAW DATA STRUCTURE for the explanation of the relevant terms. iykyk.

and here's a lil secret: slim "trains" its encoding algorithms by examining only a fraction of the data. it's sampling, not reading everything, because speed is the moment. ⭐ for example, the main algorithm — the reduced binary encoder — works by compressing a small contiguous range of values, at the expense of all values outside that range. 😭 and the ends of that range get selected based on only a sample of the data. so it's fast, but it's committing to a guess. for more on that, see Data sampling options.

lastly, the cute aliases, because this is the part everyone forgets:

  • unslim is a synonym for slim --expand
  • slimcat is a synonym for slim --stdout --preserve

periodt. 💅

✨ OPTIONS ✨

okay so bestie, every single option has a short form and a long form, and they're all listed together right here. no cap.

🎀 General options 🎀

  • -p, --preserve — Keep the original raw file when you're compressing, or keep the original slim file when you're expanding. The default vibe is to delete the input file(s) as each output file gets successfully compressed or expanded (except when the option --stdout is used). If you pick this while expanding files, the new output file is named by replacing the ".slm" extension with ".raw".
  • -X, --compress — The file or files you give as arguments are raw files to be compressed. The default is to assume raw files, except where the file names end with the .slm or .SLM suffix. This option is the only way to force compression of a file named something.slm.
  • -x, --expand — The file or files you give as arguments are slim files to be expanded. The default is to assume raw files, except where the file names end with the .slm or .SLM suffix.
  • -k, --force — Force compression or expansion to overwrite any existing file of the same name. Without this option, slim will refuse to overwrite any files.
  • -V, --version — Print the slim file version and exit.
  • -q, --quiet — Don't print any output unless there are errors.
  • -?, --help — Print a usage message and exit.

🔥 Compression options 🔥

These options are ignored when expanding files. periodt.

  • -m, --method=method — Use encoding method method (default = 2). The methods are:
    • -m2 — The reduced binary encoder (default)
    • -m5 — Runlength coder. Good for data where the values are strictly identical for long periods. All other values are reserved. Methods numbered 1, 3, and 4 were once implemented. They all proved wrong for the job and have been removed. Method 1 was a slight variation on the reduced binary system, differing only in how the parameters were computed. For more on methods 2 through 5, see ENCODING ALGORITHMS.
  • -d, --deltas — Encoders operate on the differences between successive data values (the deltas). Default is to operate on the data itself. Deltas are slightly slower to encode and decode, but they compress more effectively in almost all situations. In principle, raw data could compress to 0.5 fewer bits per data value, but this would be expected only for extremely stable channels that are not typical of physics data.
  • -b, --permit-bitrotation — Bit-rotation is an encoding feature useful in some data streams where the lowest n bits are always the same. When using this option, slim will check each channel for the existence of some number of constant lowest bits. If any are found, then the data are compressed by "rotating" the lowest bits to the top of the number and then proceeding with the usual methods. The author finds this useful with some analog-to-digital schemes that have several low-order bits fixed at zero. For more, see Bit rotation below.
  • -n, --filename — Save the original filename in the slim file header. By default, slim does not include the filename in the slim file.
  • -S, --rawsize — Save the original file's size (in bytes) in the slim file header. (Currently, this is on by default and cannot be turned off.) Note that when expanding, the raw file size is not tested against the size stored in the file header.
  • -C, --compute-crc32 — On compression, compute a CRC-32 checksum on the raw data in each section and include its value in the slim file. Default is not to compute the CRC. (On expansion, slim will compute and test the checksum if it is present in the slim file, with or without this option set. However, also see --ignore-crc32 )

💅 Expansion options 💅

These options are ignored when compressing files. iykyk.

  • -o, --stdout — Write the results of expansion to stdout instead of to a file or files. This option implies --preserve and is equivalent to running slimcat
  • -0, --ignore-crc32 — On expansion, do not compute and test the CRC-32 checksum, even if one is included in the slim file.

🦋 Data description options 🦋

The current version of slim requires that all data channels be the same (i.e. of the same raw type, having the same number of repeats per frame, and so forth). Therefore, when given, all of the options in this section apply to all channels. These requirements are not limitations inherent to the slim file specification, but only to the current implementation of slim. In the future, it could be possible to slim a more heterogeneous mix of data channels, if a configuration file parser were to be added. For more on the differences between the current restrictions on compression and the full generality of the expander, see More general slim data structures.

Data types are assumed to be 1-, 2- or 4-byte integers with little-endian byte order. (Single-byte integer types cannot currently be specified at the command line.) There are no immediate plans to add facilities for handling 8-byte integers or floating-point numbers to slim (although command-line arguments have been reserved for the floating-points).

See RAW DATA STRUCTURE for more information about how the raw data is assumed to be structured, and for the definitions of the terms channels, sections, frames, and repeats.

All data description options apply to compression only; they are all ignored when expanding a slim file.

  • -c, --num-chan=nc — The data consist of nc separate data channels.
  • -r, --repeats=reps — One frame contains reps consecutive repeated data values for each channel.
  • -F, --frames=nf — There are nf frames in each data section. Note that slim is allowed to break up these frames into multiple sections as needed to keep the section size below the hard limit of 16 MB, or for any other reason. However, slim will also enforce a change to a new section at the end of every nf frames with this option.
  • -i, --int — All data are signed 32-bit integers (the default is signed 32-bit).
  • -u, --unsigned — All data are unsigned 32-bit integers.
  • -s, --short — All data are signed 16-bit integers. Some aspects of slim are not implemented for short integers, such as bit rotation. See BUGS.
  • -v, --ushort — All data are unsigned 16-bit integers. Some aspects of slim are not implemented for short integers, such as bit rotation. See BUGS.
  • -y, --char — All data are signed 8-bit integers.
  • -f, --float — All data are 32-bit floating point numbers using the IEEE-754 standard. Currently no floating-point compression schemes are implemented, and this option is merely reserved for future use. Instead, 32-bit floating point data are treated as signed 32-bit integers.
  • -g, --double — All data are 64-bit floating point numbers using the IEEE-754 standard. Currently no floating-point compression schemes are implemented, and this option is merely reserved for future use. Instead, 64-bit data are stored without compression.

✨ Data sampling options ✨

The main encoder system (the reduced binary method) works by sampling only some fraction of the raw data and assuming that the statistical properties of the sample are adequate to predict the behavior of the entire set. The data sampling options govern what fraction of the data are used in the sampling. These control the trade-off between speed and compression ratio.

Experience suggests that sampling a greater fraction of the data offers very little benefit; we suggest sampling no more than 10% of all values (the default), unless you plan to do (or have done) a careful test of speed and compression ratio.

Note that the encoders are all universal, in the sense that all possible values can be encoded (though of course most values will not be compressed to a smaller size). This means that the occasional unusual data value can be encoded, and sparse sampling is not fatal to the resulting slim file.

Sampling can be done on a maximum of 20,000 data values. All sample options refer to percentages of either this maximum or the full number of available values, whichever is smaller. Regardless of the options given, a minimum of 20 data values or all values will be sampled, whichever is smaller.

  • -G, --sample-pct=pct — Sample pct percent of the data (between 2 and 100%) up to a maximum of 200*pct samples when determining the encoding parameters. Default is 10% up to 2000 samples.
  • -9, --18-pct, --best — Sample 18% of the data (up to 3600 samples) to determine encoding parameters.
  • -8, --16-pct — Sample 16% of the data (up to 3200 samples) to determine encoding parameters.
  • -7, --14-pct — Sample 14% of the data (up to 2800 samples) to determine encoding parameters.
  • -6, --12-pct — Sample 12% of the data (up to 2400 samples) to determine encoding parameters.
  • -5, --10-pct — Sample 10% of the data (up to 2000 samples) to determine encoding parameters. (This is the default.)
  • -4, --8-pct — Sample 8% of the data (up to 1600 samples) to determine encoding parameters.
  • -3, --6-pct — Sample 6% of the data (up to 1200 samples) to determine encoding parameters.
  • -2, --4-pct — Sample 4% of the data (up to 800 samples) to determine encoding parameters.
  • -1, --2-pct, --fast — Sample 2% of the data (up to 400 samples) to determine encoding parameters.

👑 Debugging and unimplemented features 👑

  • -B, --debug-buffer=bufsize — Read or write raw data from a debugging buffer of size bufsize bytes. This is not the optimal way to compress or expand files (though the price is small). This option is only intended to exercise and debug parts of the code that would be used in a planned slimlib library of functions.

💖 RAW DATA STRUCTURE 💖

okay bestie, before we go any further, we HAVE to talk about the ✨fundamental concept✨ of slimming a raw data file, and that is… the data channel. 🎀

one channel would normally correspond to one physical data source, like a single thermometer, or an encoder, or literally any other sensor you can think of. and here's the tea: for compression to actually be good, the distribution of the data coming out of a single channel needs to have a single mode, its statistical properties need to be more or less stationary, the rate of extreme outliers needs to be small (like, less than a few percent), and so forth. that's just the vibe check it has to pass. 💅

data from more than one channel can totally be mixed together in the raw file, but ONLY if the pattern of changes from one channel to another gets repeated consistently. the most general pattern permitted is to have M samples of channel 1, followed by N samples of channel 2, and so on and so forth until all of the channels in the raw data file have appeared. this unit of the raw file — where each channel occurs one time and gives its expect number of samples — is called the frame. 👑

and multiple frames? they make up a section. the raw data file is assumed to consist of one or more sections concatenated together. the number, type, repetition count, and order of the channels is fixed throughout a single section (and in the current version of the slim executable, there's literally no way to change these factors between sections either 😭). a section consists of one or more frames, and it IS valid for the last section in a file to end with a fractional frame — which we assume would happen because of some unexpected event, like a raw data file getting truncated during acquisition or transmission. 🙈

okay now this next bit is lowkey important so pay attention bestie. 🫶

when a raw file contains only one channel, there's an ambiguity in whether that channel is being repeated N times within a single frame, or if it's repeated only once in each of N frames. and because the first choice leads to faster execution, slim silently selects it even if your command-line options are calling for the second one. so bestie, the options

--num-chan=1 --repeats=1 --frames=20000

get silently converted to

--num-chan=1 --repeats=20000 --frames=1

no cap, it just does that. ✨

note that if one or both of the options giving the repeats or the frames are absent, then slim tries to do the smartest possible thing. you probably don't want to let it do that, but you can. 💀

🌸 More general slim data structures 🌸

the slim executable produces slim files with several restrictions that are not inherent to the definition of the slim file format itself. the expander can expand files meeting the more general specification, but there is no way at this time to construct these more general files. it's giving "we can read it but we can't write it" 😭

so here's a lil list of some ways the structure could be less restrictive than the current compression executable permits:

  • the data type of different channels need not be the same 👯
  • the number of repeats can also be different
  • the list of channels could change between one section and the next
  • the slim file specification allows all sections — not only the last — to be of arbitrary size

however, the current implementation of the slim executable does not offer a way to break up the raw data into sections at arbitrary places in the raw file. so that's a no for now bestie. 🙈

currently, slim enforces a limit that sections be no longer than 16 MB (2^24 bytes), because of the implementation detail that sections must be held in memory when compressing or expanding them. 🧠 if data comprising what is conceptually one section exceeds this limit, then slim silently divides it into two or more sections of nearly equal length and writes them one after another.

this fact is not generally relevant to the user, but it does mean that compression will not be very effective if there are channels that repeat only a few times per 16 MB of raw data. so like… keep that in mind. 💅

🌸 ENCODING ALGORITHMS 🌸

okay bestie, so slim uses a few different algorithms to turn raw data into generally smaller data. and yes, "generally" is doing a lot of work there — no system can convert all possible 32-bit values into smaller ones, and that's provable with a simple counting argument. it's giving pigeonhole principle, honestly. 💅

so the goal of slim is to recognize a small subset of values that show up most often, and map only that subset into codes that need fewer bits. every other data value outside that small subset gets expanded into more bits than it started with. but if most or all of your raw values come from the compressible subset, then the encoded data ends up smaller than the raw data. it's a whole tradeoff situation. 🦋

also important: all encoding methods currently used in slim convert a single value into an exact integer number of bits (unlike range or arithmetic coding). so each bit in a slim file is part of (or all of) the code for a single value in the raw file — unless of course it belongs to the file's header data. periodt.

the three encoding methods currently implemented are:

✨ Constant-value encoding ✨

a channel that contains exactly the same value for every instance gets encoded as a constant-value. this system compresses each value to zero bits, apart from recording the constant value in the section header. no command-line option is required for constant-value encoding: all channels get checked to see if their values are strictly constant, and if so, they get encoded by this method in preference over all others. it's the main character of compression, fr fr. 👑

✨ Run-length encoding ✨

for a channel whose values are strictly constant for long runs but not for an entire data section, run-length encoding is ideal. for example, it works great on a channel storing the integer part of the time (in seconds), if the time is sampled many times per second. this method stores a repeated run of a single value as a (value, count) pair. 💖

two cautions though, bestie, don't skip these:

  1. if the data sample used for evaluation shows that run-length encoding will not actually compress the data channel, then that channel will silently switch over to the standard reduced-binary encoding, and
  2. the longest possible run is the number of repetitions within a frame — for technical reasons, a "run" cannot cross from one frame to another. so if a channel's values do not appear as several successive words in the raw file, then run-length encoding is not a good choice.

✨ Reduced-binary encoding ✨

this is the default method, the one you're gonna meet most. the reduced-binary encoder has two parameters:

  • the number of bits N used for storing the "normal" data, and
  • the offset s subtracted from each value before encoding.

the idea is to choose the parameters so that most or all values lie in the range [ s, s+(2^N)-2 ] and can therefore be stored using only N bits. the value s+(2^N)-1 is reserved to indicate "Overflow" — meaning the value being encoded did not lie in the normal range. overflow codes are followed by the raw data value itself, stored in its natural length of 32 or 16 bits. so most 32-bit data get stored as N-bit numbers, while a small fraction require N+32 bits. slay. 🎀

the parameters are chosen using the data sample set. all possible values are tested for N, while s is varied to keep the arithmetic mean of the sample data in the middle of the normal range for the given value of N. the choice of N is that which gives the best compression on the sample data set. and ngl, keeping the mean in the middle of the range might not always be appropriate for every possible distribution — but that's how it is done. 💀

🧚 Tested but rejected encoding algorithms 🧚

three other algorithms were implemented into slim and later removed. each of them improves on the compression ratios of the reduced-binary code, but the improvements are small and come at a price in compression and expansion speed (not to mention program complexity). development on these algorithms was dropped for this reason, and we mention them in case you are also pondering ideas for improved encoding algorithms in slim. consider this your lil history lesson. 📜

💜 Codes A and B

Codes A and B are minor variations on the reduced-binary code. the former changes how the presence of overflows is signaled, and the latter also changes how their overflowing values are stored.

Code A arises from the observation that overflow values might be rare, but they are still more common than any other single value. the idea is to encode most normal values as an N-bit number but the overflow as an m-bit number, where m<N. the result is to make each overflow take up (N-m) fewer bits, while the allowed range of values contains only 2^(N)-2^(N-m) values — slightly fewer than it would have had otherwise. this results in savings of approximately 0.15 to 0.20 bits per channel for normally-distributed data. however, the method was removed from slim because of its CPU cost: decoding each normal value requires first reading m bits, testing whether they are the overflow code, and (if not) then reading in an additional (N-m) bits to find the full N-bit number. exhausting, honestly. 😭

Code B expands on Code A. it starts with the observation that overflow values are not generally taken randomly from the full range of possible values, but instead are most likely found near the allowed range of non-overflowing values. it is therefore beneficial to write the overflowed value not as a full 32-bit number (assuming the raw data are 32-bit numbers), but instead as an N-bit, or (N+1)-bit number. thus, overflows are encoded as in Code A, followed by a (prefix, value), where the prefix is an exponential Golomb code for the value's size (actually, the size with N subtracted, since the size of an overflowing value is guaranteed to be no less than N). as with Code A, it improves compression by some fraction of one bit per value, but the savings were judged not to be worth the performance penalty. rip bozo. 🪦

🌈 Huffman coding

Huffman coding of the raw data clearly accomplished the best compression ratios of all methods. however, it was also the slowest and so was removed from the program. not everything that eats is worth the plate, bestie.

the approach is to take the data and split it into "upper" and "lower" bits. the upper bits are Huffman-coded, while the lower bits are assumed to be uniformly-distributed random values and are repeated verbatim into the compressed data stream. the split into upper and lower is made such that the data sample contains no more than 127 distinct values for the upper bits. the Huffman tree is built on the sampled frequencies of the values of the upper bits (with an extra symbol to signal overflows). the choice of 127 symbols was motivated by the possibility of storing the code tree with N symbols in only (N+1) bytes, and by the observation that even as few as 32 symbols work just as well on truly Gaussian-distributed data. iykyk. 💅

the price of decoding Huffman-encoded symbols is steep: we found that decoding took twice as long when the Huffman encoders were used. the problem is that the prefix-free Huffman codes must be read one bit at a time, for there is no way to know their length in advance. it's giving "no shortcuts, only vibes and suffering". 😭

🦋 Bit rotation 🦋

bit-rotation is an advanced feature that helps in compressing channels where the lowest B bits are constant. this sort of channel violates the usual assumption that data follow a smooth distribution function, and that the lowest few bits are likely to be distributed almost uniformly. the author of slim has certainly found channels in real experiments where (for example) the six least significant bits are always zero. this might happen in a case where an IEEE 754 floating-point number with 24 bits in its sign-plus-mantissa was converted to and recorded as an integer. iconic, honestly. ✨

if such a channel were encoded by the reduced-binary encoder, then the unchanging B lowest bits would be faithfully but wastefully reproduced over and over again. bit rotation offers a way around this redundancy.

bit rotation has slim encode such a channel by bit-shifting the raw value by B bits to the right and then copying what were the B lowest bits to the top (most significant bits) of the shifted result. this makes, for example, the value 0xbeef9900 into 0x00beef99 for the case of B=8. the reduced binary encoder can then act on the bit-rotated result, in which the lowest bits of the new value are presumably less repetitive than they were before bit rotation was applied. that 0xbeef9900 is such a cute hex, ngl. 💖

the user selects only whether to try bit rotation with the --permit-bitrotation option. the program then decides whether to choose a non-zero value of B based on the sampled data set. the choice will be the number of lowest-order bits that are strictly constant throughout the sample set.

the penalty for trying bit rotation on every channel is modest when compressing — causing roughly a 1.5% increase in instructions executed, in one test. there is of course no penalty in expanding unless bit rotation was actually used. so it's basically free real estate when you're decompressing. 🎀

🧬 32- VERSUS 64-BIT ARCHITECTURES AND ENDIANNESS 🧬

in order to operate on machines with 32-bit and 64-bit processors, slim uses strictly sized data types where it matters, such as int32_t for signed 32-bit integers. there are no known problems in running it on machines with 32-bit or 64-bit word sizes. the low-level bit operations are performed on words with the same size as the C++ data type long, which would normally be the native word size of the build machine. the code works either way, but it is likely most efficient to operate on words of the native size. the author admits to not having verified this hunch empirically. 🙈 (you can override this choice and force 32 or 64 bit word sizes. you do this when running build_bit_constants during the build process, but you probably don't want to unless cross-compiling.)

because of the author's prejudice or narrow experience (or the demands of his day job), the assumption that words are in little-endian order is deeply embedded in slim. an ambitious future contributor is welcome to remove this restriction, but it is hard to see how to handle byte-swapping without a performance penalty. for now, slim fails an assertion if run on a big-endian machine. so if you're on big-endian, that's a no from slim, bestie. 💀

🌈 ENVIRONMENT 🌈

it would be very reasonable to have slim use environment variables for some things. feel free to suggest such features; there are none at this time. that's it, that's the whole section. 🫶

✨ EXAMPLES ✨

okok bestie, let's get practical. 💅

🎀 Example 1: the Multi-channel electronics system

First up, we're compressing data from a Multi-channel electronics system. The raw files are made of frames that are 4400 bytes long and have 1100 4-byte words each. Now — a couple of those channels are technically unsigned and we're just gonna ignore that and treat every single one as signed words, no cap. Each channel shows up only once per frame. What we want is to encode the differences between successive data values, and use method 2 (the reduced binary encoder).

slim -c1100 -i -r1 -dm2 raw_mce_file.dat

🦋 Example 2: one channel, so many repetitions

Next, a file that's just a single channel repeated over and over and over and over:

slim -c1 -i -dm2 one_channel_only.dat

🌸 Example 3: uncompress both of them

To uncompress both of those files:

unslim raw_mce_file.dat.slm one_channel_only.dat.slm

👀 SEE ALSO 👑

💀 BUGS AND UNIMPLEMENTED FEATURES 💀

Not everything here is finished, bestie. Read this part, fr fr.
  • The -f and -g command-line options for floating-point data are reserved — but they aren't implemented. Like, they exist on paper. That's it. 🙈
  • Bit rotation does not work when the data are 8 or 16-bit integers (whether signed or unsigned). It's giving "known limitation," and it's staying that way for now.

Copyright © 2008 Joseph Fowler, Princeton University.

This work was supported by and done for the benefit of the Atacama Cosmology Telescope collaboration.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Free Software Foundation.

and that's the whole page, bestie!! 🫶 you made it all the way to the end, which is honestly main character energy. This was Version 2.6, dated October 24, 2008, and it's the manual for slim(1). 💖 go forth and compress responsibly, no cap. ✨🎀🦋

📦 where this page comes from