DICTZIP(1)

✨ NAME ✨

dictzip, dictunzip — compress (or expand) files, allowing random access 💖

okay so real quick: this is the manual for dictzip and its lil sibling dictunzip, and they're lowkey iconic. let's get into it bestie 🫶

---

🌸 SYNOPSIS 🌸

dictzip [options] name
dictunzip [options] name

---

💅 DESCRIPTION 💅

dictzip squishes files down using the gzip(1) algorithm (that's LZ77, iykyk), and the way it does it is completely compatible with the gzip file format. no cap. 🎀

but here's the ✨ twist ✨ — there's an extension to the gzip file format (the Extra Field, described in section 2.3.1.1 of RFC 1952) that lets you stash extra data in the header of a compressed file. programs like gzip and zcat will just straight up ignore that extra data, like it's not even there 🙈. but dictd(8), the DICT protocol dictionary server, will use it — to do pseudo-random access on the file. main character behavior fr.

files in the dictzip format should end in ".dz" so you can tell them apart from regular gzip files that don't have that special header info. don't skip the .dz bestie, it's the whole vibe check 💀

---

🦋 the extra field, per RFC 1952 🦋

straight from RFC 1952, the extra field is specified like this:

If the FLG.FEXTRA bit is set, an "extra field" is present in the header, with total length XLEN bytes. It consists of a series of subfields, each of the form:

+---+---+---+---+==================================+
|SI1|SI2|  LEN  |... LEN bytes of subfield data ...|
+---+---+---+---+==================================+

SI1 and SI2 provide a subfield ID, typically two ASCII letters with some mnemonic value. Jean-Loup Gailly <gzip@prep.ai.mit.edu> is maintaining a registry of subfield IDs; please send him any subfield ID you wish to use. Subfield IDs with SI2 = 0 are reserved for future use. LEN gives the length of the subfield data, excluding the 4 initial bytes.

so dictzip uses 'R' for SI1 and 'A' for SI2 (as in, "Random Access" 👑). and after the LEN field, the data is laid out like this:

+---+---+---+---+---+---+===============================+
|  VER  | CHLEN | CHCNT |  ... CHCNT words of data ...  |
+---+---+---+---+---+---+===============================+

per RFC 1952, all the data is stored least-significant byte first. for VER 1 of the data, every value is 16-bits long (that's 2 bytes), and they're all unsigned integers.

now XLEN (which gets specified earlier in the header) is a two byte integer, so the extra field can be 0xffff bytes long. 2 bytes of that go to the subfield ID (SI1 and SI1), and 2 bytes go to the subfield length (LEN). that leaves you 0xfffb bytes — which is 0x7ffd 2-byte entries or 0x3ffe 4-byte entries. and since the zip output buffer has to be 10% + 12 bytes larger than the input buffer, we can store 58969 bytes per entry, or about 1.8GB if the 2-byte entries are used. if that ever becomes a limiting factor, another format version can be selected and defined for 4-byte entries. slay, future-proofing 💖

---

🎀 how the chunking works 🎀

for compression, the file gets divided up into "chunks" of data. each chunk is less than 64kB, and can be compressed into an area that's also less than 64kB long (taking incompressible data into account — usually the data compresses into a block that's way smaller than the original, which is the whole point 🍓).

• the CHLEN field specifies the length of a "chunk" of data.
• the CHCNT field specifies how many chunks are preset.
• the CHCNT words of data specify how long each chunk is after compression (i.e., in the current compressed file).

to do random access on the data, the offset and length of the data get handed to library routines. those routines figure out which chunk the desired data begins in, and decompress that chunk. consecutive chunks get decompressed as necessary. it's giving ✨efficiency✨

---

🌈 TRADEOFFS 🌈

• Speed — true random file access is not realized, because any access, even for a single byte, requires that a 64kB chunk be read and decompressed. this is slower than accessing a flat text file, but it is much, much faster than performing serial access on a fully compressed file. so like... not perfect, but ate 💅
• Space — for the textual dictionary databases we're working with, using 64kB chunks and maximal LZ77 compression gives you a file that's only about 4% larger than the same file compressed all at once. that's it?? that's the whole cost?? iconic.

---

👑 OPTIONS 👑

• -d, --decompress — Decompress. this is the default if the executable is called dictunzip.
• -c, --stdout — write output on standard output; keep the original files unchanged. this one is only available when decompressing (because parts of the header must be updated after a write when compressing — soz).
• -f, --force — force compression or decompression even if the output file already exists.
• -h, --help — display help.
• -k, --keep — do not delete the original file.
• -l, --list — for each compressed file, list the following fields:
  • type: dzip, gzip, or text (this includes files in unknown formats)
  • crc: CRC checksum
  • date and time: from header
  • chunks: number of chunks in file
  • size: size of each uncompressed chunk
  • compr.: compressed size
  • uncompr.: uncompressed size
  • ratio: compression ratio (0.0% if unknown)
  • name: name of uncompressed file

unlike gzip, the compression method is not detected. just so you know bestie 💀

• -L, --license — display the dictzip license and quit.
• -t, --test — check the compressed file integrity. this option is not implemented. instead, it will list the header information. (yes really. it's giving bait and switch but we love her anyway 🙈)
• -v, --verbose — verbose. display extra information during compression.
• -V, --version — version. display the version number and compilation options then quit.
• -s start, --start start — specify the offer to start decompression, using decimal numbers. the default is at the beginning of the file.
• -e size, --size size — specify the size of the portion of the file to decompress, using decimal numbers. the default is the whole file.
• -S start, --Start start — specify the offer to start decompression, using base64 numbers. the default is at the beginning of the file.
• -E size, --Size start — specify the size of the portion of the file to decompress, using base64 numbers. the default is the whole file.
• -p prefilter, --pre prefilter — specify a shell command to execute as a filter before compression or decompression of a chunk. the pre- and post-compression filters can be used to provide additional compression or output formatting. the filters may not increase the buffer size significantly. the pre- and post-compression filters were designed to provide the most general interface possible. very ✨general✨ of them.
• -P postfilter, --post postfilter — specify a shell command to execute as a filter after compression or decompression.

---

💜 CREDITS 💜

dictzip was written by Rik Faith (faith@cs.unc.edu) and is distributed under the terms of the GNU General Public License. if you need to distribute under other terms, write to the author. 📬

the main libraries used by this program (zlib, regex, libmaa) are distributed under different terms, so you may be able to use those libraries for applications which are incompatible with the GPL — please see the copyright notices and license information that come with the libraries for more information, and consult with your attorney to resolve these issues. (yes, actual attorney. not a vibe check, a lawyer 💀)

---

🌟 SEE ALSO 🌟

dict(1), dictd(8), gzip(1), gunzip(1), zcat(1)

---

and that's the whole page bestie!! this one's from 22 Jun 1997, part of the DICTZIP(1) manual — yes she's vintage, and yes she still serves 💖✨🎀🦋 go compress something, ily, bye 🫶👑🌈