gztool(1)

📦
version 1.8.2 · 5 versions
, in ubuntu stonking

💖 gztool(1) 💖 — from the General Commands Manual

okay bestie, sit down, hydrate, because this one is a whole journey and I am about to make it make sense. 🧚✨

🎀 NAME 🎀

gztool — extract random-positioned data from gzip files, even like tail -f 💅 (yes fr, even that)

✨ SYNOPSIS ✨

gztool
 [ [-[abLnsv] #] [-[1..9]AcCdDeEfFhilOpPqrRStTwWxXzZ|u[cCdD]] [-I <INDEX>] ] "files" ...

okay so quick lore drop before you panic at that line: the actions -bcStT go straight into index file creation (if there isn't one already) INTERLEAVED with the data flow. like, they happen at the same time, so there's literally no wasted time, no cap. 🫶

and the best part? you can interrupt any action at literally any moment and the leftover index file will just get reused (and finished up if it needs to be) the next time you run gztool over that same data. iconic behavior honestly. 💖

🌸 DESCRIPTION 🌸

gztool is a GZIP files indexer, compressor and data retriever. it can make smol little indexes for gzipped files and then use them to do quick, random-positioned data extraction. it's giving main character energy. 👑

gztool can pull random-positioned data out of gzip files with no penalty, and that includes tailing a gzip like tail -f does. 🔥

it creates an index file (.gzi) for every gzip it touches, and this whole thing is interleaved with compression/uncompression so there's no waste of time. any action can be interrupted whenever, and the remaining index will be reused on the next runs. no drama, no lost work. 💅

extraction works from any byte position (or line position) in the uncompressed data — use -b for bytes, or -L for lines. 🎯

if the uncompressed file is a text file, then using the -x modifier makes the index take care of lines, so later on gztool can be asked to extract data from a particular line with -L. 📝

the full listing of capabilities lives in INTERNALS. go look, iykyk.

💜 OPTIONS 💜

  • files — one or more files. if you don't name a file, standard input gets used instead. simple as that. 🍓
  • -[1..9] — the compression factor to use with -[c|u[cC]], running from best speed (-1) all the way to best compression (-9). default is -6. ✨
  • -a # — Await # seconds between reads when you're doing -[ST]|Ec. default is 4 s. patience is a virtue bestie 🧘
  • -A — a modifier for -[rR] that says the range of bytes/lines is in absolute values, instead of the default incremental ones.
  • -b # — extract data from the indicated uncompressed byte position of the gzip file (creating or reusing an index file) out to STDOUT. it accepts '0', '0x', and the suffixes 'kmgtpe' (^10) or 'KMGTPE' (^2). 📤
  • -C — always create a 'Complete' index file, ignoring whatever possible errors show up. bold of you, love that 💅
  • -c — compress a file like gzip would, and create an index at the same time. two birds, one slay. 🦋
  • -d — decompress a file like gzip would.
  • -D — do not delete the original file when you use -[cd].
  • -e — if multiple files are indicated, keep going after an error (if there is one). we do not stop for haters 🚫
  • -E — end processing on the first GZIP end-of-file marker at EOF. that said, with -c, -E waits for more data even at EOF. 👀
  • -f — force file overwriting if the destination file already exists.
  • -F — force index creation/completion first, and then do the action: if you don't use -F, the index gets created interleaved with the actions instead.
  • -h — print brief help; -hh prints this help. ✨ (hi, that's this)
  • -i — create an index for the indicated gzip file. for 'file.gz' the default index file name will be 'file.gzi'. this is the default action, so if you do nothing else, you're doing this.
  • -I string — the index file name will be whatever string you indicated. 🏷️
  • -l — check and list the info contained in the indicated index file. -ll and -lll crank up the level of index checking detail. 🔍
  • -L # — extract data from the indicated uncompressed line position of the gzip file (creating or reusing an index file) out to STDOUT. accepts '0', '0x', and the suffixes 'kmgtpe' (^10) or 'KMGTPE' (^2).
  • -n # — says that the first byte on the compressed input is #, not 1, which means truncated compressed inputs can be used if an index exists. ✂️
  • -O — do not check for file Overwriting. necessary only in Windows.
  • -p — says the gzip input stream may be made up of various incorrectly terminated GZIP streams, so a careful Patching of the input might be needed to extract correct data. 🩹
  • -P — like -p, but when used with -[ST] it implies that checking for errors in the stream is done as quick as possible as the gzip file grows. > Warning: this may lead to some errors not being patched. ⚠️
  • -q — extract data as indicated with -[bL] from a still-growing gzip file and keep on Supervising & extracting to STDOUT even after eof. the "i will wait for you" option 🥺
  • -r # — (range): the number of bytes to extract when using -[bL]. accepts '0', '0x', and the suffixes 'kmgtpe' (^10) 'KMGTPE' (^2).
  • -R # — (Range): the number of lines to extract when using -[bL]. accepts '0', '0x', and the suffixes 'kmgtpe' (^10) 'KMGTPE' (^2).
  • -s # — the span, in uncompressed MiB, between index points when creating the index. by default it's 10. 📏
  • -S — Supervise the indicated file. creates a growing index for a still-growing gzip file. (-i is implicit here).
  • -t — tail (extract the last bytes) to STDOUT on the indicated gzip file. 🐈
  • -T — tail (extract the last bytes) to STDOUT on the indicated still-growing gzip file, and continue Supervising & extracting to STDOUT. 👑
  • -u [cCdD] — a utility to compress (-u c) or decompress (-u d) zlib-format files to STDOUT. use -u C and -u D to manage raw compressed files instead. no index involved at all. 🚫
  • -v # — output verbosity, from 0 (none) to 5 (nuts). default is 1 (normal). pick your chaos level 💀
  • -w — wait for creation if the file doesn't exist, when you're using -[cdST].
  • -W — do not Write the index to disk. but if one is already available, read it and use it. super useful if the index is still under a -S run. 🫶
  • -x — create the index with line number information (win/*nix compatible). please note that gztool's index counts the last line even if the last char isn't a newline char — whilst the wc command will not count it in this case! 📌 this is implicit unless -X or -z are indicated.
  • -X — like -x, but the newline character is '\r' (old mac). 🍎
  • -z — create the index without line number information.
  • -Z — adjust index points to a byte boundary: no previous byte needed. snapped ✨

🌈 QUICK EXAMPLE 🌈

extract data from the 1 GiB byte (byte 2^30) onward, out of myfile.gz, into the file myfile.txt. also gztool will create (or reuse, or complete) an index file named myfile.gzi. three things, one command, we love efficiency 💅

$ gztool -b 1G myfile.gz > myfile.txt

⭐ MORE EXAMPLES ⭐

  • make an index for test.gz. the index will be named test.gzi:
$ gztool -i test.gz
  • make an index for test.gz with the name test.index, using -I:
$ gztool -I test.index test.gz
  • also, -I can be used to indicate the complete path to an index in another directory. this way the directory where the gzip file lives could be read-only and the index could be created in another read-write path:
$ gztool -I /tmp/test.gzi test.gz
  • retrieve data from the uncompressed byte position 1000000 inside test.gz. the index file will be created at the same time (named test.gzi):
$ gztool -b 1m test.gz
  • Supervise a still-growing gzip file and generate the index for it on-the-fly. the index file name will be openldap.log.gzi in this case. gztool will run until it's interrupted (it can also stop at the first end-of-gzip data with -E):
$ gztool -S openldap.log.gz
  • the previous command can be sent to the background with no verbosity, so we can just forget about it entirely:
$ gztool -v0 -S openldap.log.gz &
  • creating and index for all "*gz" files in a directory:
$ gztool -i *gz
  • extract data from project.gz at byte 1 GiB to STDOUT, and run grep on that output. the index file name will be project.gzi:
$ gztool -b 1G project.gz | grep -i "balance = "
  • please note that STDOUT is used for data extraction with the -bcdtT modifiers, so you need an explicit command line redirection if the output is going to be stored in a file:
$ gztool -b 99m project.gz > uncompressed.data
  • extract data from a gzipped file whose index is still growing, with a gztool -S process monitoring that (still-growing) gzip file: in this case using -W will not try to update the index on disk, so the other process doesn't get disturbed! (note that gztool always tries to update the index it used if it thinks it's necessary):
$ gztool -Wb 100k still-growing-gzip-file.gz > mytext
  • extract data from line 10 million, to STDOUT:
$ gztool -L 10m compressed_text_file.gz
  • nonetheless, note that if in the previous example an index was previously created for the gzip file without the -x parameter (or not using -L), then since it doesn't contain line numbering info, gztool will complain and stop. 😤 you can get around this by telling gztool to use another new index file name (-I), or even by not using one at all with -W (do not write index) plus an index file name that doesn't exist (in this case None — it won't be created because of -W), and so ((just) this time) the gzip will be processed from the beginning:
$ gztool -L 10m -WI None compressed_text_file.gz
  • extract data from line 10 million, to STDOUT, on a still-growing file (-q), so if that line number still hasn't been reached, gztool will patiently wait until the gzip file grows enough to store that line, and then show just it (-R 1). -q for waiting on content works with both -L (line numbers) and -b (byte positions):
$ gztool -q -L 10m -R 1 compressed_text_file.gz
  • extract all data from a rsyslog veryRobustZip (//www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html#veryRobustZip) that contains dirty data. these corrupted-gzip-files can arise when using rsyslog's veryRobustZip omfile option and the process that is logging gets abruptly terminated and then restarted — this produces an incorrectly-terminated-gzip stream that's followed by another gzip stream in the same file. gzip (nor zlib) cannot read these files beyond the point of error. but gztool can correctly extract all data (and only good data) using the -p (patch) parameter:
$ gztool -p -b0 compressed_text_file.gz

this creates, as usual, the index file compressed_text_file.gzi. in order to not create it, -W (do not Write index) can be used:

$ gztool -pWb0 compressed_text_file.gz

note that -p can require up to twice the time for decompression, because it performs two decompression processes: the usual one, and another one that's performed in advance of the usual one, which is the one that detects errors, marks them, and finds new entry points to end/begin the decompression while circumventing the problems. 🕵️

note also that these corrupted-gzip-files should always be decompressed with the -p parameter, even if a gztool index file exists for them, because the index file stores entry points, but does not store where errors occur in the gzip file. that said, if the -[bL] point of extraction is beyond the point(s) of error in the gzip file and an index file exists, then the decompression can proceed fine without -p, since the index points stored in the index file are always clean. ✨

  • owo a note about tailing gzip files that could be messy 🫶 — so when you're tailing a still-growing gzip file with -T and it might contain errors at some point, you may still wanna get output out of that gzip stream ASAP. That's literally what the patching option -P is for (yes it's like -p but capitalized, bestie). Here's the tea: with -p, gztool decompresses the stream about 48 kiB ahead of the output that's actually shown/written, specifically so it can catch possible gzip-stream errors ahead of your output and therefore keep the output always clean, with no error-introduced artifacts sneaking in. The ✨side effect✨ (there's always one) is that output must always wait for that 48 kiB of data to be available in advance, which — if the file grows slowly — can take a very long time. With -P the buffer-ahead restriction is relaxed to just as few bytes as are available before hitting end-of-file and waiting for new data, so responsiveness is as quick as without -p. The side effect of -P (iykyk): depending on the gzip file, some errors may lead to incorrect output being shown/written — though in this case a "PATCHING WARNING" would be shown (to stderr). Not ideal but at least it tells you. 💀
$ gztool -PT application_log.gz

Same energy applies to -S, except in this case there's no output at all, since only the index is being constructed:

$ gztool -PS application_log.gz
  • tail to stdout like tail -f 🦋 — to tail an still-growing gzip file to stdout (and yes, an index file will be created with the name still-growing-gzip-file.gzi in this case):
$ gztool -T still-growing-gzip-file.gz
  • more on files still being "Supervised" (-S) by another gztool instance 👑 — they can also be tailed à la tail -f without updating the index on disk, using -W:
$ gztool -WT still-growing-gzip-file.gz
  • compress (-c) an still growing (-E) file 🔥 — in this case both still-growing-file.gz and still-growing-file.gzi files will be created on-the-fly as the source file grows. Heads up though: in order to terminate compression, Ctrl+C must be used to kill gztool. That results in an incomplete-gzip-file as per GZIP standard, but don't panic — this is not important, because it will contain all the source data, and both gzip and gztool (or any other tool) can correctly and completely decompress it:
$ gztool -Ec still-growing-file
  • fixing an incomplete index file 🎀 — say you have an incomplete index file (it just doesn't have the length of the source data, because it didn't correctly finish) and you want to make it complete so the length of the uncompressed data is stored. Just unconditionally complete it with -C with a new -i run over your gzip file. Note that since the existent index data is used (in this case the file my-incomplete-gzip-data.gzi), only the last compressed bytes are decompressed to complete this action:
$ gztool -Ci my-incomplete-gzip-data.gz
  • decompress like gzip (-d) but don't delete the original (-D) 🌸 — decompressed file will be myfile. Note that the gzipped file must have a .gz extension or gztool will complain. It will complain. 😭
$ gztool -Dd myfile.gz
  • decompress a file that doesn't have the .gz extension, like with gzip (-d):
$ cat mycompressedfile | gztool -d > my_uncompressed_file
  • show internals of all index files in this directory 🔍 — -e is used not to stop the process on the first error, in case a *.gzi file is not a valid gzip index file. The -ll list option repetition will show data about each index point. And -lll also decompresses each point's window to ensure index integrity:
$ gztool -ell *.gzi

If gztool finds the gzip file companion of the index file, some statistics are shown, like the index/gzip size ratio, or the ratio of compression of the gzip file. Also, if the gzip is complete, the size of the uncompressed data is shown. This number is interesting if the gzip file is bigger than 4 GiB — in which case gunzip -l cannot correctly calculate it, as it is limited to a 32 bit counter (see //tools.ietf.org/html/rfc1952#page-5) — or if the gzip file is in bgzip format, in which case gunzip -l would only show data about the first block (< 64 kiB).

Note that gztool -l tries to guess the companion gzip file of the index, looking for a file with the same name but without the i of the .gzi file name extension, or without the .gzi. But the gzip file name can also be directly indicated with this format:

$ gztool -l -I index_filename gzip_filename

In this latter case only a pair of index+gzip filenames can be indicated with each use.

  • truncated gzip file, big yikes, but we ball 💅 — use a truncated gzip file (the 100000 first bytes are removed: not zeroed, removed; if they're zeroed the cautions are the same, but -n is not needed), to extract from byte 20 MiB, using a previously generated index. As far as the -b parameter refers to a byte after an index point (see -ll) and -n is less than that needed first index point, this is always possible. In this case -I gzip_filename.gzi is implicit:
$ gztool -n 100001 -b 20M gzip_filename.gz

Take into account that, as shown, the first byte of the truncated gzip_filename.gz file is numbered 100001 — that is, the bytes retain the order number in which they appear in the original file (that's literally the reason why it's not the 1 byte).

Please note that index point positions in the index file may require also the previous byte to be available in the truncated gzip file, because a gzip stream is not byte-rounded but a stream of pure bits. So if you're thinking of truncating a gzip file, please do it always at least by one byte before the indicated index point in the gzip. As said, it may not be needed, but in 7 of 8 cases it is needed. Another option is to use -Z when creating the index, as indicated below.

  • clean cut points with -Z ✨ — create an index for a gzip file in which every index entry point is adjusted to a byte boundary, so no previous byte (bits) is needed. Note that in general the byte at which the index entry point begins does not represent a clear cut point, because the gzip window needs up to 7 bits from the previous byte. This is so because gzip is a bit-level stream compressor. With -Z the cut point is always clean and no bits from the previous byte are required. This will result in index points spaced by more than -s bytes between then, and so, may be, less points in the index. But this is completely safe and sound. 🫶
$ gztool -Z my_gzip_file.gz

-Z exists since gztool v1.6.0.

  • the source file got overwritten? keep going 🌈 — since v1.5.0, using -[fW] (-f: force index overwriting; -W: do not write index) together with -[ST] (-S: create index on still-growing gzip file; -T: tail and continue decompressing to stdout) indicates gztool to continue operations even after the source file is overwritten. If using -f, the index file will be overwritten. For example:
$ gztool -WT log_filename.gz
...
File overwriting detected and restarting decompression...
Processing 'log_filename.gz'...
  • Windows girlies, this one's for you 🪟 — in version v1.8.1 the modifier -O was introduced to be used in Windows, because on this platform the POSIX function used to obtain file sizes (stat()) can provide erroneous results when the file is being updated. This can occur for example with growing gzip files processed using -[ST]. Also with -cED if the source uncompressed file is for example a log file that is growing (-E waits for more data even if the end-of-file is reached; -D is used to not delete the uncompressed file when finished! — because gztool -c mimics gzip behaviour).

🧠 INTERNALS 🧠

By default gzip-compressed files cannot be accessed in random mode: any byte required at position N requires the complete gzip file to be decompressed from the beginning to the N byte. Nonetheless Mark Adler, the author of zlib (//github.com/madler/zlib), provided years ago a cryptic file named zran.c (//github.com/madler/zlib/blob/master/examples/zran.c) that creates an "index" of "windows" filled with 32 kiB of uncompressed data at different positions along the un/compressed file, which can be used to initialize the zlib library and make it behave as if compressed data begin there.

gztool builds upon zran.c to provide a useful command line tool. Also, some optimizations has been made:

  • gztool can correctly read incomplete gzip-concatenated-files (using -p), that is, a gzip composed of a concatenation of gzip files, some of which are not correctly terminated. This can happen, for example, when using rsyslog's veryRobustZip omfile option (//www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html#veryrobustzip) and the process that is logging is abruptly terminated and then restarted.
  • gztool can store line numbering information in the index (use only if source data is text!), and retrieve data from a specific line number using -L. (Using -[xXz] when creating the index selects Unix new line format (default), old Mac new line format, or no line information respectively.)
  • gztool can Supervise an still-growing gzip file (for example, a log created by rsyslog directly in gzip format) and generate the index on-the-fly, thus reducing in the practice to zero the time of index creation. See -S.
  • extraction of data and index creation are interleaved, so there's no waste of time for the index creation.
  • index files are reusable, so they can be stopped at any time and reused and/or completed later.
  • an ex novo index file format has been created to store the index
  • span between index points is raised by default from 1 MiB to 10 MiB, and can be adjusted with -s (span).
  • windows are compressed in file
  • windows are not loaded in memory unless they're needed, so the application memory footprint is fairly low (< 1 MiB)
  • gztool can compress files (-c) and at the same time generate an index that is about 10-100 times smaller than if the index is generated after the file has already been compressed with gzip.
  • Compatible with bgzip files (//www.htslib.org/doc/bgzip.html)
  • Compatible with complete gzip concatenated files
  • Compatible with rsyslog's veryRobustZip omfile option (variable-short-uncompressed complete-gzip-block sizes)
  • data can be provided from/to stdin/stdout
  • gztool can be used to remotely retrieve just a small part of a bigger gzip compressed file and successfully decompress it locally. See //unix.stackexchange.com/questions/429197/#541903. Just note that the gztool index file must be also available.

🏠 PROJECT HOME PAGE 🏠

//github.com/circulosmeos/gztool

👀 SEE ALSO 👀

gzip(1), gunzip(1), zlib(3)

🫶 AUTHOR 🫶

This program was written by Roberto S. Galende <roberto.s.galende@gmail.com> on work by Mark Adler's zlib (examples/zran.c) and is copyrighted under zlib licence terms.

and that's the whole page, bestie!! 🌈💖 This was gztool v1.8.2, dated Jun 12 2026, living in section gztool(1) — go forth and index your gzips, you absolute icon. 👑✨

📦 where this page comes from