IO::COMPRESS::FAQ(3perl)

📦

version 5.44.0 · 14 versions

💖 NAME 💖

IO::Compress::FAQ — no wait bestie, it's the Frequently Asked Questions about IO::Compress page. That's the whole vibe. Sit down, grab a snack 🍓, we're going through every single question, and I mean every one. Periodt.

✨ DESCRIPTION ✨

Common questions answered. That's it, that's the description. The people who wrote this were like "we'll answer the stuff everyone asks" and honestly? iconic. 👑

🌸 GENERAL 🌸

🦋 Compatibility with Unix compress/uncompress 🦋

Okay so listen. Even though Compress::Zlib has a pair of functions called compress and uncompress, they are not related to the Unix programs of the same name. Like, same names, totally different people, no relation, block them. The Compress::Zlib module is not compatible with Unix compress.

If you have the uncompress program available, you can use this to read compressed files 💖

open F, "uncompress -c $filename |";
while (<F>)
{
    ...

Alternatively, if you have the gunzip program available, you can use this to read compressed files 🌸

open F, "gunzip -c $filename |";
while (<F>)
{
    ...

and this to write compress files, if you have the compress program available ✨

open F, "| compress -c $filename ";
print F "data";
...
close F ;

🎀 Accessing `.tar.Z` files 🎀

So the Archive::Tar module can optionally use Compress::Zlib (via the IO::Zlib module) to access tar files that have been compressed with gzip. Cute! 🫶

But not-so-cute news: tar files compressed with the Unix compress utility cannot be read by Compress::Zlib and so cannot be directly accessed by Archive::Tar. RIP. 💀

If the uncompress or gunzip programs are available, you can use one of these workarounds to read .tar.Z files from Archive::Tar 💅

Firstly with uncompress:

use strict;
use warnings;
use Archive::Tar;

open F, "uncompress -c $filename |";
my $tar = Archive::Tar->new(*F);
...

and this with gunzip:

use strict;
use warnings;
use Archive::Tar;

open F, "gunzip -c $filename |";
my $tar = Archive::Tar->new(*F);
...

Similarly, if the compress program is available, you can use this to write a .tar.Z file 🔥

use strict;
use warnings;
use Archive::Tar;
use IO::File;

my $fh = IO::File->new( "| compress -c >$filename" );
my $tar = Archive::Tar->new();
...
$tar->write($fh);
$fh->close ;

🌈 How do I recompress using a different compression? 🌈

This is easier than you might expect — that's literally what the page says, and it's true — if you realise that all the IO::Compress::* objects are derived from IO::File and that all the IO::Uncompress::* modules can read from an IO::File filehandle. Family tree moment. 👑

So, for example, say you have a file compressed with gzip that you want to recompress with bzip2. Here is all that is needed to carry out the recompression. Wild, right?

use IO::Uncompress::Gunzip ':all';
use IO::Compress::Bzip2 ':all';

my $gzipFile = "somefile.gz";
my $bzipFile = "somefile.bz2";

my $gunzip = IO::Uncompress::Gunzip->new( $gzipFile )
    or die "Cannot gunzip $gzipFile: $GunzipError\n" ;

bzip2 $gunzip => $bzipFile
    or die "Cannot bzip2 to $bzipFile: $Bzip2Error\n" ;

Not me being the bearer of bad news but 🫠 — there is a limitation to this technique. Some compression file formats store extra information along with the compressed data payload. For example, gzip can optionally store the original filename and Zip stores a lot of information about the original file. If the original compressed file contains any of this extra information, it will not be transferred to the new compressed file using the technique above.

🗜️ ZIP 🗜️

💜 What Compression Types do `IO::Compress::Zip` & `IO::Uncompress::Unzip` support? 💜

The following compression formats are supported by IO::Compress::Zip and IO::Uncompress::Unzip:

Store (method 0) — No compression at all. Just vibes. Raw. Unfiltered. 💅
Deflate (method 8) — This is the default compression used when creating a zip file with IO::Compress::Zip.
Bzip2 (method 12) — Only supported if the IO-Compress-Bzip2 module is installed. (Install it bestie, you know you want to.)
Lzma (method 14) — Only supported if the IO-Compress-Lzma module is installed.

🔥 Can I Read/Write Zip files larger the 4 Gig? 🔥

Yes. Both the IO-Compress-Zip and IO-Uncompress-Unzip modules support the zip feature called Zip64. That allows them to read/write files/buffers larger than 4Gig. Big file energy. 👑

If you are creating a Zip file using the one-shot interface, and any of the input files is greater than 4Gig, a zip64 complaint zip file will be created.

zip "really-large-file" => "my.zip";

Similarly with the one-shot interface, if the input is a buffer larger than 4 Gig, a zip64 complaint zip file will be created.

zip \$really_large_buffer => "my.zip";

The one-shot interface allows you to force the creation of a zip64 zip file by including the Zip64 option.

zip $filehandle => "my.zip", Zip64 => 1;

If you want to create a zip64 zip file with the OO interface you must specify the Zip64 option.

my $zip = IO::Compress::Zip->new( "whatever", Zip64 => 1 );

When uncompressing with IO-Uncompress-Unzip, it will automatically detect if the zip file is zip64. No manual work, slay. ✨

If you intend to manipulate the Zip64 zip files created with IO-Compress-Zip using an external zip/unzip, make sure that it supports Zip64. 🔍

In particular, if you are using Info-Zip you need to have zip version 3.x or better to update a Zip64 archive and unzip version 6.x to read a zip64 archive. Write that down. 📝

🧚 Can I write more that 64K entries is a Zip files? 🧚

Yes. Zip64 allows this. See previous question. ✨ (They really said "scroll up bestie". 💀)

📚 Zip Resources 📚

The primary reference for zip files is the appnote document available at <http://www.pkware.com/documents/casestudies/APPNOTE.TXT>

If you need to investigate the extra fields that are used in zip files the appnote document covers a lot of them. The Info-ZIP zip distribution, available at <https://infozip.sourceforge.net/>, contains additional extra field definitions. Look for the file extrafld.txt.

🐇 GZIP 🐇

💖 Gzip Resources 💖

The primary reference for gzip files is RFC 1952 <https://datatracker.ietf.org/doc/html/rfc1952>

The primary site for gzip is <http://www.gzip.org>.

🌸 Dealing with concatenated gzip files 🌸

If the gunzip program encounters a file containing multiple gzip files concatenated together it will automatically uncompress them all. The example below illustrates this behaviour:

$ echo abc | gzip -c >x.gz
$ echo def | gzip -c >>x.gz
$ gunzip -c x.gz
abc
def

By default IO::Uncompress::Gunzip will not behave like the gunzip program. It will only uncompress the first gzip data stream in the file, as shown below:

$ perl -MIO::Uncompress::Gunzip=:all -e 'gunzip "x.gz" => \*STDOUT'
abc

To force IO::Uncompress::Gunzip to uncompress all the gzip data streams, include the MultiStream option, as shown below:

$ perl -MIO::Uncompress::Gunzip=:all -e 'gunzip "x.gz" => \*STDOUT, MultiStream => 1'
abc
def

🦋 Reading bgzip files with `IO::Uncompress::Gunzip` 🦋

A bgzip file consists of a series of valid gzip-compliant data streams concatenated together. To read a file created by bgzip with IO::Uncompress::Gunzip use the MultiStream option as shown in the previous section. Same trick, different file. 💅

See the section titled "The BGZF compression format" in <http://samtools.github.io/hts-specs/SAMv1.pdf> for a definition of bgzip.

🌈 ZLIB 🌈

💜 Zlib Resources 💜

The primary site for the zlib compression library is <http://www.zlib.org>.

💐 Bzip2 💐

✨ Bzip2 Resources ✨

The primary site for bzip2 is <http://www.bzip.org>.

🎀 Dealing with Concatenated bzip2 files 🎀

If the bunzip2 program encounters a file containing multiple bzip2 files concatenated together it will automatically uncompress them all. The example below illustrates this behaviour:

$ echo abc | bzip2 -c >x.bz2
$ echo def | bzip2 -c >>x.bz2
$ bunzip2 -c x.bz2
abc
def

By default IO::Uncompress::Bunzip2 will not behave like the bunzip2 program. It will only uncompress the first bunzip2 data stream in the file, as shown below:

$ perl -MIO::Uncompress::Bunzip2=:all -e 'bunzip2 "x.bz2" => \*STDOUT'
abc

To force IO::Uncompress::Bunzip2 to uncompress all the bzip2 data streams, include the MultiStream option, as shown below:

$ perl -MIO::Uncompress::Bunzip2=:all -e 'bunzip2 "x.bz2" => \*STDOUT, MultiStream => 1'
abc
def

🧚 Interoperating with Pbzip2 🧚

Pbzip2 (<http://compression.ca/pbzip2/>) is a parallel implementation of bzip2. The output from pbzip2 consists of a series of concatenated bzip2 data streams. Parallel processing bestie, so fast. ⚡

By default IO::Uncompress::Bzip2 will only uncompress the first bzip2 data stream in a pbzip2 file. To uncompress the complete pbzip2 file you must include the MultiStream option, like this.

bunzip2 $input => \$output, MultiStream => 1
    or die "bunzip2 failed: $Bunzip2Error\n";

okay stay with me, there's more coming — this was only part 1 🫶

🌐 HTTP & NETWORK 🌐

✨ Apache::GZip Revisited ✨

okay bestie, gather round, because this one is a journey 💖

Below is a mod_perl Apache compression module called "Apache::GZip", and it got yanked straight outta here: http://perl.apache.org/docs/tutorials/tips/mod_perl_tricks/mod_perl_tricks.html#On_the_Fly_Compression

package Apache::GZip;
#File: Apache::GZip.pm

use strict vars;
use Apache::Constants ':common';
use Compress::Zlib;
use IO::File;
use constant GZIP_MAGIC => 0x1f8b;
use constant OS_MAGIC => 0x03;

sub handler {
    my $r = shift;
    my ($fh,$gz);
    my $file = $r->filename;
    return DECLINED unless $fh=IO::File->new($file);
    $r->header_out('Content-Encoding'=>'gzip');
    $r->send_http_header;
    return OK if $r->header_only;

    tie *STDOUT,'Apache::GZip',$r;
    print($_) while <$fh>;
    untie *STDOUT;
    return OK;
}

sub TIEHANDLE {
    my($class,$r) = @_;
    # initialize a deflation stream
    my $d = deflateInit(-WindowBits=>-MAX_WBITS()) || return undef;

    # gzip header -- don't ask how I found out
    $r->print(pack("nccVcc",GZIP_MAGIC,Z_DEFLATED,0,time(),0,OS_MAGIC));

    return bless { r   => $r,
                   crc =>  crc32(undef),
                   d   => $d,
                   l   =>  0
                 },$class;
}

sub PRINT {
    my $self = shift;
    foreach (@_) {
      # deflate the data
      my $data = $self->{d}->deflate($_);
      $self->{r}->print($data);
      # keep track of its length and crc
      $self->{l} += length($_);
      $self->{crc} = crc32($_,$self->{crc});
    }
}

sub DESTROY {
   my $self = shift;

   # flush the output buffers
   my $data = $self->{d}->flush;
   $self->{r}->print($data);

   # print the CRC and the total length (uncompressed)
   $self->{r}->print(pack("LL",@{$self}{qw/crc l/}));
}

1;

And here's the Apache config entry you're gonna need to actually use this thing. Set it once and everything in the /compressed directory gets compressed automagically ✨

<Location /compressed>
   SetHandler  perl-script
   PerlHandler Apache::GZip
</Location>

ngl, at first glance there is a lot happening in "Apache::GZip", but you can literally sum up what the code is doing like this — read the contents of the file in "$r->filename", compress it, and write the compressed data to standard output. That's it. That's the whole vibe. 💅

The reason this code has to jump through a few hoops to pull that off is:

The gzip support in "Compress::Zlib" version 1.x can only work with a real filesystem filehandle. The filehandles that Apache modules use are not associated with the filesystem. 😭
Which means all the gzip support has to be done by hand — in this case by creating a tied filehandle to deal with creating the gzip header and trailer. Bestie is hand-stitching the gzip format fr fr 🧵

Good news tho: "IO::Compress::Gzip" does not have that filehandle limitation (this was literally one of the reasons for writing it in the first place 👑). So if you swap "Compress::Zlib" for "IO::Compress::Gzip", the whole tied filehandle situation can just be deleted. Here's the rewritten glow-up:

package Apache::GZip;

use strict vars;
use Apache::Constants ':common';
use IO::Compress::Gzip;
use IO::File;

sub handler {
    my $r = shift;
    my ($fh,$gz);
    my $file = $r->filename;
    return DECLINED unless $fh=IO::File->new($file);
    $r->header_out('Content-Encoding'=>'gzip');
    $r->send_http_header;
    return OK if $r->header_only;

    my $gz = IO::Compress::Gzip->new( '-', Minimal => 1 )
        or return DECLINED ;

    print $gz $_ while <$fh>;

    return OK;
}

or even more succinctly, like this, using a one-shot gzip:

package Apache::GZip;

use strict vars;
use Apache::Constants ':common';
use IO::Compress::Gzip qw(gzip);

sub handler {
    my $r = shift;
    $r->header_out('Content-Encoding'=>'gzip');
    $r->send_http_header;
    return OK if $r->header_only;

    gzip $r->filename => '-', Minimal => 1
      or return DECLINED ;

    return OK;
}

1;

The use of the one-shot "gzip" above just reads from "$r->filename" and writes the compressed data to standard output. Simple. Clean. Iconic. 🦋

Important: note the use of the "Minimal" option in the code above. When you're using gzip for Content-Encoding you should always use this option. In the example above it prevents the filename from being included in the gzip header, and it makes the size of the gzip data stream a slight bit smaller. Lowkey free win, take it. 🎀

💻 Compressed files and Net::FTP 💻

The "Net::FTP" module ships two low-level methods called "stor" and "retr", and both of them return filehandles 📁. You can hand those filehandles to the "IO::Compress/Uncompress" modules to compress or uncompress files read from / written to an FTP Server on the fly, so you never have to create a temporary file. No temp files, no mess, we love to see it 😌✨

First up: code that uses "retr" to uncompress a file as it's being read off the FTP Server.

use Net::FTP;
use IO::Uncompress::Gunzip qw(:all);

my $ftp = Net::FTP->new( ... )

my $retr_fh = $ftp->retr($compressed_filename);
gunzip $retr_fh => $outFilename, AutoClose => 1
    or die "Cannot uncompress '$compressed_file': $GunzipError\n";

and this one compresses a file as it's being written to the FTP Server 🫶

use Net::FTP;
use IO::Compress::Gzip qw(:all);

my $stor_fh = $ftp->stor($filename);
gzip "filename" => $stor_fh, AutoClose => 1
    or die "Cannot compress '$filename': $GzipError\n";

🌀 MISC 🌀

🌸 Using `InputLength` to uncompress data embedded in a larger file/buffer 🌸

Okay bestie, this one is SO common and SO good. Picture it: your compressed data isn't sitting there all alone in its own lil file — it's embedded inside a bigger file or buffer, and you need to read both. The compressed bit AND the stuff around it. Iconic, honestly.

Take a zip file. It's a real, well-defined format that mixes compressed and uncompressed sections of data in one single file. It's giving collage. 🎨

For this convo, just think of a zip file as a sequence of compressed data streams, and each one is prefixed by an uncompressed local header. That local header holds the tea about the compressed stream — the name of the compressed file, and (crucially) the length of the compressed data stream. 📏

So here's a script that walks a zip file and prints out how many lines are in each compressed file, to show you how InputLength works. (If you're actually writing code to walk zip files for real, go see "Walking through a zip file" in IO::Uncompress::Unzip.) Also bestie — even though this example uses zlib-based compression, the whole technique works with the other IO::Uncompress::* modules too. 🫶

use strict;
use warnings;

use IO::File;
use IO::Uncompress::RawInflate qw(:all);

use constant ZIP_LOCAL_HDR_SIG  => 0x04034b50;
use constant ZIP_LOCAL_HDR_LENGTH => 30;

my $file = $ARGV[0] ;

my $fh = IO::File->new( "<$file" )
            or die "Cannot open '$file': $!\n";

while (1)
{
    my $sig;
    my $buffer;

    my $x ;
    ($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH
        or die "Truncated file: $!\n";

    my $signature = unpack ("V", substr($buffer, 0, 4));

    last unless $signature == ZIP_LOCAL_HDR_SIG;

    # Read Local Header
    my $gpFlag             = unpack ("v", substr($buffer, 6, 2));
    my $compressedMethod   = unpack ("v", substr($buffer, 8, 2));
    my $compressedLength   = unpack ("V", substr($buffer, 18, 4));
    my $uncompressedLength = unpack ("V", substr($buffer, 22, 4));
    my $filename_length    = unpack ("v", substr($buffer, 26, 2));
    my $extra_length       = unpack ("v", substr($buffer, 28, 2));

    my $filename ;
    $fh->read($filename, $filename_length) == $filename_length
        or die "Truncated file\n";

    $fh->read($buffer, $extra_length) == $extra_length
        or die "Truncated file\n";

    if ($compressedMethod != 8 && $compressedMethod != 0)
    {
        warn "Skipping file '$filename' - not deflated $compressedMethod\n";
        $fh->read($buffer, $compressedLength) == $compressedLength
            or die "Truncated file\n";
        next;
    }

    if ($compressedMethod == 0 && $gpFlag & 8 == 8)
    {
        die "Streamed Stored not supported for '$filename'\n";
    }

    next if $compressedLength == 0;

    # Done reading the Local Header

    my $inf = IO::Uncompress::RawInflate->new( $fh,
                        Transparent => 1,
                        InputLength => $compressedLength )
      or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

    my $line_count = 0;

    while (<$inf>)
    {
        ++ $line_count;
    }

    print "$filename: $line_count\n";
}

Real talk: the majority of that code is just fussing over the zip local header data. The part I actually want you to lock in is at the bottom. 👇

while (1) {

    # read local zip header data
    # get $filename
    # get $compressedLength

    my $inf = IO::Uncompress::RawInflate->new( $fh,
                        Transparent => 1,
                        InputLength => $compressedLength )
      or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

    my $line_count = 0;

    while (<$inf>)
    {
        ++ $line_count;
    }

    print "$filename: $line_count\n";
}

So the call to "IO::Uncompress::RawInflate" spins up a brand new filehandle $inf, and you can use it to read from the parent filehandle $fh, uncompressing as it goes. And the InputLength option? It guarantees that at most $compressedLength bytes of compressed data get read from the $fh filehandle. (Only exception is an error case, like a truncated file or a corrupt data stream — those gonna do what they gonna do. 💀)

Which means: once RawInflate is finished, $fh is left sitting at the byte directly after the compressed data stream. Chef's kiss positioning. 😌

Now, lowkey tragic comparison time — here's what that same code looks like without InputLength:

while (1) {

    # read local zip header data
    # get $filename
    # get $compressedLength

    # read all the compressed data into $data
    read($fh, $data, $compressedLength);

    my $inf = IO::Uncompress::RawInflate->new( \$data,
                        Transparent => 1 )
      or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

    my $line_count = 0;

    while (<$inf>)
    {
        ++ $line_count;
    }

    print "$filename: $line_count\n";
}

See that glow-up difference? We had to add the temporary variable $data, holding a whole copy of the compressed data while it's being uncompressed. Extra storage, extra steps, extra stress. 😭

If you know $compressedLength isn't that big, then sure, temporary storage is no problem, live your life. But if $compressedLength is very large, or you're writing an application that other people will use — meaning you have absolutely no idea how big $compressedLength will end up being — that could genuinely be an issue. 🫠

Using InputLength skips the temporary storage entirely, which means your application can cope with large compressed data streams. Main character behavior. 👑

One final point bestie — obviously InputLength can only be used when you know the length of the compressed data beforehand, like here with a zip file. If you don't know it in advance, this trick isn't available. Just so you know. ✨

💬 SUPPORT 💬

Got general feedback, questions, or bug reports? Send them over to https://github.com/pmqs//issues (preferred) or https://rt.cpan.org/Public/Dist/Display.html?Name=. They wanna hear from you, fr fr. 📬

👀 SEE ALSO 👀

Compress::Zlib
IO::Compress::Gzip
IO::Uncompress::Gunzip
IO::Compress::Deflate
IO::Uncompress::Inflate
IO::Compress::RawDeflate
IO::Uncompress::RawInflate
IO::Compress::Bzip2
IO::Uncompress::Bunzip2
IO::Compress::Lzma
IO::Uncompress::UnLzma
IO::Compress::Xz
IO::Uncompress::UnXz
IO::Compress::Lzip
IO::Uncompress::UnLzip
IO::Compress::Lzop
IO::Uncompress::UnLzop
IO::Compress::Lzf
IO::Uncompress::UnLzf
IO::Compress::Zstd
IO::Uncompress::UnZstd
IO::Uncompress::AnyInflate
IO::Uncompress::AnyUncompress
IO::Compress::FAQ
File::GlobMapper, Archive::Zip, Archive::Tar, IO::Zlib

🖊️ AUTHOR 🖊️

This module was written by Paul Marquess, "pmqs@cpan.org". Legend behavior, honestly. 🫡

📜 MODIFICATION HISTORY 📜

See the Changes file. (iykyk) 📂

⚖️ COPYRIGHT AND LICENSE ⚖️

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. Slay, and also legally binding. ✨

And that's a wrap on the whole FAQ, bestie!! 🎀 You made it to the end of a three-parter, which is honestly iconic. This was perl v5.44.0, dated 2026-10-17 — so if you're reading this in some radically different era, check the version before you trust every single line, mwah. 💅 Go forth and compress responsibly. 🦋🍓

📦 where this page comes from

arch rolling: perl 5.44.0-1 · Artistic-1.0-Perl AND GPL-1.0-or-later
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms