Unpack2(3)

📦
version 1.0 · 2 versions
, in fedora 43

💖 Unpack2(3) 💖

User Contributed Perl Documentation · Unpack2(3)

okay bestie, buckle up, we're doing the File::Unpack2 page and it's a long one (three parts, no skimping) 🫶

✨ NAME ✨

File::Unpack2 — a strong bz2/gz/zip/tar/cpio/rpm/deb/cab/lzma/7z/rar/... archive unpacker, based on mime-types 🎀

🌸 VERSION 🌸

Version 0.69 (nice) 💅

🦋 SYNOPSIS 🦋

This perl module comes with an executable script, bestie:

/usr/bin/file_unpack2 -h

/usr/bin/file_unpack2 [-1] [-m] ARCHIVE_FILE ...

File::Unpack2 is an unpacker for archives and files (bz2/gz/zip/tar/cpio/iso/rpm/deb/cab/lzma/7z/rar ... pdf/odf) based on MIME types. We call it strong because it is not fooled by file suffixes, or multiply wrapped packages. It recursively descends into each archive found until it finally exposes all unpackable payload contents. Iconic behaviour, honestly. 👑

A logfile can be written, precisely describing MIME types and unpack actions.

use File::Unpack2;

my $log;
my $u = File::Unpack2->new(logfile => \$log);

my $m = $u->mime('/etc/init.d/rc');
print "$m->[0]; charset=$m->[1]\n";
# text/x-shellscript; charset=us-ascii

map { print "$_->{name}\n" } @{$u->mime_helper()};
# application/%rpm
# application/%tar+gzip
# application/%tar+bzip2
# ...

$u->unpack("inputfile.tar.bz2");
while ($log =~ m{^\s*"(.*?)":}g) # it's JSON.
  {
    print "$1\n";   # report all files unpacked
  }

...

Most of the known archive file formats are supported. Shell-script-style plugins can be added to support additinal formats. 🐚

Helper shell-scripts can be added to support additional mime-types. Example:

$ echo "ar x $1" >
/usr/share/File-Unpack2/helper/application=x-debian-package

$ chmod a+x /usr/share/File-Unpack2/helper/application=x-debian-package

This example creates a trivial external equivalent of the builtin MIME helper for *.deb packages. For details see the documentation of the unpack() method. 📦

"unpack" examines the contents of an archive file or directory using an extensive mime-type analysis. The contents is unpacked recursively to the given destination directory; a listing of the unpacked files is reported through the built in logging facility during unpacking. Most common archive file formats are handled directly; more can easily be added as mime-type helper plugins. ✨

🧚 SUBROUTINES/METHODS 🧚

💜 new

my $u = new(destdir => '.', logfile => \*STDOUT, maxfilesize => '2G',
verbose => 1,
            world_readable => 0, one_shot => 0, no_op => 0,
archive_name_as_dir => 0,          follow_file_symlinks => 0,
log_params => {}, log_type => 'JSON');

Creates an unpacker instance, your brand new lil unpacking bestie 🫶

The parameter "destdir" must be a writable location; all output files and directories are placed inside this destdir. Subdirectories will be created in an attempt to reflect the structure of the input. Destdir defaults to the current directory; relative paths are resolved immediatly, so that chdir() after calling new is harmless. (no cap, chdir away 💅)

The parameter "logfile" can be a reference to a scalar, a filename, or a filedescriptor. The logfile starts with a JSON formatted prolog, where all lines start with printable characters. For each file unpacked, a one line record is appended, starting with a single whitespace ' ', and terminated by "\n". The format is a JSON-encoded ""key": {value},\n" pair, where key is the filename, and value is a hash including 'mime', 'size', and other information. The logfile is terminated by an epilog, where each line starts with a printable character. As part of the epilog, a dummy file named "\" with an empty hash is added to the list. It should be ignored while parsing. Per default, the logfile is sent to STDOUT. 📝

The parameter "maxfilesize" is a safeguard against compressed sparse files and test-files for archivers. Such files could easily fill up any available disk space when unpacked. Files hitting this limit will be silently truncated. Check the logfile records or epilog to see if this has happened. 👀 BSD::Resource is used manipulate RLIMIT_FSIZE.

The parameter "one_shot" can optionally be set to non-zero, to limit unpacking to one step of unpacking. Unpacking of well known compressed archives like e.g. '.tar.bz2' is considered one step only. If uncompressing an archive is considered an extra step before unpacking the archive depends on the configured mime helpers. 1️⃣

The parameter "no_op" causes unpack() to only print one shell command to STDOUT and exit. This implies one_shot=1. 🚫

The parameter "world_readable" causes unpack() change all directories to 0755, and all files to 444. Otherwise 0700 and 0400 (user readable) is asserted. 🌍

The parameter "follow_file_symlinks" causes some or all symlinks to files to be included. A value of 1 follows symlinks that exist in the input directory and point to a file. This has no effect if the input is an archive file. A value of 2 also follows symlinks that were extracted from archives.

⚠️ CAUTION: This may cause unpack() to visit files or archives elsewhere in the local filesystem. Directory symlinks are always excluded. 😭

The parameter "archive_name_as_dir" causes the unpacker to store all unpacked files inside a directory with the same name as their archive. 📁

The default depends on how many files are unpacked from the archive: If exactly one file (or one toplevel directory) is unpacked, then no extra directory is used. E.g. foo.tar.gz would unpack to foo.tar or foo-1.0.zip would unpack to foo-1.0/* and no files outside this directory. If multiple files (or directories) are unpacked, and the suffix of the archive can be removed with the "suffix_re" of its "mime_helper", then the shortened name is used as a directory. E.g. foo.tar would unpack to foo/*. Otherwise ._ is appended to the archive name. E.g. foo.tar would unpack to foo.tar._/*.

In any case, the suffix ._ or ._NNN is used to avoid conflicts with already existing names where NNN is a numeric value. 🔢

🎀 exclude

exclude(add => ['.svn', '*.orig' ], del => '.svn', force => 1,
follow_file_symlinks => 0)

Defines the exclude-list for unpacking. This list is advisory for the MIME helpers. The exclude-list items are shell glob patterns, where '*' or '?' never match '/'. 🚫

You can use force to have any of these removed after unpacking. Use (vcs => 1) to exclude a long list of known version control system directories, use (vcs => 0) to remove them. The default is "exclude(empty => 1)", which is the same as "exclude(empty_file => 1, empty_dir => 1)" — having the obvious meaning. 🗑️

(re => 1) returns the active exclude-list as a regexp pattern. Otherwise "exclude" always returns the list as an array ref.

Some symbolic links are included if {follow_file_symlinks} is nonzero. For details see "<new()>". 🔗

If exclude patterns were effective, or if symlinks, fifos, sockets, ... were encountered during unpack(), the logfile contains an additional 'skipped' keyword with statistics. 📊

🌈 unpack

$u->unpack($archive, [$destdir])

Determines the contents of an archive and recursivly extracts its files. An archive may be the pathname of a file or directory. The extracted contents will be stored in destdir/$subdir/$dest_name, where dest_name is the filename component of archive without any leading pathname components, and possibly stripped or added suffix. (Subdir defaults to ''.) If archive is a directory, then dest_name will also be a directory. If archive is a file, the type of dest_name depends on the type of packing: If the archive expands to multiple files, dest_name will be a directory, otherwise it will be a file. If a file of the same name already exists in the destination subdir, an additional subdir component is created to avoid any conflicts. ✨

For each extracted file, a record is written to the logfile. When unpacking is finished, the logfile contains one valid JSON structure. Unpack achieves this by writing suitable prolog and epilog lines to the logfile. The logfile can also be parsed line by line. All file records is one line and start with a ' ' whitespace, and end in a ',' comma. Everything else is prolog or epilog. 📜

The actual unpacking is dispatched to MIME type specific helpers, selected using "mime". A MIME helper can either be built-in code, or an external shell-script found in a directory registered with "mime_helper_dir". The standard place for external helpers is /usr/share/File-Unpack2/helper; it can be changed by the environment variable FILE_UNPACK2_HELPER_DIR or the "new" parameter "helper_dir". 🛠️

The naming of helper scripts is described under mime_helper().

A MIME helper must have executable permission and is called with 6 parameters: source_path, destfile, destination_path, mimetype, description, and config_dir. Note, that destination_path is a freshly created empty working directory, even if the unpacker is expected to unpack only a single file. The unpacker is called after chdir into destination_path, so you usually do not need to evaluate the third parameter. 👑

The directory "config_dir" contains unpack configuration in .sh, .js and possibly other formats. A MIME helper may use this information, but need not. All data passed into "new" is reflected there, as well as the active exclude-list. Using the config information can help a MIME helper to skip unwanted work or otherwise optimize unpacking. 🧠

"unpack" monitors the available filesystem space in destdir. If there is less space than configured with "minfree", a warning can be printed and unpacking is optionally paused. It also monitors the MIME helpers progress reading the archive at source_path and reports percentages to STDERR (if verbose is 1 or more). 📉

After the MIME helper is finished, "unpack" examines the files it created. If it created no files in destdir, an error is reported, and the source_path may be passed to other unpackers, or finally be added to the log as is. 😬

If the MIME helper wants to express that source_path is already unpacked as far as possible and should be added to the log without any error messages, it creates a symbolic link destdir pointing to source_path. 🔗

The system considers replacing the directory with a file, if all of the following conditions are met:

  • There is exactly one file in the directory.
  • The file name is identical with the directory name, except for one changed or removed suffix-word. (*.tar.gz -> *.tar; or *.tgz -> *.tar)
  • The file must not already exist in the parent directory.

to be continued… part 2 is coming, bestie, don't go anywhere 💖

unpack rolls out 20 empty subdirectory levels and chdirs the unpacker right into them. That number is tweakable — just hit it with new(dot_dot_safeguard => 20). While the MIME helper is doing its thing, a directory 20 levels up from the current working dir has mode 0. And if you want, unpack can also chmod(0) the parent of that subdirectory after it's chdir'd the unpacker inside — turn that on with new(jail_chmod0 => 1), it's off by default. When it's on, a MIME helper tryna drop files outside the destination_path you specified might just get slapped with a 'permission denied'. 💅

These are ✨special hacks✨ to keep badly constructed tar-balls, cpio-, or zip-archives at bay.

Heads up bestie: this can help against archives with relative paths (like ones starting with '../../../foo'), but it is ineffective against absolute paths (ones starting with '/foo'). It is the responsibility of MIME helpers to not create absolute paths, and unpack should not be run as the root user, to minimize the risk of compromising the root filesystem. 😭

A missing MIME helper is just skipped, and the next helpers may take effect. A MIME helper is expected to return an exit status of 0 on success. If it runs into a problem, it should print lines starting with the affected filenames to stderr. Such errors are recorded in the log with the unpacked archive, and — as far as files were created — also with these files.

Symbolic links are ignored while unpacking. 🚫🔗

Currently you can call unpack only once.

🏃 run

$u->run([argv0, ...], @redir, ... { init => sub ..., in, out, err, watch, every, prog, ... })

A general-purpose fork-exec wrapper, based on IPC::Run. STDIN is closed, unless you specify an in => as described in IPC::Run. STDERR and STDOUT are both printed to STDOUT, prefixed with 'E: ' and 'O: ' respectively — unless you specify out =>, err =>, or out_err => … for both. 🫶

Using redirection operators in @redir takes precedence over the in/out/err redirections above. See also IPC::Run. If you use the options in/out/err, you should restrict your redirection operators to the forms '<', '0<', '1>', '2>', or '>&' due to limitations in the precedence logic. Piping via '|' is properly recognized, but background execution '&' may confuse the precedence logic. 🙈

This run method is completly independent of the rest of File::Unpack2. It works both as a static function and as a method call. It is used internally by unpack, but is exported to be of use elsewhere.

Init is run after construction of redirects. Calling chdir() in init thus has no effect on redirects with relative paths.

Return value in scalar context is the first nonzero result code, if any. In list context all return values are returned.

🎀 fmt_run_shellcmd

File::Unpack2::fmt_run_shellcmd( $m->{argvv} )

Static function to pretty print the return value $m of method find_mime_helper(); it formats a command array used with run() as a properly escaped shell command string. ✨

🌸 mime_helper_dir, mime_helper

$u->mime_helper_dir($dir, ...)
$u->mime_helper($mime_name, $suffix_regexp, \@argv, @redir, ...)

Registers one or more directories where external MIME helper programs are found. Helper plugins are shellscripts that serve as specialized MIME type handlers for unpacking. A list of helpers comes builtin, interfacing most well-known archivers. This list can be appended to using the mime_helper_dir() or mime_helper() methods. Multiple directories can be registered — they are searched in reverse order, i.e. last added takes precedence. Any external MIME helper takes precedence over built-in code. 👑

The suffix_regexp is used to derive the destination name from the source name. It is not used for selecting helpers.

When collecting external helper scripts via mime_helper_dir(), there is no suffix_regexp. Instead, external helper scripts can explicitly create a toplevel directory with the desired name.

Helpers are mapped to MIME types by their mime_name. The name can be constructed from the MIME type by replacing the '/' with a '=' character, and by using the word 'ANY' as a wildcard component. The '=' character is interpreted as an implicit '=ANY+' if needed.

Examples:

 Mimetype                   helper names tried from top to bottom
 -----------------------------------------------------------------
 image/png                  image=png
                             image=ANY
                              image
                               ANY=png
                                ANY=ANY
                                 ANY

 application/vnd.oasis+zip  application=vnd.oasis+zip
                             application=ANY+zip
                              application=ANYzip
                               application=zip
                                application=ANY
                                     ...

A trailing '=ANY' is implicit, as shown by these examples. The rules for precedence are this:

  • Search in the latest directory is exhaused first, then the previously added directory is considered in turn, up to all directories have been traversed, or until a matching helper is found.
  • A matching name with wildcards has lower precedence than a matching name without.
  • A wildcard before the '=' sign lowers precedence more than one after it.

The mapping takes place when mime_helper_dir is called. Adding helper scripts to a directory afterwards has no effect. mime_helper does not do any implicit expansions. Call it multiple times with the same helper command and different names if needed. The default argument list is %(src)s %(destfile)s %(destdir)s %(mime)s %(descr)s %(configdir)s — this is applied, if no args are given and no redirections are given. See also unpack for more semantics and how a helper should behave. 💖

Both methods return an ARRAY-ref of HASHes describing all known (old and newly added) mime helpers.

📋 list

Returns an ARRAY of preformatted patterns and MIME helpers.

Example:

printf @$_ for $u->list();

🔍 find_mime_helper

$u->find_mime_helper($mimetype)

Returns a MIME helper suitable for unpacking the given $mimetype. If called in list context, a second return value indicates which mime helpers would be suitable, but could not be found in the system. 😭

💾 minfree

$u->minfree(factor => 10, bytes => '100M', percent => '3%', warning => sub { .. })
THESE TESTS ARE TO BE IMPLEMENTED.

Guard the filesystem (destdir) against becoming full during unpack. Before unpacking each source archive, the free space is measured and compared against three conditions:

  • The archive size multiplied with the given factor must fit into the filesystem.
  • The given number of bytes (in optional K, M, G, or T units) must be free.
  • The filesystem must have at least the given free percentage. The '%' character is optional.

The warning method is called if any of the above conditions fail. Its signature is:

&warning->($pathname, $full_percentage, $free_bytes, $free_inodes);

It is expected to print an appropriate warning message, and delay a few seconds. It should return 0 to cause a retry. It should return nonzero to continue unpacking. The default warning method prints a message to STDERR, waits 30 seconds, and returns 0.

The filesystem may still become full and unpacking may fail, if e.g. factor was chosen lower than the average compression ratio of the archives. 💀

🧚 mime

$u->mime($filename)

$u->mime(file => $filename)

$u->mime(buf => "#!/bin ...", file => "what-was-read")

$u->mime(fd => \*STDIN, file => "what-was-opened")

Determines the MIME type (and optionally additional information) of a file. The file can be specified by filename, by a provided buffer, or an opened file descriptor. For the latter two cases, specifying a filename is optional, and used only for diagnostics. 🦋

mime uses libmagic by Christos Zoulas exposed via File::LibMagic and also uses the shared-mime-info database from freedesktop.org exposed via File::MimeInfo::Magic, if available. Either one is sufficient, but having both is better. LibMagic sometimes says 'text/x-pascal', although we have a .desktop file, or says 'text/plain', but has contradicting details in its description.

File::MimeInfo::Magic::magic is consulted where the libmagic output is dubious. E.g. when the desciption says something interesting like 'Debian binary package (format 2.0)' but the mimetype says 'application/octet-stream'. The combination of both libraries gives us excellent reliability in the critical field of MIME type recognition. 🔥

This implementation also features multi-level MIME type recognition for efficient unpacking. When e.g. unpacking a large bzipped tar archive, this saves us from creating a huge temporary tar-file which unpack would extract in a second step. The multi-level recognition returns 'application/x-tar+bzip2' in this case, and allows for a MIME helper to e.g. pipe the bzip2 contents into tar (which is exactly what tar jxvf does, making a very simple and efficient MIME helper). ⭐

mime returns a 3 or 4 element arrayref with mimetype, charset, description, diff; where diff is only present when the libfile and shared-mime-info methods disagree.

In case of 'text/plain', an additional rule based on file name suffix is used to allow recognition of well known plain text pack formats. We return 'text/x-suffix-XX+plain', where XX is one of the recognized suffixes (in all lower case and without the dot). E.g. a plain mmencoded file has no header and looks like 'plain/text' to all the known magic libraries. We recognize the suffixes .mm, .b64, and .base64 for this (case insignificant). A similar rule exitst for 'application/octect-stream'. It may trigger e.g. for LZMA compressed files which fail to provide a magic number.

Examples:

 [ 'text/x-perl', 'us-ascii', 'a /usr/bin/perl -w script text']

 [ 'text/x-mpegurl', 'utf-8', 'M3U playlist text',
   [ 'text/plain', 'application/x-mpegurl']]

 [ 'application/x-tar+bzip2, 'binary',
   "bzip2 compressed data, block size = 900k\nPOSIX tar archive (GNU)", ...]

👑 AUTHOR

Juergen Weigert, "<jnw at cpan.org>"

💀 BUGS

The implementation of mime is an ugly hack. We suffer from the existence of multiple file magic databases, and multiple conflicting implementations. With Perl we have at least 5 modules for this; here we use two.

The builtin list of MIME helpers is incomplete. Please submit your handler code. 🫶

Please report any bugs or feature requests to "bug-file-unpack at rt.cpan.org", or through the web interface at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=File-Unpack2>. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

ok so when they were building File::Unpack2, the author went on a whole ✨journey✨ through a bunch of other perl modules. Some of them are straight-up carrying the team (dependencies + recommendations), and others got a close look and lowkey got their wheels reinvented here. no shade, just facts 💅

🌸 Used Modules 🌸

  • File::LibMagic — this is the preferred mimetype engine, periodt. It does NOT care about the suffix (iconic behaviour tbh), it recognizes more types than any of the alternatives, and on openSUSE systems it's literally the exact same engine as /usr/bin/file. It also hands back charset and description info. The description gets crossreferenced against the mimetype to sniff out weaknesses, and then File::MimeInfo::Magic plus some homegrown logic gets consulted for stuff like detecting LZMA compression, which fails to provide any recognizable magic. Required if you use mime; otherwise not a hard requirement.
  • File::MimeInfo::Magic — this one uses BOTH magic info and file suffixes to figure out the mimetype. Its magic() function gets called in a few cases where File::LibMagic drops the ball. Example: as of June 2010, libmagic does not recognize 'image/x-targa'. File::MimeInfo::Magic may be slower, but bestie, it features the shared-mime-info database from freedesktop.org. Recommended if you use mime.
  • String::ShellQuote — used to call external MIME helpers. Required.
  • BSD::Resource — used to reliably restrict the maximum file size. Recommended.
  • File::Path — mkpath(). Required.
  • Cwd — fast_abs_path(). Required.
  • JSON — used for formatting the logfile. Required.

🙈 Modules Not Used 🙈

  • Archive::Extract — so this one tries to determine what type of archive you're passing it by inspecting its suffix first. And perldoc itself says: 'Maybe this module should use something like "File::Type" to determine the type, rather than blindly trust the suffix'. [quoted from perldoc] 💀

If you set $Archive::Extract::PREFER_BIN to 1, it will prefer using command line programs and won't eat so much memory. Default: it uses Archive::Tar.

  • Archive::Zip — perldoc says: If you are just going to be extracting zips (and/or other archives) you are recommended to look at using Archive::Extract. [quoted from perldoc] It's pure perl, so it's a lot slower than your /usr/bin/zip.
  • Archive::Tar — it's pure Perl, so it's a lot slower than your /bin/tar. And it's heavy on memory, everything gets read into memory. [quoted from perldoc]
  • File::MMagic, File::MMagic::XS, File::Type — compared to File::LibMagic and File::MimeInfo::Magic, these three are just... inferior, ngl. They often say 'text/plain' or 'application/octet-stream' where the latter two actually report useful mimetypes. not them being vague 😭

🫶 SUPPORT 🫶

You can find documentation for this module with the perldoc command, bestie.

perldoc File::Unpack2

You can also go looking for info at:

  • RT: CPAN's request tracker

<http://rt.cpan.org/NoAuth/Bugs.html?Dist=File-Unpack2>

  • AnnoCPAN: Annotated CPAN documentation

<http://annocpan.org/dist/File-Unpack2>

  • CPAN Ratings

<http://cpanratings.perl.org/d/File-Unpack2>

  • Search CPAN

<http://search.cpan.org/dist/File-Unpack2/>

🌈 SOURCE REPOSITORY 🌈

<http://search.cpan.org/search?query=File%3A%3AUnpack2>

<https://github.com/openSUSE/perl-File-Unpack2>

git clone <https://github.com/openSUSE/perl-File-Unpack2.git>

💖 ACKNOWLEDGEMENTS 💖

MIME type recognition relies heavily on libmagic by Christos Zoulas. The author had long hesitated implementing File::Unpack2, but set to work when he dicovered that File::LibMagic brings your library to perl. Thanks Christos. And thanks for tcsh too. 🎀

Copyright 2023 Sebastian Riedel Copyright 2010,2011,2012,2013 Juergen Weigert.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.

and that's the whole page, bestie 🦋 you made it to the end of all three parts, main character energy fr. this little icon of a page is perl v5.42.0, dated 2025-07-25, and lives in section Unpack2(3). go forth and unpack responsibly 💅✨

📦 where this page comes from