UUlib(3)

💖 UUlib(3) — Convert::UUlib 💖

okay so bestie, welcome to UUlib(3) — the Convert::UUlib page from User Contributed Perl Documentation, part 1 of 2 🫶. Grab a snack, we're decoding a truly massive pile of files.

✨ NAME ✨

Convert::UUlib — decode uu/xx/b64/mime/yenc/etc-encoded data from a massive number of files. That's the whole vibe: decoder central, no cap. 💅

🌸 SYNOPSIS 🌸

Short version, bestie: load files, loop over them, check state, decode, print filename. Like this, exactly:

use Convert::UUlib ':all';

# read all the files named on the commandline and decode them
# into the CURRENT directory. See below for a longer example.
LoadFile $_ for @ARGV;

for my $uu (GetFileList) {
   if ($uu->state & FILE_OK) {
     $uu->decode;
     print $uu->filename, "\n";
   }
}

🦋 DESCRIPTION 🦋

This module started as an interface to the uulib/uudeview library by Frank Pilhofer, which can be used to decode all kinds of usenet (and other) binary messages. Then upstream abandoned the project, and the library got continuously bugfixed and improved in this module, with major focuses on security fixes, correctness and speed. That does not mean this library is considered safe with untrusted data, but it surely is safer than the original uudeview. 🫶

Read the file doc/library.pdf from the distribution for in-depth information about the C-library used in this interface, and the rest of this document and especially the non-trivial decoder program at the end. iykyk. 📚

💅 EXPORTED CONSTANTS 💅

🎬 Action code constants 🎬

• ACT_IDLE — we don't do anything 😴
• ACT_SCANNING — scanning an input file 🔍
• ACT_DECODING — decoding into a temp file 🧚
• ACT_COPYING — copying temp to target 📦
• ACT_ENCODING — encoding a file ✨

🚨 Message severity levels 🚨

• MSG_MESSAGE — just a message, nothing important 💬
• MSG_NOTE — something that should be noticed 👀
• MSG_WARNING — important msg, processing continues ⚠️
• MSG_ERROR — processing has been terminated 💀
• MSG_FATAL — decoder cannot process further requests 🛑
• MSG_PANIC — recovery impossible, app must terminate 😱

⚙️ Options ⚙️

• OPT_VERSION — version number MAJOR.MINORplPATCH (ro) 👑
• OPT_FAST — assumes only one part per file 🏃
• OPT_DUMBNESS — switch off the program's intelligence 😵‍💫
• OPT_BRACKPOL — give numbers in [] higher precedence 🧠
• OPT_VERBOSE — generate informative messages 🗣️
• OPT_DESPERATE — try to decode incomplete files 😭
• OPT_IGNREPLY — ignore RE:plies (off by default) 🙈
• OPT_OVERWRITE — whether it's OK to overwrite ex. files 🧐
• OPT_SAVEPATH — prefix to save-files on disk 📁
• OPT_IGNMODE — ignore the original file mode 🙅
• OPT_DEBUG — print messages with FILE/LINE info 🐛
• OPT_ERRNO — get last error code for RET_IOERR (ro) 🚨
• OPT_PROGRESS — retrieve progress information 📊
• OPT_USETEXT — handle text messages 📝
• OPT_PREAMB — handle Mime preambles/epilogues 🎀
• OPT_TINYB64 — detect short B64 outside of Mime 🔎
• OPT_ENCEXT — extension for single-part encoded files 🏷️
• OPT_REMOVE — remove input files after decoding (dangerous) 💀
• OPT_MOREMIME — strict MIME adherence 🧐
• OPT_DOTDOT — ".."-unescaping has not yet been done on input files 🧚
• OPT_RBUF — set default read I/O buffer size in bytes 📥
• OPT_WBUF — set default write I/O buffer size in bytes 📤
• OPT_AUTOCHECK — automatically check file list after every loadfile ✅

📋 Result/Error codes 📋

• RET_OK — everything went fine ✅
• RET_IOERR — I/O Error - examine errno 🚨
• RET_NOMEM — not enough memory 😵
• RET_ILLVAL — illegal value for operation 🚫
• RET_NODATA — decoder didn't find any data 🕳️
• RET_NOEND — encoded data wasn't ended properly 🔚
• RET_UNSUP — unsupported function (encoding) 🙅
• RET_EXISTS — file exists (decoding) 📁
• RET_CONT — continue -- special from ScanPart ➡️
• RET_CANCEL — operation canceled ❌

🗂️ File States 🗂️

This code is zero, i.e. "false":

• UUFILE_READ — Read in, but not further processed 🫶

The following state codes are or'ed together:

• FILE_MISPART — Missing Part(s) detected 🧩
• FILE_NOBEGIN — No 'begin' found 🚫
• FILE_NOEND — No 'end' found 🔚
• FILE_NODATA — File does not contain valid uudata 🕳️
• FILE_OK — All Parts found, ready to decode ✅
• FILE_ERROR — Error while decoding 💀
• FILE_DECODED — Successfully decoded 🎉
• FILE_TMPFILE — Temporary decoded file exists 📄

🧬 Encoding types 🧬

• UU_ENCODED — UUencoded data 💌
• B64_ENCODED — Mime-Base64 data 🧬
• XX_ENCODED — XXencoded data ❌
• BH_ENCODED — Binhex encoded 🐝
• PT_ENCODED — Plain-Text encoded (MIME) 📝
• QP_ENCODED — Quoted-Printable (MIME) 🧾
• YENC_ENCODED — yEnc encoded (non-MIME) 📦

✨ EXPORTED FUNCTIONS ✨

🧼 Initializing and cleanup 🧼

Initialize is automatically called when the module is loaded and allocates quite a small amount of memory for today's machines ;) CleanUp releases that again.

On my machine, a fairly complete decode with DBI backend needs about 10MB RSS to decode 20000 files.

🧹 CleanUp 🧹

Release memory, file items and clean up files. Should be called after a decoding run, if you want to start a new one. 🫧

⚙️ Setting and querying options ⚙️

$option = GetOption OPT_xxx
SetOption OPT_xxx, opt-value

See the OPT_xxx constants above to see which options exist. Easy peasy. 💅

📞 Setting various callbacks 📞

SetMsgCallback [callback-function]
SetBusyCallback [callback-function]
SetFileCallback [callback-function]
SetFNameFilter [callback-function]

🎀 Call the currently selected FNameFilter 🎀

$file = FNameFilter $file

📥 Loading sourcefiles, optionally fuzzy merge and start decoding 📥

🫶 ($retval, $count) = LoadFile $fname, [$id, [$delflag, [$partno]]] 🫶

Load the given file and scan it for encoded contents. Optionally tag it with the given $id, and if $delflag is true, delete the file after it is no longer necessary. If you are certain of the part number, you can specify it as the last argument.

A better (usually faster) way of doing this is using the SetFNameFilter functionality. 🔥

🌀 $retval = Smerge $pass 🌀

If you are desperate, try to call Smerge with increasing $pass values, beginning at 0, to try to merge parts that usually would not have been merged.

Careful, bestie: most probably this will result in garbled files, so never do this by default, except: if the OPT_AUTOCHECK option has been disabled (by default it is enabled) to speed up file loading, then you have to call Smerge -1 after loading all files as an additional pre-pass (which is normally done by LoadFile). 😭

🔎 $item = GetFileListItem $item_number 🔎

Return the $item structure for the $item_number'th found file, or undef if no file with that number exists.

The first file has number 0, and the series has no holes, so you can iterate over all files by starting with zero and incrementing until you hit undef.

This function has to walk the linear list of files on each access, so if you want to iterate over all items, it is usually faster to use GetFileList.

💖 @items = GetFileList 💖

Similar to GetFileListItem, but returns all files in one go, which is very much faster for large number of items, and has no drawbacks when used for a small number of items. Obsessed. 💖

🧚 Decoding files 🧚

🏷️ $retval = $item->rename ($newname) 🏷️

Change the ondisk filename where the decoded file will be saved.

📄 $retval = $item->decode_temp 📄

Decode the file into a temporary location, use $item->infile to retrieve the temporary filename.

🗑️ $retval = $item->remove_temp 🗑️

Remove the temporarily decoded file again.

🎯 $retval = $item->decode ([$target_path]) 🎯

Decode the file to its destination, or the given target path.

📞 $retval = $item->info (callback-function) 📞

That's the signature, bestie — this one is just listed here. 🫶

🔍 Querying (and setting) item attributes 🔍

$state    = $item->state
$mode     = $item->mode ([newmode])
$uudet    = $item->uudet
$size     = $item->size
$filename = $item->filename ([newfilename})
$subfname = $item->subfname
$mimeid   = $item->mimeid
$mimetype = $item->mimetype
$binfile  = $item->binfile

🧩 Information about source parts 🧩

🧵 $parts = $item->parts 🧵

Return information about all parts (source files) used to decode the file as a list of hashrefs with the following structure:

 {
   partno   => <integer describing the part number, starting with 1>,
   # the following member sonly exist when they contain useful information
   sfname   => <local pathname of the file where this part is from>,
   filename => <the ondisk filename of the decoded file>,
   subfname => <used to cluster postings, possibly the posting filename>,
   subject  => <the subject of the posting/mail>,
   origin   => <the possible source (From) address>,
   mimetype => <the possible mimetype of the decoded file>,
   mimeid   => <the id part of the Content-Type>,
 }

Usually you are interested mostly the sfname and possibly the partno and filename members.

🙈 Functions below are not documented and not very well tested - feedback welcome 🙈

• QuickDecode
• EncodeMulti
• EncodePartial
• EncodeToStream
• EncodeToFile
• E_PrepSingle
• E_PrepPartial

These are here, but the original page says they're not documented and not very well tested — feedback welcome. So don't come for me if they act up. 😭

🌈 EXTENSION FUNCTIONS 🌈

Functions found in this module but not documented in the uulib documentation:

• $msg = straction ACT_xxx — Return a human readable string representing the given action code. 🎬
• $msg = strerror RET_xxx — Return a human readable string representing the given error code. 🚨
• $str = strencoding xxx_ENCODED — Return the name of the encoding type as a string. 🏷️
• $str = strmsglevel MSG_xxx — Returns the message level as a string. 📶

🫶 SetFileNameCallback $cb 🫶

Sets (or queries) the FileNameCallback, which is called whenever the decoding library can't find a filename and wants to extract a filename from the subject line of a posting. The callback will be called with two arguments, the subject line and the current candidate for the filename. The latter argument can be undef, which means that no filename could be found (and likely no one exists, so it is safe to also return undef in this case). If it doesn't return anything (not even undef!), then nothing happens, so this is a no-op callback:

sub cb {
   return ();
}

If it returns undef, then this indicates that no filename could be found. In all other cases, the return value is taken to be the filename.

This is a slightly more useful callback:

sub cb {
   return unless $_[1]; # skip "Re:"-plies et al.
   my ($subject, $filename) = @_;
   # if we find some *.rar, take it
   return $1 if $subject =~ /(\w+\.rar)/;
   # otherwise just pass what we have
   return ();
}

🌸 LARGE EXAMPLE DECODER 🌸

okay so the decode workflow is literally this, bestie:

1. Set options with SetOption or SetXXXCallback.
2. Load all source files with LoadFile.
3. Optionally Smerge.
4. Iterate over all GetFileList items (i.e. result files).
5. CleanUp to delete files and free items.

The page then gives you the actual example-decoder file from the distribution to illustrate that workflow in a non-trivial example. Here it is, verbatim, no cap:

#!/usr/bin/perl

# decode all the files in the directory uusrc/ and copy
# the resulting files to uudst/

use Convert::UUlib ':all';

sub namefilter {
   my ($path) = @_;

   $path=~s/^.*[\/\\]//;

   $path
}

sub busycb {
   my ($action, $curfile, $partno, $numparts, $percent, $fsize) = @_;
   $_[0]=straction($action);
   print "busy_callback(", (join ",",@_), ")\n";
   0
}

SetOption OPT_RBUF, 128*1024;
SetOption OPT_WBUF, 1024*1024;
SetOption OPT_IGNMODE, 1;
SetOption OPT_IGNMODE, 1;
SetOption OPT_VERBOSE, 1;
SetOption OPT_AUTOCHK, 0;

# show the three ways you can set callback functions. I normally
# prefer the one with the sub inplace.
SetFNameFilter \&namefilter;

SetBusyCallback "busycb", 333;

SetMsgCallback sub {
   my ($msg, $level) = @_;
   print uc strmsglevel $_[1], ": $msg\n";
};

# the following non-trivial FileNameCallback takes care
# of some subject lines not detected properly by uulib:
SetFileNameCallback sub {
   return unless $_[1]; # skip "Re:"-plies et al.
   local $_ = $_[0];

   # the following rules are rather effective on some newsgroups,
   # like alt.binaries.games.anime, where non-mime, uuencoded data
   # is very common

   # if we find some *.rar, take it as the filename
   return $1 if /(\S{3,}\.(?:[rstuvwxyz]\d\d|rar))\s/i;

   # one common subject format
   return $1 if /- "(.{2,}?\..+?)" (?:yenc )?\(\d+\/\d+\)/i;

   # - filename.par (04/55)
   return $1 if /- "?(\S{3,}\.\S+?)"? (?:yenc )?\(\d+\/\d+\)/i;

   # - (xxx) No. 1 sayuri81.jpg 756565 bytes
   # - (20 files) No.17 Roseanne.jpg [2/2]
   return $1 if /No\.[ 0-9]+ (\S+\....) (?:\d+ bytes )?\[/;

   # try to detect some common forms of filenames
   return $1 if /([a-z0-9_\-+.]{3,}\.[a-z]{3,4}(?:.\d+))/i;

   # otherwise just pass what we have
   ()
};

# now read all files in the directory uusrc/*
for (<uusrc/*>) {
   my ($retval, $count) = LoadFile ($_, $_, 1);
   print "file($_), status(", strerror $retval, ") parts($count)\n";
}

Smerge -1;

SetOption OPT_SAVEPATH, "uudst/";

# now wade through all files and their source parts
for my $uu (GetFileList) {
   print "file ", $uu->filename, "\n";
   print " state ", $uu->state, "\n";
   print " mode ", $uu->mode, "\n";
   print " uudet ", strencoding $uu->uudet, "\n";
   print " size ", $uu->size, "\n";
   print " subfname ", $uu->subfname, "\n";
   print " mimeid ", $uu->mimeid, "\n";
   print " mimetype ", $uu->mimetype, "\n";

   # print additional info about all parts
   print " parts";
   for ($uu->parts) {
      for my $k (sort keys %$_) {
         print " $k=$_->{$k}";
      }
      print "\n";
   }

   $uu->remove_temp;

   if (my $err = $uu->decode) {
      print " ERROR ", strerror $err, "\n";
   } else {
      print " successfully saved as uudst/", $uu->filename, "\n";
   }
}

print "cleanup...\n";

CleanUp;

That's the whole non-trivial example — LoadFile, optional Smerge, GetFileList, then CleanUp. Iconic. 💅

🧚 PERLMULTICORE SUPPORT 🧚

This module supports the perlmulticore standard (see <http://perlmulticore.schmorp.de/> for more info) for the following functions — generally these are the ones accessing the disk and/or using considerable CPU time:

LoadFile
$item->decode
$item->decode_temp
$item->remove_temp
$item->info

The perl interpreter gets reacquired/released on every callback invocation, so for performance reasons, callbacks should be avoided if that is costly.

Future versions might enable multicore support for more functions.

💀 BUGS AND LIMITATIONS 💀

The original uulib library this module uses was written back when main memory was measured in megabytes and buffer overflows as a security thing weren't even a concept. A lot of security fixes have been applied over the years — including some defense in depth mechanism that can shield against a lot of as-of-yet undetected bugs — but using this library for security purposes requires care.

Likewise, file sizes when the uulib library was written were tiny compared to today, so do not expect this library to handle files larger than 2GB.

Lastly, this module uses a very "C-like" interface, which means it doesn't protect you from invalid points as you might expect from "more perlish" modules — for example, accessing a file item object after calling CleanUp will likely result in crashes, memory corruption, or worse. 💀

👑 AUTHOR 👑

Marc Lehmann <schmorp@schmorp.de> — the original uulib library was written by Frank Pilhofer <fp@informatik.uni-frankfurt.de>, and later heavily bugfixed by Marc Lehmann.

🦋 SEE ALSO 🦋

perl(1), and the uudeview homepage at <http://www.fpx.de/fp/Software/UUDeview/>. Go click around, bestie. ✨

---

This page is UUlib(3) from perl v5.40.2, dated 2024-06-10 — love u, go un-uuencode your dreams 💖🎀