PDL::IO::FastRaw(3pm)

📦
version 2.106 · 9 versions
, in ubuntu stonking

💖 PDL::IO::FastRaw 💖

okay bestie, so this is the manual page for PDL::IO::FastRaw, which lives in the User Contributed Perl Documentation and is literally the simplest, fastest little io format for PerlDL. think of it as the raw-dump bestie your ndarrays deserve. ✨

🌸 NAME 🌸

PDL::IO::FastRaw — a simple, fast and convenient io format for PerlDL. no cap, that's the whole vibe.

✨ SYNOPSIS ✨

use PDL;
use PDL::IO::FastRaw;

writefraw($pdl,"fname");         # write a raw file

$pdl2 = readfraw("fname");       # read a raw file
$pdl2 = PDL->readfraw("fname");

gluefraw($pdlx, "fname");        # append to existing file
$pdlx->gluefraw("fname");

$pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet

$pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.

like, that's IT. four functions, one format, we're slaying. 💅

🦋 DESCRIPTION 🦋

okay so this is a very simple and fast io format for PerlDL. the disk data is made of two files:

  • a header metadata file, in ASCII
  • a binary file that is literally just consecutive bytes, shorts or whatever

the hope (and the dream, honestly) is that this makes a simple PerlDL module for saving and retrieving these files — and makes it easy for other programs to use these files too. group project energy, but the good kind. 🫶

the format of the ASCII header is simply

<typeid>
<ndims>
<dim0> <dim1> ...

you should probably stick with the default header name. but you may want to specify your own header, like when you have a big ol' collection of data files with identical dimensions and data types. in that case, just specify the "Header" option in the options hash. easy.

heads up: the binary files are in general NOT interchangeable between different architectures, because the binary file is simply dumped from the memory region of the ndarray. that's what makes the approach efficient, so like... tradeoffs, bestie. 😭

it's also possible to mmap the file, which can give a huge speedup in certain situations and save a lot of memory by using a disk file as virtual memory. when a file is mapped, parts of it are read only as they're accessed in memory — or as the kernel decides. if you're reading the pages in order, it may well preread some for you. main character kernel behavior, honestly. 👑

note that memory savings and copy-on-write are operating-system dependent — see Core.xs and your operating system documentation for the exact semantics of whatever. basically:

  • if you write to a mmapped file without "ReadOnly", the change gets reflected in the file immediately.
  • "ReadOnly" doesn't ~really~ make it impossible to write to the ndarray, but it maps the memory privately, so the file will not be changed when you change the ndarray.
be aware though: mmapping a 40Mb file without "ReadOnly" spends no virtual memory, but with "ReadOnly" it does reserve 40Mb. 💀

🍓 Example: Converting ASCII to raw

so you have a whole slew of data files in ASCII from an experiment you ran in your lab. you're still tweaking the analysis and plots, so you'd love for your data to load as fast as possible. eventually you'll read the data into your scripts using "readfraw", but the first thing you might do is create a script that converts all the data files to raw files:

#!/usr/bin/perl
# Assumes that the data files end with a .asc or .dat extension
# and saves the raw file output with a .bdat extension.
# call with
#  >./convert_to_raw.pl file1.dat file2.dat ...
# or
#  >./convert_to_raw.pl *.dat

use PDL;
use PDL::IO::FastRaw;  # for saving raw files
use PDL::IO::Misc;             # for reading ASCII files with rcols
while(shift) {                 # run through the entire supplied list of file names
        ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
        print "Saving contents of $_ to $newName\n";
        $data = rcols($_);
        writefraw($data, $newName);
}

🌈 Example: readfraw

now that you've got your data into a raw file format, you can start working on your analysis scripts. if your scripts used "rcols" in the past, the reading portion of the script should go much, much faster now:

#!/usr/bin/perl
# My plotting script.
# Assume I've specified the files to plot on the command line like
#  >./plot_script.pl file1.bdat file2.bdat ...
# or
#  >./plot_script.pl *.bdat

use PDL;
use PDL::IO::FastRaw;
while(shift) {                 # run through the entire supplied list of file names
        $data = readfraw($_);
        my_plot_func($data);
}

🎀 Example: Custom headers

in the first example, I allow "writefraw" to use the standard header file name, which would be "file.bdat.hdr". however, I often measure time series that have identical length, so all of those header files are redundant. to fix that, I simply pass the Header option to the "writefraw" command. a modified script would look like this:

#!/usr/bin/perl
# Assumes that the data files end with a .asc or .dat extension
# and saves the raw file output with a .bdat extension.
# call with
#  >./convert_to_raw.pl [-hHeaderFile] <fileglob> [-hHeaderFile] <fileglob> ...

use PDL;
use PDL::IO::FastRaw;  # for saving raw files
use PDL::IO::Misc;             # for reading ASCII files with rcols
my $header_file = undef;
CL_OPTION: while($_ = shift @ARGV) {   # run through the entire list of command-line options
        if(/-h(.*)/) {
                $header_file = $1;
                next CL_OPTION;
        }
        ($newName = $_) =~ s/\.(asc|dat)/.bdat/;
        print "Saving contents of $_ to $newName\n";
        $data = rcols($_);
        writefraw($data, $newName, {Header => $header_file});
}

modifying the read script is left as an exercise for the reader. :] (yes that's the original smiley, and yes it's iconic)

🧚 Example: Using mapfraw

sometimes you'll want to use "mapfraw" rather than the read/write functions. in fact, the original author of the module doesn't use the read/write functions anymore, prefering to always use "mapfraw". (their spelling, not mine, but we respect the legacy 💜) so how would you go about doing this?

assuming you've already saved your data into the raw format, the only change you'd have to make to the script in example 2 would be to change the call to "readfraw" to "mapfraw". that's it. you will probably see differences in performance, though I (David Mertens) couldn't tell you about them because I haven't played around with "mapfraw" much myself.

what if you eschew the use of "writefraw" and prefer to only use "mapfraw"? how would you save your data to a raw format? in that case, you'd have to create a "mapfraw" ndarray with the correct dimensions first using

$ndarray_on_hd = mapfraw('fname', {Creat => 1, Dims => [dim1, dim2, ...]});
note that you must specify the dimensions and you must tell "mapfraw" to create the new ndarray for you by setting the "Creat" option to a true value — not "Create" (note the missing final 'e'). this is not a typo, bestie, it's the actual option name. 🙈

💅 FUNCTIONS 💅

⭐ writefraw

write a raw format binary file

writefraw($pdl,"fname");
writefraw($pdl,"fname", {Header => 'headerfname'});

the "writefraw" command supports the following option:

  • Header — specify the header file name.

⭐ readfraw

read a raw format binary file

$pdl2 = readfraw("fname");
$pdl2 = PDL->readfraw("fname");
$pdl2 = readfraw("fname", {Header => 'headerfname'});

the "readfraw" command supports the following option:

  • Header — specify the header file name.

⭐ gluefraw

append a single data item to an existing binary file written by "writefraw". error if dims not compatible with existing data.

gluefraw($file, $pdl[, $opts]);

⭐ mapfraw

memory map a raw format binary file (see the module docs also)

$pdl3 = mapfraw("fname2",{ReadOnly => 1});

the "mapfraw" command supports the following options (not all combinations make sense):

  • Dims, Datatype — if creating a new file or if you want to specify your own header data for the file, you can give an array reference and a scalar, respectively.
  • Creat — create the file. also writes out a header for the file.
  • Trunc — set the file size. automatically enabled with "Creat".
NOTE: Trunc also clears the file to all zeroes. like, gone gone. 😭
  • ReadOnly — disallow writing to the file.
  • Header — specify the header file name.

⭐ maptextfraw

memory map a text file (see the module docs also).

note that this function maps the raw format, so if you are using an operating system which does strange things to e.g. line delimiters upon reading a text file, you get the raw (binary) representation.

the file doesn't really need to be text — it is just mapped as one large binary chunk.

this function is just a convenience wrapper which firsts "stat"s the file and sets the dimensions and datatype.

$pdl4 = maptextfraw("fname", {options}

the options other than Dims, Datatype of "mapfraw" are supported.

💀 BUGS 💀

should be documented better. "writefraw" and "readfraw" should also have options (the author nowadays only uses "mapfraw" ;)

👑 AUTHOR 👑

copyright (C) Tuomas J. Lukka 1997. all rights reserved. there is no warranty. you are allowed to redistribute this software / documentation under certain conditions. for details, see the file COPYING in the PDL distribution. if this file is separated from the PDL distribution, the copyright notice should be included in the file.

and that's the whole page, bestie 💖 you just read PDL::IO::FastRaw(3pm) from perl v5.42.3, dated 2026-08-20 — go forth and raw-dump ur ndarrays, mwah 🎀✨🦋

📦 where this page comes from