Fsdb::F...ipeline(3)

📦
version 3.9 · 3 versions
, in epel 9

🎀 Fsdb::F...ipeline(3) 🎀

User Contributed Perl Documentation — aka the manual for dbpipeline, the module that lets you assemble db commands as pipelines inside Perl. 💅✨

✨ NAME ✨

dbpipeline — allow db commands to be assembled as pipelines in Perl. 🫶

🌸 SYNOPSIS 🌸

use Fsdb::Filter::dbpipeline qw(:all);
dbpipeline(
    dbrow(qw(name test1)),
    dbroweval('_test1 += 5;')
);

Or, for more customized versions, go peek at "dbpipeline_filter", "dbpipeline_sink", "dbpipeline_open2", and "dbpipeline_close2_hash". ✨

🦋 DESCRIPTION 🦋

This module makes it easy to create pipelines in Perl using separate processes. (Back in the day, we used to use Perl threads. 👵) By default — same as all Fsdb modules — input comes from STDIN and output goes to STDOUT. The two helper functions fromfile and tofile can grab data from files. 💖

dbpipeline is lowkey different from all the other Fsdb::Filter modules in a few ways: it does not have a corresponding Unix command (it is used only from within Perl), and it does not log its presence to the output stream. That last part is arguably a bug, but it doesn't actually do anything. 💀

💅 OPTIONS 💅

Unlike most Fsdb modules, dbpipeline defaults to --autorun. Periodt.

This module also supports the standard fsdb options:

  • -d — enable debugging output.
  • -i or --input InputSource — read from InputSource, typically a file name, or "-" for standard input, or (if you're in Perl) an IO::Handle, Fsdb::IO, or Fsdb::BoundedQueue object.
  • -o or --output OutputDestination — write to OutputDestination, typically a file name, or "-" for standard output, or (if you're in Perl) an IO::Handle, Fsdb::IO, or Fsdb::BoundedQueue object.
  • --autorun or --noautorun — by default, programs process automatically, but Fsdb::Filter objects in Perl do not run until you invoke the run() method. The --(no)autorun option controls that behavior within Perl. 👑
  • --header H — use H as the full Fsdb header, rather than reading a header from the input. This option is especially useful when using Fsdb under Hadoop, where split files don't have headers. 🫶
  • --help — show help.
  • --man — show the full manual.

🌈 SEE ALSO 🌈

Fsdb(3) — go read it, bestie. 💜

🧚 CLASS FUNCTIONS 🧚

dbpipeline

dbpipeline(@modules);

This shorthand routine creates a dbpipeline object and then immediately runs it. So your Perl code becomes nearly as terse as shell code:

dbpipeline(
    dbcol(qw(name test1)),
    dbroweval('_test1 += 5;'),
);

The following commands currently have shorthand aliases:

and

  • dbsubprocess(3)

dbpipeline_filter

my($result_reader, $fred) = dbpipeline_filter($source, $result_reader_aref, @modules);

Set up a pipeline of @MODULES that filters data pushed through it, where the data comes from $SOURCE (any Fsdb::Filter::parse_io_option object, like a Fsdb::IO::Reader object, queue, or filename). 💖

Returns a $RESULT_READER Fsdb::IO::Reader object, created with $RESULT_READER_AREF as options. This reader will produce the filtered data, and a $FRED that must be joined to guarantee output has completed. 🔥

Or, if $RESULT_READER_AREF is "[-raw_fh, 1]", it just returns the IO::Handle to the pipe. 🫶

As an example, this code uses "dbpipeline_filter" to insure the input (from $in, which is a filename or Fsdb::IO::Reader) is sorted numerically by column "x":

use Fsdb::Filter::dbpipeline qw(dbpipeline_filter dbsort);
my($new_in, $new_fred) = dbpipeline_filter($in,
    [-comment_handler => $self->create_delay_comments_sub],
    dbsort(qw(--nolog -n x)));
while (my $fref = $new_in->read_rowwobj()) {
    # do something
};
$new_in->close;
$new_fred->join();

dbpipeline_sink

my($fsdb_writer, $fred) = dbpipeline_sink($writer_arguments_aref, @modules);

Set up a pipeline of @MODULES that is a data "sink", where the output is given by a "--output" argument, or goes to standard output by default. The caller generates input into the pipeline by writing to a newly created $FSDB_WRITER, whose configuration is specified by the mandatory first argument $WRITER_ARGUMENTS_AREF. (These arguments should include the schema.) Returns this writer, and a $FRED that must be joined to guarantee output has completed. 🫶

If the first argument to modules is "--fred_exit_sub", then the second is taken as a CODE block that runs at fred exit (and the two are not passed to modules). If the first argument to modules is "--fred_description", then the second is taken as a text description of the Fred. 🧚

dbpipeline_open2

my($fsdb_reader_fh, $fsdb_writer, $fred) =
    dbpipeline_open2($writer_arguments_aref, @modules);

Set up a pipeline of @MODULES that is a data sink and source (both!). The caller generates input into the pipeline by writing to a newly created $FSDB_WRITER, whose configuration is specified by the mandatory argument $WRITER_ARGUMENTS_AREF. (These arguments should include the schema.) The output of the pipeline comes out to the newly created $FSDB_READER_FH. Returns this read queue and writer, and a $PID that must be joined to guarantee output has completed. 💀

(Unfortunately, the interface is asymmetric with a read queue but a write "Fsdb::IO" object, because "Fsdb::IO::Reader" blocks on input of the header.) Like IPC::Open2, with all of its pros and cons like potential deadlock. 😭

dbpipeline_close2_hash

my($href) = dbpipeline_close2_hash($fsdb_read_fh, $fsdb_writer, $pid);

Reads and returns one line of output from $FSDB_READER, after closing $FSDB_WRITER and joining the $PID. Useful, for example, to get dbcolstats output cleanly. ✨

new

$filter = new Fsdb::Filter::dbpipeline(@arguments);

set_defaults

$filter->set_defaults();

Internal: set up defaults. 🎀

parse_options

$filter->parse_options(@ARGV);

Internal: parse options. 👑

setup

$filter->_reap();

Internal: reap any forked threads.

setup

$filter->setup();

Internal: setup, parse headers.

run

$filter->run();

Internal: run over all IO. 🌀

finish

$filter->finish();

Internal: we would write a trailer, but we don't, because we depend on the last command in the pipeline to do that. We don't actually have a valid output stream. 💀

Copyright (C) 1991-2018 by John Heidemann <johnh@isi.edu>.

This program is distributed under terms of the GNU general public license, version 2. See the file COPYING with the distribution for details. 💖

That's the whole vibe check, bestie. This page is from perl v5.32.1, dated 2024-08-02, and it's the Fsdb::F...ipeline(3) manual. Mwah. 💋✨

📦 where this page comes from