Catman...porter(3pm)

💖 NAME 💖

Catmandu::Importer — bestie, it's the namespace for packages that can import 🫶

Like, if something in Catmandu-land pulls data in? It lives here. That's the whole vibe. ✨

---

✨ SYNOPSIS ✨

okay so first, the command line era 💅

# From the command line

# JSON is an importer and YAML an exporter
$ catmandu convert JSON to YAML < data.json

# OAI is an importer and JSON an exporter
$ catmandu convert OAI --url http://biblio.ugent.be/oai to JSON

# Fetch remote content
$ catmandu convert JSON --file http://example.com/data.json to YAML

and now the Perl era, because we contain multitudes 🦋

# From Perl

use Catmandu;
use Data::Dumper;

my $importer = Catmandu->importer('JSON', file => 'data.json');

$importer->each(sub {
    my $item = shift;
    print Dumper($item);
});

my $num = $importer->count;

my $first_item = $importer->first;

# Convert OAI to JSON in Perl
my $importer = Catmandu->importer('OAI', url => 'http://biblio.ugent.be/oai');
my $exporter = Catmandu->exporter('JSON');

$exporter->add_many($importer);

---

🌸 DESCRIPTION 🌸

A Catmandu::Importer is a Perl package that can generate structured data from sources such as JSON, YAML, XML, RDF or network protocols such as Atom, OAI-PMH, SRU and even DBI databases. no cap, it really does all that 😭

Given an Catmandu::Importer a programmer can read data from using one of the many Catmandu::Iterable methods:

$importer->to_array;
$importer->count;
$importer->each(\&callback);
$importer->first;
$importer->rest;
...etc...

Every Catmandu::Importer is also Catmandu::Fixable and thus inherits a 'fix' parameter that can be set in the constructor. When given a 'fix' parameter, then each item returned by the generator will be automatically Fixed using one or more Catmandu::Fixes. E.g.

my $importer = Catmandu->importer('JSON',fix => ['upcase(title)']);
$importer->each( sub {
    my $item = shift ; # Every $item->{title} is now upcased...

});

# or via a Fix file
my $importer = Catmandu->importer('JSON',fix => ['/my/fixes.txt']);
$importer->each( sub {
    my $item = shift ; # Every $item->{title} is now upcased...

});

the way that fix just... does it for you? iconic. 👑

---

🎀 CONFIGURATION 🎀

• file — Read input from a local file given by its path. If the path looks like a url, the content will be fetched first and then passed to the importer. Alternatively a scalar reference can be passed to read from a string. 💅
• fh — Read input from an IO::Handle. If not specified, Catmandu::Util::io is used to create the input stream from the "file" argument or by using STDIN.
• encoding — Binmode of the input stream "fh". Set to ":utf8" by default. 🦋
• fix — An ARRAY of one or more Fix-es or Fix scripts to be applied to imported items.
• data_path — The data at "data_path" is imported instead of the original data. not me obsessing over nested structures 🙈

# given this imported item:
{abc => [{a=>1},{b=>2},{c=>3}]}
# with data_path 'abc', this item gets imported instead:
[{a=>1},{b=>2},{c=>3}]
# with data_path 'abc.*', 3 items get imported:
{a=>1}
{b=>2}
{c=>3}

• variables — Variables given here will interpolate the "file" and "http_body" options. The syntax is the same as URI::Template.

# named arguments
my $importer = Catmandu->importer('JSON',
    file => 'http://{server}/{path}',
    variables => {server => 'biblio.ugent.be', path => 'file.json'},
);

# positional arguments
my $importer = Catmandu->importer('JSON',
    file => 'http://{server}/{path}',
    variables => 'biblio.ugent.be,file.json',
);

# or
my $importer = Catmandu->importer('JSON',
    url => 'http://{server}/{path}',
    variables => ['biblio.ugent.be','file.json'],
);

# or via the command line
$ catmandu convert JSON --file 'http://{server}/{path}' --variables 'biblio.ugent.be,file.json'

---

🌈 HTTP CONFIGURATION 🌈

These options are only relevant if "file" is a url. See LWP::UserAgent for details about these options. 👀

• http_body — Set the GET/POST message body.
• http_method — Set the type of HTTP request 'GET', 'POST' , ...
• http_headers — A reference to a HTTP::Headers objects.

---

💅 Set an own HTTP client 💅

• user_agent(LWP::UserAgent->new(...)) — Set an own HTTP client

---

🦋 Alternative set the parameters of the default client 🦋

• http_agent — A string containing the name of the HTTP client.
• http_max_redirect — Maximum number of HTTP redirects allowed.
• http_timeout — Maximum execution time.
• http_verify_hostname — Verify the SSL certificate. 🔒
• http_retry — Maximum times to retry the HTTP request if it temporarily fails. Default is not to retry. See LWP::UserAgent::Determined for the HTTP status codes that initiate a retry.
• http_timing — Maximum times and timeouts to retry the HTTP request if it temporarily fails. Default is not to retry. See LWP::UserAgent::Determined for the HTTP status codes that initiate a retry and the format of the timing value.

---

💜 METHODS 💜

first, each, rest , ...

See Catmandu::Iterable for all inherited methods. 🫶

---

🍓 CODING 🍓

Create your own importer by creating a Perl package in the Catmandu::Importer namespace that implements "Catmandu::Importer". Basically, you need to create a method 'generate' which returns a callback that creates one Perl hash for each call:

my $importer = Catmandu::Importer::Hello->new;

$importer->generate(); # record
$importer->generate(); # next record
$importer->generate(); # undef = end of stream

Here is an example of a simple "Hello" importer:

package Catmandu::Importer::Hello;

use Catmandu::Sane;
use Moo;

with 'Catmandu::Importer';

sub generator {
    my ($self) = @_;
    state $fh = $self->fh;
    my $n = 0;
    return sub {
        $self->log->debug("generating record " . ++$n);
        my $name = $self->fh->readline;
        return defined $name ? { "hello" => $name } : undef;
    };
}

1;

This importer can be called via the command line as:

$ catmandu convert Hello to JSON < /tmp/names.txt
$ catmandu convert Hello to YAML < /tmp/names.txt
$ catmandu import Hello to MongoDB --database_name test < /tmp/names.txt

Or, via Perl

use Catmandu;

my $importer = Catmandu->importer('Hello', file => '/tmp/names.txt');
$importer->each(sub {
    my $items = shift;
});

like?? writing your own importer?? main character energy fr fr 👑

---

⭐ SEE ALSO ⭐

Catmandu::Iterable , Catmandu::Fix , Catmandu::Importer::CSV, Catmandu::Importer::JSON , Catmandu::Importer::YAML

---

and that's the whole page bestie 💖 this one came from perl v5.40.1, dated 2026-01-24, and yeah the header/footer said Catman...porter(3pm) because terminal columns are haters. go import something slay 💅✨