Catmandu::Fix(3pm)

✨ Catmandu::Fix ✨

okay bestie, welcome to Catmandu::Fix(3pm) — the Perl class that does data transformations, aka the glow-up your records have been begging for. 💅

---

🌸 NAME 🌸

Catmandu::Fix — a Catmandu class used for data transformations. yes that's it, that's the whole name, she's simple but she runs the whole show. 👑

---

🎀 SYNOPSIS 🎀

okay so first up, the command line era:

# From the command line

$ catmandu convert JSON --fix 'add(foo,bar)' < data.json
$ catmandu convert YAML --fix 'upcase(job) remove(test)' < data.yml
$ catmandu convert CSV  --fix 'sort(tags)' < data.csv
$ catmandu run /tmp/myfixes.txt
$ catmandu convert OAI --url http://biblio.ugent.be/oai --fix /tmp/myfixes.txt

and when you wanna get fancy with preprocessing:

# With preprocessing
$ catmandu convert JSON --var field=foo --fix 'add({{field}},bar)' < data.json

then from Perl, where the real main character energy lives:

# From Perl

use Catmandu;

my $fixer = Catmandu->fixer('upcase(job)','removed(test)');
my $fixer = Catmandu->fixer('/tmp/myfixes.txt');

# Convert data
my $arr      = $fixer->fix([ ... ]);
my $hash     = $fixer->fix({ ... });
my $importer = Catmandu->importer('YAML', file => 'data.yml');
my $fixed_importer = $fixer->fix($importer);

# With preprocessing
my $fixer = Catmandu::Fix->new(
    variables => {x => 'foo', y => 'bar'},
    fixes => ['add({{x}},{{y}})'],
);

# Inline fixes
use Catmandu::Fix::upcase as => 'my_upcase';
use Catmandu::Fix::remove as => 'my_remove';

my $hash = { 'job' => 'librarian' , deep => { nested => '1'} };

my_upcase($hash,'job');
my_remove($hash,'deep.nested');

literally all of that is verbatim, do not come for me. 🫶

---

🦋 DESCRIPTION 🦋

so a Catmandu::Fix is a Perl package that can transform data. these packages exist so that non-programmers can do easy data manipulation — yes bestie, you don't need to be a whole engineer for this. the main intention is to use fixes on the command line or inside Fix scripts. there's a small DSL language available so you can run many Fix commands on a stream of data. 🌈

and here's the tea: when a "fix" argument gets handed to a Catmandu::Importer, Catmandu::Exporter or Catmandu::Store, then the transformations run on every single item in the stream. every one. no skipping. periodt. 💖

---

🔥 FIX LANGUAGE 🔥

a Fix script is a collection of one or more Fix commands. the fixes execute on every record in the dataset. so if you run this on the command line:

$ catmandu convert JSON --fix 'upcase(title); add(deep.nested.field,1)' < data.json

then all the title fields get upcased and a brand new deeply nested field gets added. so this:

{ "title":"foo" }
{ "title":"bar" }

becomes this:

{ "title":"FOO" , "deep":{"nested":{"field":1}} }
{ "title":"BAR" , "deep":{"nested":{"field":1}} }

✨ ate, no crumbs.

when you're on the command line, Fix commands need a semicolon (;) as the separator. but all those same commands can go into a Fix script, where semicolons are not required:

$ catmandu convert JSON --fix script.fix < data.json

where script.fix contains:

upcase(title)
add(deep.nested.field,1)

see? no semicolons, we're chilling. 🧚

conditionals, bestie

conditionals give you the logic for when to actually run fixes:

if exists(error)
    set(valid, 0)
end

if exists(error)
    set(is_valid, 0)
elsif exists(warning)
    set(is_valid, 1)
    log(...)
else
    set(is_valid, 1)
end

unless all_match(title, "PERL")
    add(is_perl, "noooo")
end

exists(error) and set(is_valid, 0)
exists(error) && set(is_valid, 0)

exists(title) or log('title missing')
exists(title) || log('title missing')

binds 🪢

binds are how you manipulate the context in which Fixes get executed. like, say, running a fix on every item in a list:

 # 'demo' is an array of hashes
 bind list(path:demo)
    add_field(foo,bar)
 end
 # do is an alias for bind
 do list(path:demo)
    add_field(foo,bar)
 end

(and yes do is literally just an alias for bind, twins fr fr)

reject & select 💀

to delete records from a stream of data, use the "reject" Fix:

reject()           #  Reject all in the stream

if exists(foo)
    reject()       # Reject records that contain a 'foo' field
end

reject exists(foo) # Reject records that contain a 'foo' field

and the opposite of "reject" is "select" — the yin to its yang:

select()           # Keep all records in the stream

select exists(foo) # Keep only the records that contain a 'foo' field

comments 💬

comments in Fix scripts are any lines (or parts of a line) that start with a hash (#):

# This is ignored
add(test,123)      # This is also a comment

use 🎁

you can load fixes from another namespace with the "use" statement:

# this will look for fixes in the Foo::Bar namespace and make them
# available prefixed by fb
use(foo.bar, as: fb)
fb.baz()

# this will look for Foo::Bar::Condition::is_baz
if fb.is_baz()
   ...
   fix()
   ...
end

---

💅 FIX COMMANDS, ARGUMENTS AND OPTIONS 💅

Fix commands manipulate data or, in some cases, execute side effects. Fix commands have zero or more arguments and zero or more options. arguments are separated by commas ",". options are name/value pairs separated by a colon ":".

# A command with zero arguments
my_command()

# A command with multiple arguments
my_other_command(foo,bar,test)

# A command with optional arguments
my_special_command(foo,bar,color:blue,size:12)

okay important thing: all command arguments are treated as strings. these strings can be FIX PATHs pointing to values, or they can be string literals. when command line arguments don't contain special characters — comma ",", equal "=", greater than ">" or colon ":" — then you can just write them as-is. otherwise they need to be quoted with single or double quotes:

# Both commands below have the same effect
my_other_command(foo,bar,test)
my_other_command("foo","bar","test")

# Illegal syntax
my_special_command(foo,http://test.org,color:blue,size:12) # <- syntax error

# Correct syntax
my_special_command(foo,"http://test.org",color:blue,size:12)

# Or, alternative
my_special_command("foo","http://test.org",color:"blue",size:12)

not me watching you forget the quotes on a URL… bestie the syntax error is RIGHT there. 😭

---

🌈 FIX PATHS 🌈

most Fix commands use paths to point to values inside a data record. e.g. 'foo.2.bar' is a key 'bar' which is the 3-rd value of the key 'foo'. and e.g. "foo.''" is an empty string key which is the value of the key 'foo'. (yes, empty string keys are a thing, she contains multitudes 🍓)

a special case is when you want to point to all items in an array — that's when the wildcard '*' comes out to play. e.g. 'foo.*' points to all the items in the 'foo' array.

for array values there are some extra special wildcards available:

• $append — add a new item at the end of an array
• $prepend — add a new item at the start of an array
• $first — syntactic sugar for index '0' (the head of the array)
• $last — syntactic sugar for index '-1' (the tail of the array)

e.g.

# Create { mods => { titleInfo => [ { 'title' => 'a title' }] } };
add('mods.titleInfo.$append.title', 'a title');

# Create { mods => { titleInfo => [ { 'title' => 'a title' } , { 'title' => 'another title' }] } };
add('mods.titleInfo.$append.title', 'another title');

# Create { mods => { titleInfo => [ { 'title' => 'foo' } , { 'title' => 'another title' }] } };
add('mods.titleInfo.$first.title', 'foo');

# Create { mods => { titleInfo => [ { 'title' => 'foo' } , { 'title' => 'bar' }] } };
add('mods.titleInfo.$last.title', 'bar');

some Fix commands can implement an alternative path syntax to point to values. see for example Catmandu::MARC, <Catmandu:PICA>:

# Copy the MARC 245a field to the my.title field
marc_map(245a,my.title)

---

⭐ OPTIONS ⭐

🎯 fixes

an array of fixes. Catmandu::Fix will execute every fix in consecutive order. a fix can be the name of a Catmandu::Fix::* routine, or the path to a plain text file containing all the fixes to be executed. Required. (yes she's mandatory, no she will not let that go)

🎀 preprocess

if set to 1, fix files or inline fixes will first be preprocessed as a moustache template. see "variables" below for an example. default is 0 — no preprocessing.

🧁 variables

an optional hashref of variables that get used to preprocess the fix files or inline fixes as a moustache template. setting the "variables" option also sets "preprocess" to 1. sneaky but iconic.

my $fixer = Catmandu::Fix->new(
    variables => {x => 'foo', y => 'bar'},
    fixes => ['add({{x}},{{y}})'],
);
my $data = {};
$fixer->fix($data);
# $data is now {foo => 'bar'}

---

👑 METHODS 👑

fix(HASH)

execute all the fixes on a HASH. returns the fixed HASH. 💖

fix(ARRAY)

execute all the fixes on every element in the ARRAY. returns an ARRAY of fixes.

fix(Catmandu::Iterator)

execute all the fixes on every item in an Catmandu::Iterator. returns a (lazy) iterator on all the fixes. lazy queen, she'll get there when she gets there. 🦋

fix(sub {})

executes all the fixes on a generator function. returns a new generator with fixed data.

log

return the current logger. see Catmandu for activating the logger in your main code. 📝

---

okay that's part 1 bestie, and there is MORE, so don't you dare close the tab… 🌸

💻 CODING 💻

ok so bestie, you can literally extend the Fix language by writing your own custom-made fixes. iconic behaviour. there are three ways to make a new fix function:

• Simplest: make a class that implements a fix method.
• For most use cases: make a class that consumes the Catmandu::Fix::Builder role and use Catmandu::Path to build your fixer.
• Hardest: make a class that emits Perl code that will be evaled by the Fix module.

Both methods will be explained shortly. 🫶

---

🌸 Quick and easy 🌸

A Fix function is a Perl class in the Catmandu::Fix namespace that implements a fix method. The fix method accepts a Perl hash as input and returns a (fixed) Perl hash as output. As an example, the code below implements the meow Fix which inserts a 'meow' field with value 'purrrrr'.

package Catmandu::Fix::meow;

use Moo;

sub fix {
    my ($self,$data) = @_;
    $data->{meow} = 'purrrrr';
    $data;
}

1;

Given this Perl class, the following fix statement can be used in your application:

# Will add 'meow' = 'purrrrr' to the data
meow()

Use the quick and easy method when your fixes are not dependent on reading or writing data from/to a JSON path. Your Perl classes need to implement their own logic to read or write data into the given Perl hash.

Fix arguments are passed as arguments to the new function of the Perl class. As in:

# In the fix file...
meow('test123', count: 4)

# ...will be translated into this pseudo code
my $fix = Catmandu::Fix::meow->new('test123', count: 4);

Using Moo these arguments can be catched with the Catmandu::Fix::Has package:

package Catmandu::Fix::meow;

use Catmandu::Sane;
use Moo;
use Catmandu::Fix::Has;

has msg   => (fix_arg => 1); # required parameter 1
has count => (fix_opt => 1, default => sub { 4 }); # optional parameter 'count' with default value 4

sub fix {
    my ($self,$data) = @_;
    $data->{meow} = $self->msg x $self->count;
    $data;
}

1;

Using this code the fix statement can be used like:

# Will add 'meow' = 'purrpurrpurrpurr'
meow('purr', count: 4)

To allow using the fix as inline function in Perl code use Catmandu::Fix::Inlineable:

with 'Catmandu::Fix::Inlineable';

---

✨ SEE ALSO ✨

• Catmandu::Fixable
• Catmandu::Importer
• Catmandu::Exporter
• Catmandu::Store
• Catmandu::Bag

---

and that's the whole page bestie, you ate that 💅 this was perl v5.40.1, dated 2026-01-24, and it's Catmandu::Fix(3pm) — go build something cute with it 🌸