Catman...Breaker(3pm)

💖 Catmandu::Breaker — the Catman...Breaker(3pm) page, bestie edition 💖

okay so. you opened the perl manual for Catmandu::Breaker, a 3pm page (that's the perl module section, iykyk), and it's literally the package that exports data in a Breaker format. that's the whole name: Catmandu::Breaker — Package that exports data in a Breaker format. and yes the page header is all "Catman...Breaker(3pm)" because terminals gotta truncate, she's iconic like that ✨

---

🌈 SYNOPSIS 🌈

okay here's the vibe, straight from the command line:

# From the command line

# Using the default breaker
$ catmandu convert JSON to Breaker < data.json

# Break a OAI-PMH harvest
$ catmandu convert OAI --url http://biblio.ugent.be/oai to Breaker

# Using a MARC breaker
$ catmandu convert MARC to Breaker --handler marc < data.mrc

# Using an XML breaker plus create a list of unique record fields
$ catmandu convert XML --path book to Breaker --handler xml --fields data.fields < t/book.xml > data.breaker

# Find the usage statistics of fields in the XML file above
$ catmandu breaker data.breaker

# Use the list of unique fields in the report
$ catmandu breaker --fields data.fields data.breaker

# verbose output
$ catmandu breaker -v data.breaker

# The breaker commands needs to know the unique fields in the dataset to build statistics.
# By default it will scan the whole file for fields. This can be a very
# time consuming process. With --maxscan one can limit the number of lines
# in the breaker file that can be scanned for unique fields
$ catmandu breaker -v --maxscan 1000000 data.breaker

# Alternatively the fields option can be used to specify the unique fields
$ catmandu breaker -v --fields 245a,022a data.breaker

$ cat data.breaker | cut -f 2 | sort -u > data.fields
$ catmandu breaker -v --fields data.fields data.breaker

# Export statistics as CSV. See L<Catmandu::Exporter::Stat> for supported formats.
$ catmandu breaker --as CSV data.breaker

lowkey important bestie: the breaker commands need to know the unique fields in the dataset to build statistics, and by default it scans the whole file for them — which can be a very time consuming process. that's literally why --maxscan exists 🫶

---

🦋 DESCRIPTION 🦋

inspired by the article "Metadata Analysis at the Command-Line" by Mark Phillips, over at <http://journal.code4lib.org/articles/7818> 💅 — this exporter takes metadata records and breaks them into the Breaker format, which you can then analyze further with command line tools. no cap, that's the whole point. slay.

---

🎀 BREAKER FORMAT 🎀

when you break an input using catmandu convert {format} to Breaker, every metadata field gets transformed into a breaker format:

<record-identifier><tab><metadata-field><tab><metadata-value><tab><metadatavalue>...

for the default JSON breaker, the input format gets broken down into JSON-like Paths. E.g. when you give this YAML input:

---
name: John
colors:
   - black
   - yellow
   - red
institution:
   name: Acme
   years:
      - 1949
      - 1950
      - 1951
      - 1952

then the breaker command catmandu convert YAML to Breaker < file.yml will generate:

1 colors[]  black
1 colors[]  yellow
1 colors[]  red
1 institution.name  Acme
1 institution.years[] 1949
1 institution.years[] 1950
1 institution.years[] 1951
1 institution.years[] 1952
1 name  John

so, the columns? let me break it down bestie:

• the first column is a counter for each record — or the content of the _id field when present 💫
• the second column gives you a JSON path to the data (with the array-paths translated to [])
• the third column is the field value

you can absolutely pair this output with Unix tools like grep, sort, cut, etc. to inspect the breaker output:

$ catmandu convert YAML to Breaker < file.yml | grep 'institution.years'

✨ handlers, because some formats are extra ✨

some input formats — like MARC — don't get much out of the JSON-path format, because the field names are part of the data. so it's possible to use a special handler to create a more verbose breaker output 👑

for instance, without a special handler:

$ catmandu convert MARC to Breaker < t/camel.usmarc
fol05731351   record[][]  LDR
fol05731351   record[][]  _
fol05731351   record[][]  00755cam  22002414a 4500
fol05731351   record[][]  001
fol05731351   record[][]  _
fol05731351   record[][]  fol05731351
fol05731351   record[][]  082
fol05731351   record[][]  0
fol05731351   record[][]  0
fol05731351   record[][]  a

with the special marc handler:

$ catmandu convert MARC to Breaker --handler marc < t/camel.usmarc

fol05731351   LDR 00755cam  22002414a 4500
fol05731351   001 fol05731351
fol05731351   003 IMchF
fol05731351   005 20000613133448.0
fol05731351   008 000107s2000    nyua          001 0 eng
fol05731351   010a     00020737
fol05731351   020a  0471383147 (paper/cd-rom : alk. paper)
fol05731351   040a  DLC
fol05731351   040c  DLC
fol05731351   040d  DLC

and the handler roster, bestie 🧚:

• for the Catmandu::PICA tools a pica handler is available.
• for the Catmandu::MAB2 tools a mab handler is available.
• for the Catmandu::XML tools an xml handler is available:

$ catmandu convert XML --path book to Breaker --handler xml < t/book.xml

---

👑 BREAKER STATISTICS 👑

statistical information can be calculated from a breaker output using the catmandu breaker command:

$ catmandu convert MARC to Breaker --handler marc < t/camel.usmarc > data.breaker
$ catmandu breaker data.breaker

| name | count | zeros | zeros% | min | max | mean | median | mode   | variance | stdev | uniq%| entropy |
|------|-------|-------|--------|-----|-----|------|--------|--------|----------|-------|------|---------|
| 001  | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 100  | 3.3/3.3 |
| 003  | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 10   | 0.0/3.3 |
| 005  | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 100  | 3.3/3.3 |
| 008  | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 100  | 3.3/3.3 |
| 010a | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 100  | 3.3/3.3 |
| 020a | 9     | 1     | 10.0   | 0   | 1   | 0.9  | 1      | 1      | 0.09     | 0.3   | 90   | 3.3/3.3 |
| 040a | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 10   | 0.0/3.3 |
| 040c | 10    | 0     | 0.0    | 1   | 1   | 1    | 1      | 1      | 0        | 0     | 10   | 0.0/3.3 |
| 040d | 5     | 5     | 50.0   | 0   | 1   | 0.5  | 0.5    | [0, 1] | 0.25     | 0.5   | 10   | 1.0/3.3 |

the output table gives you statistical info on the usage of fields in the original format 💖 we see that the 001 field was counted 10 times in the data set, but the 040d value is only present 5 times. the 020a is empty in 10% (zeros%) of the records. the 001 has very unique values (entropy is maximum), but all 040c fields contain the same information (entropy is minimum). the way that table just tells you things?? obsessed 🫶

see Catmandu::Exporter::Stat for more information about the statistical fields and supported output formats.

---

🍓 MODULES 🍓

• Catmandu::Exporter::Breaker
• Catmandu::Cmd::breaker

---

🌸 SEE ALSO 🌸

Catmandu, Catmandu::MARC, Catmandu::XML, Catmandu::Stat

---

💜 AUTHOR 💜

Patrick Hochstenbach, <patrick.hochstenbach at ugent.be>

💜 CONTRIBUTORS 💜

Jakob Voss, nichtich at cpan.org

Johann Rolschewski, jorol at cpan.org

---

and that's the page bestie!! 💅 this cutie was documented as perl v5.36.0, dated 2023-02-08, and lives on your system as the Catman...Breaker(3pm) page 🎀 go break some metadata, you main character ✨🦋