MARC::...utorial(3pm)

🌸 MARC::Doc::Tutorial — the manual page (that's MARC::...utorial(3pm) in User Contributed Perl Documentation, bestie) 🎀

✨ NAME ✨

MARC::Doc::Tutorial — a documentation-only module for new users of MARC::Record. That's it, that's the whole job, she just sits there and explains things 💅

---

✨ SYNOPSIS ✨

perldoc MARC::Doc::Tutorial

that's literally it. one line. go read it 🫶

---

🦋 INTRODUCTION 🦋

📚 What is MARC?

okay so MARC stands for MAchine Readable Cataloging, and it was designed by the Library of Congress back in the late 1960s so libraries could take their card catalogs and turn them into a digital format. iconic behavior honestly 👑

and like, the advantages of having computerized card catalogs were realized so fast, and now MARC is being used by all sorts of libraries around the world to give people computerized access to their collections. 🌍

here's the tea though: MARC data in transmission format is optimized for processing by computers, so it's not very readable for the normal human. it's giving robot language fr fr 😭

if you wanna know more about the MARC format, go visit the Library of Congress at http://www.loc.gov/marc/ — they have the lore 📖

🍓 What is this Tutorial?

the document you're reading rn is a beginners guide to using Perl to processing MARC data, written in the 'cookbook' style. inside, you'll find recipes on how to read, write, update and convert MARC data using the MARC::Record CPAN package. 🧑🍳

and like any cookbook, you should feel free to dip in at any section and use the recipe you find interesting. if you're new to Perl, you might wanna read from the beginning though, no cap.

this document is distributed with the MARC::Record package, but in case you're reading it somewhere else, you can find the latest version at CPAN: http://www.cpan.org/modules/by-module/MARC/. ✨

you'll notice that some sections aren't filled in yet, which is a result of this document being a work in progress. not me exposing the author's WIP 😭 if you have ideas for new sections, please make a suggestion to perl4lib: https://perl4lib.perl.org/.

💜 History of MARC on CPAN

In 1999, a group of developers began working on MARC.pm to provide a Perl module for working with MARC data. MARC.pm was quite successful since it grew to include many new options that were requested by the Perl/library community. slay for them.

However — and this is the plot twist — in adding these features the module swiftly outgrew its own clothes, and maintenance and addition of new features became extremely difficult. 💀

on top of that, as libraries began using MARC.pm to process large MARC data files (>1000 records), they noticed that memory consumption would skyrocket. 🚀

memory consumption became an issue for large batches of records because MARC.pm's object model was based on the 'batch' rather than the record... so each record in the file would often be read into memory. there were ways of getting around this, but they were not obvious. some effort was made to reconcile the two approaches (batch and record), but with limited success. 😔

In mid 2001, Andy Lester released MARC::Record and MARC::Field, which provided a much simpler and maintainable package for processing MARC data with Perl. and as its name suggests, MARC::Record treats an individual MARC record as the primary Perl object, rather than having the object represent a given set of records. which, honestly? main character energy 👑

instead of forking the two projects, the developers agreed to encourage use of the MARC::Record framework, and to work on enhancing MARC::Record rather than extending MARC.pm further. soon afterwards, MARC::Batch was added, which allows you to read in a large data file without having to worry about memory consumption. 🧚

In Dec., 2004, the MARC::Lint module — an extension to check the validity of MARC records — was removed from the MARC::Record distribution, to become a separately distributed package. this tutorial contains examples for using MARC::Lint. 📝

🌈 Brief Overview of MARC Classes

the MARC::Record package is made up of several separate packages. this can be somewhat confusing to people new to Perl, or Object Oriented Programming. however, this framework allows easy extension, and is built to support new input/output formats as their need arises. 💪

for a good introduction to using the object oriented features of Perl, see the perlboot documentation that came with your version of Perl.

here are the packages that get installed with MARC::Record:

🎀 MARC::Batch

A convenience class for accessing MARC data contained in an external file.

🎀 MARC::Field

An object for representing the indicators and subfields of a single MARC field.

🎀 MARC::Record

This primary class represents a MARC record, being a container for multiple MARC::Field objects. the whole point ✨

🎀 MARC::Doc::Tutorial

This document! hi bestie, you're reading her 💖

🎀 MARC::File

A superclass for representing files of MARC data.

🎀 MARC::File::MicroLIF

A subclass of MARC::File for working with data encoded in the MicroLIF format.

🎀 MARC::File::USMARC

A subclass of MARC::File for working with data encoded in the USMARC format.

👑 Help Wanted!

it's already been mentioned but it's worth mentioning again: MARC::Doc::Tutorial is a work in progress, and you are encouraged to submit any suggestions for additional recipes via the perl4lib mailing list at https://perl4lib.perl.org/.

also, patches and issue reports are welcome at https://github.com/perl4lib/marc-perl. 🫶

---

yes this is a work in progress and yes you can contribute. we love a collaborative queen 🌸

---

more coming in the next parts, don't go anywhere bestie 💕

💖 READING 💖

Okay bestie, this is the part where we actually do stuff with records. Strap in. 🎀

🌸 Reading a record from a file 🌸

Say you've got a USMARC record chillin' in 'file.dat' and you just wanna read it in and print its title. That's it. That's the whole goal. And it's like... four lines?? Obsessed.

1   ## Example R1
2
3   ## create a MARC::Batch object.
4   use MARC::Batch;
5   my $batch = MARC::Batch->new('USMARC', 'file.dat');
6
7   ## get a MARC record from the MARC::Batch object.
8   ## the $record will be a MARC::Record object.
9   my $record = $batch->next();
10
11   ## print the title contained in the record.
12   print $record->title(),"\n";

If you run this against the distribution's 't/camel.usmarc', your result should be, and I quote:

ActivePerl with ASP and ADO / Tobias Martinsson.

Iconic. No notes. 💅

---

🦋 Iterating through a batch file 🦋

So now plot twist: 'file.dat' actually has multiple records and you want every title, not just one. Good news — your program barely changes at all. You literally just wrap a loop around your next() call. That's the whole update. Slay.

1   ## Example R2
2
3   ## create a MARC::Batch object.
4   use MARC::Batch;
5   my $batch = MARC::Batch->new('USMARC','file.dat');
6
7   while (my $record = $batch->next()) {
8
9     ## print the title contained in the record.
10     print $record->title(),"\n";
11
12   }

Here's the tea on line 7: that call to next() hands you the next record from the file, and when there are no more records left it returns undef — which is what makes the while loop stop. It's a genuinely useful idiom for reading in all the records in a file, no cap.

With 'camel.usmarc' your results should look like:

ActivePerl with ASP and ADO / Tobias Martinsson.
Programming the Perl DBI / Alligator Descartes and Tim Bunce.
.
.
.
Cross-platform Perl / Eric F. Johnson.

(the . . . is the vibes of "and a bunch more in between", not actual output, just so we're clear 💀)

---

😭 Checking for errors 😭

Bestie. Get in the habit of checking for errors. Please. MARC/Perl was literally designed to help you do this.

Here's the tricky bit: calls to next() while iterating a batch file return undef when there are no more records to return... AND also when an error was encountered. Same undef, two completely different meanings. (The next recipe shows you how to subvert that, don't worry.) So you probs wanna verify that you didn't just abruptly stop reading a batch file because something went wrong.

1   ## Example R3
2
3   ## create a MARC::Batch object.
4   use MARC::Batch;
5   my $batch = MARC::Batch->new('USMARC','file.dat');
6
7   ## get a marc record from the MARC::Batch object.
8   ## $record will be a MARC::Record object.
9   while ( my $record = $batch->next() ) {
10       print $record->title(),"\n";
11   }
12
13   ## make sure there weren't any problems.
14   if ( my @warnings = $batch->warnings() ) {
15       print "\nWarnings were detected!\n", @warnings;
16   }

That warnings() call on line 14 grabs any warning messages and stashes them in @warnings, so you can tell when next() bailed early — like, before it actually reached the end of the file. And when a warning gets detected, an explanation goes off to STDERR. So if we sneak an error into 'camel.usmarc', we'd get this on STDOUT:

Warnings were detected!
Invalid indicators "a0" forced to blanks in record 1 for tag 245

---

🙈 Recovering from errors 🙈

Maybe you don't want to stop reading a batch file just because an error showed up? Valid. In that case, turn strict mode off with the strict_off() method. You can also stop warnings from being printed to STDERR with warnings_off().

Now listen — by default, strict is on, and it's on as a safety precaution so you don't go around using corrupt MARC data like it's fine. Once you've turned it off, you can turn both strict and warnings back on with the strict_on() and warnings_on() methods.

1   ## Example R4
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC', 'file.dat');
5   $batch->strict_off();
6
7   while ( my $record = $batch->next() ) {
8      print $record->title(),"\n";
9   }
10
11   ## make sure there weren't any problems.
12   if ( my @warnings = $batch->warnings() ) {
13       print "\nWarnings were detected!\n", @warnings;
14   }

Introducing a second error to the 'camel.usmarc' file gives us:

ActivePerl with ASP and ADO / Tobias Martinsson.
Programming the Perl DBI / Alligator Descartes and Tim Bunce.
.
.
.
Cross-platform Perl / Eric F. Johnson.

Warnings were detected!
Invalid indicators "a0" forced to blanks in record 1 for tag 245
Invalid indicators "a0" forced to blanks in record 5 for tag 245

Two warnings, still got all the titles. We love a graceful recovery. 🫶

---

✨ Looking at a field ✨

Everything so far used MARC::Record's title() method to peep the 245 field, but real talk — you're gonna want programs that read lots of other MARC fields. That's what field() is for. It gives you complete access to the data in any MARC field, and it hands you back a MARC::Field object, which you can use to get at the data, the indicators, and even the individual subfields. Behold:

1   ## Example R5
2
3   ## open a file.
4   use MARC::Batch;
5   my $batch = MARC::Batch->new('USMARC','file.dat');
6
7   ## read a record.
8   my $record = $batch->next();
9
10   ## get the 100 field as a MARC::Field object.
11   my $field = $record->field('100');
12   print "The 100 field contains: ",$field->as_string(),"\n";
13   print "The 1st indicator is ",$field->indicator(1),"\n";
14   print "The 2nd indicator is ",$field->indicator(2),"\n";
15   print "Subfield d contains: ",$field->subfield('d'),"\n";

Which gives you something like:

The 100 field contains: Martinsson, Tobias, 1976-
The 1st indicator is 1
The 2nd indicator is
Subfield d contains: 1976-

(yeah the 2nd indicator is just... blank. it's giving minimal. 🍓)

And same as before — throw a while loop on it to iterate through all the records in a batch.

---

👑 Looking at repeatable fields 👑

So what about fields that show up more than once?? How do you get at those? Good news, field() has your back here too.

In our previous example, line 11 used field() in a scalar context — because the result was being assigned to the variable $field. But in list context, field() returns all the fields in the record of that particular type. Like so:

1   ## Example R6
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','file.dat');
5   my $record = $batch->next();
6
7   ## get all the 650 fields (list context).
8   my @fields = $record->field('650');
9
10   ## examine each 650 field and print it out.
11   foreach my $field (@fields) {
12     print $field->as_string(),"\n";
13   }

Which prints this out for the first record of 't/camel.usmarc':

Active server pages.
ActiveX.

Two 650s, two lines. We ate. 🔥

---

🌈 Looking at a set of related fields 🌈

field() also lets you grab a whole family of similar fields by using . as a wildcard. Yes bestie, a wildcard.

1   ## Example R7
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','file.dat');
5   my $record = $batch->next();
6
7   # retrieve all title fields in one shot.
8   foreach my $field ($record->field('2..')) {
9     print $field->tag(),' contains ',$field->as_string(),"\n";
10   }

Notice the shorthand going on in line 8 — it compacts lines 7-13 of our previous example into one neat little thing. Instead of storing the fields in an array first, field() still returns a list, right there in the for loop.

Line 9 uses the tag() method, which returns the tag number for a particular MARC field. Super useful when you're not 100% sure which tag you're currently dealing with. Sample output from this recipe:

245 contains ActivePerl with ASP and ADO / Tobias Martinsson.
260 contains New York : John Wiley & Sons, 2000.

You can also return all the tags for a specific record by using '...' in field — though, see the next recipe. 👀

---

💜 Looking at all the fields in a record 💜

Last one in this little section: how to retrieve every field in a record using the fields() method. It's similar to passing '...' as a wildcard (and if you want the alternative access route, check the previous recipe — that's literally what I just told you to do).

1   ## Example R8
2
3   use MARC::Batch;
4   my $file = MARC::Batch->new('USMARC','file.dat');
5   my $record = $batch->next();
6
7   ## get all of the fields using the fields() method.
8   my @fields = $record->fields();
9
10   ## print out the tag, the indicators and the field contents.
11   foreach my $field (@fields) {
12     print
13       $field->tag(), " ",
14       defined $field->indicator(1) ? $field->indicator(1) : "",
15       defined $field->indicator(2) ? $field->indicator(2) : "",
16       " ", $field->as_string, " \n";
17   }

The above code would print the following for the first record of 't/camel.usmarc':

001  fol05731351
003  IMchF
.
.
.
300    xxi, 289 p. : ill. ; 23 cm. + 1 computer  laser disc (4 3/4 in.)
500    "Wiley Computer Publishing."
650  0 Perl (Computer program language)
630 00 Active server pages.
630 00 ActiveX.

Every tag, every indicator, every field content. The whole record, laid bare. Main character energy, periodt. ✨

✨ CREATING ✨

So in Section 1 we were reading USMARC data that already existed in a file, right? Well, Section 2 is where we make a MARC record out of absolutely nothing. Like, from scratch-scratch. 💅

And bestie, this is the good stuff — the techniques here let you write programs that create MARC records that you could then load into an online catalog or send off to a third party. Main character energy fr fr. 👑

---

🎀 Creating a record 🎀

To create a brand new MARC record you gotta do three-ish things, in order:

1. Make a MARC::Record object
2. Add a leader (though ngl, MARC::Record can create leaders automatically if you don't specifically define one — so this one's optional)
3. Create MARC::Field objects and add them to your MARC::Record object

Like so:

  1   ## Example C1
  2
  3   ## create a MARC::Record object.
  4   use MARC::Record;
  5   my $record = MARC::Record->new();
  6
  7   ## add the leader to the record. optional.
  8   $record->leader('00903pam  2200265 a 4500');
  9
 10   ## create an author field.
 11   my $author = MARC::Field->new(
 12     '100',1,'',
 13       a => 'Logan, Robert K.',
 14       d => '1939-'
 15     );
 16   $record->append_fields($author);
 17
 18   ## create a title field.
 19   my $title = MARC::Field->new(
 20     '245','1','4',
 21       a => 'The alphabet effect /',
 22       c => 'Robert K. Logan.'
 23     );
 24   $record->append_fields($title);

The ✨key✨ to creating records from scratch is append_fields(), which adds a field to the end of the record. And because every single field gets added at the end, it's up to you to order the fields the way you want them. No one's gonna do it for you, bestie.

insert_fields_before() and insert_fields_after() are similar methods that let you define where the field gets added instead. We get into those in more detail below. 🫶

---

🌈 WRITING 🌈

Sections 1 and 2 showed you how to read and create USMARC data. But once you know how to read and create, it becomes super important to know how to write that USMARC data to disk, otherwise your work just kinda... evaporates. 😭 And we're not doing that.

In these examples we're gonna create a new record and save it to a file called record.dat.

---

🌸 Writing records to a file 🌸

  1   ## Example W1
  2
  3   ## create a MARC::Record object.
  4   use MARC::Record;
  5   my $record = MARC::Record->new();
  6
  7   ## add the leader to the record. optional.
  8   $record->leader('00903pam  2200265 a 4500');
  9
 10   ## create an author field.
 11   my $author = MARC::Field->new(
 12     '100',1,'',
 13       a => 'Logan, Robert K.',
 14       d => '1939-'
 15     );
 16
 17   ## create a title field.
 18   my $title = MARC::Field->new(
 19     '245','1','4',
 20       a => 'The alphabet effect /',
 21       c => 'Robert K. Logan.'
 22     );
 23
 24   $record->append_fields($author, $title);
 25
 26   ## open a filehandle to write to 'record.dat'.
 27   open(OUTPUT, '> record.dat') or die $!;
 28   print OUTPUT $record->as_usmarc();
 29   close(OUTPUT);

Okay so the as_usmarc() method call on line 28? It returns a scalar value that is the raw USMARC data for $record. And then that raw data gets promptly printed to the OUTPUT file handle. Chef's kiss. 👩‍🍳💋

Want to output multiple records to a file? Just repeat the process at line 28 for the additional records. Boom, done.

Also, notice something about the append_fields method here: unlike recipe C1, which called the method once for each field added, this recipe shows that append_fields can accept multiple arguments. Slay. 💅

Note to the curious: the as_usmarc() method is actually an alias to the MARC::File::USMARC encode() method. Having separate encode() methods is a design feature of the MARC class hierarchy, since it lets extensions be built that translate MARC::Record objects into different data formats. iykyk 🧚

---

🦋 Debugging with as_formatted() 🦋

Raw USMARC data is... not exactly a vibe to read with human eyes. 💀 So it's often useful to see the contents of your MARC::Record object represented in a 'pretty' way for debugging purposes. Got a MARC::Record object you'd like to pretty-print? Use the as_formatted() method.

  1   ## Example W2
  2
  3   ## create a MARC::Record object.
  4   use MARC::Record;
  5   my $record = MARC::Record->new();
  6
  7   $record->leader('00903pam  2200265 a 4500');
  8
  9   $record->append_fields(
 10    MARC::Field->new('100','1','', a=>'Logan, Robert K.', d=>'1939-'),
 11    MARC::Field->new('245','1','4', a=>'The alphabet effect /', c=>'Robert K. Logan.')
 12   );
 13
 14   ## pretty print the record.
 15   print $record->as_formatted(), "\n";

And this code will pretty print the contents of the newly created record:

  LDR 00903pam  2200265 a 4500
  100 1  _aLogan, Robert K.
         _d1939-
  245 14 _aThe alphabet effect /
         _cRobert K. Logan.

Okay but notice on lines 9-12 how you can add a whole list of new fields by creating MARC::Field objects right inside a call to append_fields(). That's yet another shorthand method compared to the ones shown in recipes C1 and W1. Obsessed. 😍

And for more pretty-printing capabilities, try marcdump() in our next recipe.

---

⭐ Debugging with marcdump() ⭐

Say you've written USMARC data to a file (like in recipe W2) and now you wanna verify that the data is stored correctly. Enter the marcdump command line utility, which was installed with the MARC::Record package:

 % marcdump record.dat
 record.dat
 LDR 00122pam  2200049 a 4500
 100 1  _aLogan, Robert K.
        _d1939-
 245 14 _aThe alphabet effect /
        _cRobert K. Logan.

  Recs  Errs Filename
 ----- ----- --------
     1     0 record.dat

As you can see, this command results in the record being pretty printed to your screen (STDOUT) — similarly to the as_formatted method from recipe W2. It's genuinely useful for verifying your USMARC data after it's been stored on disk. And there's more debugging tea later in VALIDATING. 👀

---

💖 UPDATING 💖

Okay so now you know how to read, write and create MARC data. Which means you have all the tools you need to update or edit existing MARC data. And updating MARC data is a super common task for library catalogers.

Sometimes there are, like, huge amounts of records that need to be touched up... and while those touch-ups are very detail oriented, they're also highly repetitive. 🙈 Luckily, computers are tireless and not very prone to error (assuming the programmer isn't).

So here's the situation: when libraries receive large batches of MARC records for electronic text collections such as NetLibrary, Making of America, or microfiche sets like Early American Imprints, the records are often loaded into an online system and then the system is used to update the records. Unfortunately, not all these systems are created equal, and catalogers end up spending a great deal of time touching up each individual record. 😭

An alternative would be to process the records prior to import — and then, once they're in the system, the records wouldn't need editing at all. This scenario would save the cataloger a great deal of time, freeing them up to spend that time doing original cataloging... which computers are notably bad at! 💀

---

🍓 Adding a field 🍓

Picture a batch of records in file.dat that you'd like to add local notes (590) to, and then save your changes:

  1   ## Example U1
  2
  3   ## create our MARC::Batch object.
  4   use MARC::Batch;
  5   my $batch = MARC::Batch->new('USMARC','file.dat');
  6
  7   ## open a file handle to write to.
  8   open(OUT,'>new.dat') or die $!;
  9
 10   ## read each record, modify, then print.
 11   while ( my $record = $batch->next() ) {
 12
 13       ## add a 590 field.
 14       $record->append_fields(
 15          MARC::Field->new('590','','',a=>'Access provided by Enron.')
 16       );
 17
 18       print OUT $record->as_usmarc();
 19
 20   }
 21
 22   close(OUT);

---

💜 Preserving field order 💜

As its name suggests, append_fields() will add the 590 field in recipe U1 to the end of the record. But if you want to preserve a particular order, you can use the insert_fields_before() and insert_fields_after() methods instead. To use them, you need to locate the field you want to insert before or after first. Here's an example (insert_fields_after() works similarly):

  1   ## Example U2
  2
  3   use MARC::Batch;
  4   my $batch = MARC::Batch->new('USMARC','file.dat');
  5   open(OUT,'>new.dat') or die $!;
  6
  7   ## read in each record.
  8   while ( my $record = $batch->next() ) {
  9
 10       ## find the tag after 590.
 11       my $before;
 12       foreach ($record->fields()) {
 13           $before = $_;
 14           last if $_->tag() > 590;
 15       }
 16
 17       ## create the 590 field.
 18       my $new = MARC::Field->new('590','','',a=>'Access provided by Enron.');
 19
 20       ## insert our 590 field after the $before.
 21       $record->insert_fields_before($before,$new);
 22
 23       ## and print out the new record.
 24       print OUT $record->as_usmarc();
 25
 26   }

---

🧚 Deleting a field 🧚

You can also delete fields you don't want anymore. But bestie, you're probably gonna want to check that the field actually contains what you expect before deleting it. Let's say Enron has gone out of business and that 590 field needs to be deleted:

  1   ## Example U3
  2
  3   use MARC::Batch;
  4   my $batch = MARC::Batch->new('USMARC','new.dat');
  5   open(OUT,'>newer.dat') or die $1;
  6
  7   while ( my $record = $batch->next() ) {
  8
  9     ## get the 590 record.
 10     my $field = $record->field('590');
 11
 12     ## if there is a 590 AND it has the word "Enron"...
 13     if ($field and $field->as_string() =~ /Enron/i) {
 14
 15       ## delete it!
 16       $record->delete_field($field);
 17
 18     }
 19
 20     ## output possibly modified record.
 21     print OUT $record->as_usmarc();
 22
 23   }

So the 590 field gets retrieved on line 10 — but notice how we check that we actually received a valid $field, and that it then contains the word 'Enron', before we delete anything. Very demure, very mindful. 💅

And heads up: you need to pass delete_field() a MARC::Field object that can be retrieved with the field() method. 🫶

💖 Changing existing fields 💖

okay so maybe you don't wanna add or delete a whole field — maybe you just need to modify one that's already there. bestie, that's a whole three-step vibe: first read in the MARC record you wanna update, then grab the field you care about. from there, call the field's update or replace_with methods to change its contents, then resave the record. lowkey iconic. here's an example where we update existing 590 fields that contain the word 'Enron' to say access is now provided through Arthur Andersen:

1   ## Example U4
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','new.dat');
5   open(OUT,'>newer.dat') or die $1;
6
7   while ( my $record = $batch->next() ) {
8
9     ## look for a 590 containing "Enron"...
10     my $field = $record->field('590');
11     if ($field and $field->as_string =~ /Enron/i) {
12
13       ## create a new 590 field.
14       my $new_field = MARC::Field->new(
15         '590','','', a => 'Access provided by Arthur Andersen.' );
16
17       ## replace existing with our new one.
18       $field->replace_with($new_field);
19
20     }
21
22     ## output possibly modified record.
23     print OUT $record->as_usmarc();
24
25   }

so in this example, we used MARC::Field's replace_with() method to swap an existing field in the record for a brand-new field we made. to use replace_with(), you gotta retrieve the field you wanna replace from a MARC::Record object (line 10), create a new field to replace the existing one with (lines 13-15), and then call the existing field's replace_with() method passing the new field as an argument (line 18). and yes, you must pass replace_with() a valid MARC::Field object. no cap. 🫶

---

✨ Updating subfields and indicators ✨

if you'd rather not replace an entire field, you can also edit the contents of the field itself with the update() method. say you've got a batch of records and you wanna make sure the 2nd indicator for the 245 field is properly set for titles that begin with 'The' (where the indicator should be '4'). 💅

1   ## Example U5
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','file.dat');
5   open(OUT,'>new.dat') or die $!;
6
7   while (my $record = $batch->next()) {
8
9     ## retrieve the 245 record.
10     my $field_245 = $record->field('245');
11
12     ## if we got 245 and it starts with 'The'...
13     if ($field_245 and $field_245->as_string() =~ /^The /) {
14
15       ## if the 2nd indicator isn't 4, update
16       if ($field_245->indicator(2) != 4) {
17         $field_245->update( ind2 => 4 );
18       }
19
20     }
21
22     print OUT $record->as_usmarc();
23
24   }

in a similar fashion, you can update individual or multiple subfields:

$field_245->update( a => 'History of the World :', b => 'part 1' );

but beware, bestie: you can only update the first occurrence of a subfield using update(). if you need to do more finer grained updates, you're advised to build a new field and replace the existing field with replace_with(). periodt. 💀

---

🌈 Changing a record's leader 🌈

the above procedure works for fields, but editing the leader requires you to use the leader() method. when called with no arguments, leader() returns the current leader, and when you pass a scalar value as an argument, the leader gets set to that value. this example shows how you might update position 6 of a record's leader to reflect a computer file. ✨

1   ## Example U6
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','file.dat');
5   open(OUT,'>new.dat') or die $!;
6   my $record = $batch->next();
7
8   ## get the current leader.
9   my $leader = $record->leader();
10
11   ## replace position 6 with 'm'
12   substr($leader,6,1) = 'm';
13
14   ## update the leader
15   $record->leader($leader);
16
17   ## save the record to a file
18   print OUT $record->as_usmarc();

---

🦋 Modifying fields without indicators 🦋

MARC::Record and MARC::Field are smart and know that you don't have field indicators with tags less than 010. here's an example of updating/adding an 005 field to indicate a new transaction time. for a little pizzazz, we use Perl's localtime() to generate the data we need for this field. 🎀

1   ## Example U7
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','file.dat');
5   open(OUT,'>new.dat') or die $!;
6
7   while (my $record = $batch->next() ) {
8
9     ## see if there is a 005 field.
10     my $field_005 = $record->field('005');
11
12     ## delete it if we find one.
13     $record->delete_field($field_005) if $field_005;
14
15     ## figure out the contents of our new 005 field.
16     my ($sec,$min,$hour,$mday,$mon,$year) = localtime();
17     $year += 1900; $mon += 1; # catering to offsets.
18     my $datetime = sprintf("%4d%02d%02d%02d%02d%02d.0",
19                             $year,$mon,$mday,$hour,$min,$sec);
20
21     ## create a new 005 field using our new datetime.
22     $record->append_fields( MARC::Field->new('005',$datetime) );
23
24     ## save record to a file.
25     print OUT $record->as_usmarc();
26
27   }

---

💜 Reordering subfields 💜

you may find yourself in a situation where you wanna programmatically reorder, and possibly modify, subfields in a particular field. for example, imagine you have a batch of records with 856 fields that contain subfields z, u, and possibly 3... in any order! now imagine you'd like to standardize the subfield z, and reorder them so that subfield 3 precedes subfield z, which precedes subfield u. this is tricky, but it can be done like this: read in a record, extract the existing 856 field, build a new 856 field based on the existing one, replace the existing field with your newly created version. 💅

1   ## Example U8
2
3   use MARC::Batch;
4   my $batch = MARC::Batch->new('USMARC','856.dat');
5   open(OUT,'>856_new.dat') or die $!;
6
7   while (my $record = $batch->next()) {
8
9     my $existing = $record->field('856');
10
11     ## make sure 856 exists.
12     if ($existing) {
13
14       ## our ordered subfields.
15       my @subfields = ();
16
17       ## if we have a subfield 3, add it.
18       if (defined($existing->subfield('3'))) {
19         push(@subfields,'3',$existing->subfield('3'));
20       }
21
22       ## now add subfields z and u.
23       push(@subfields,'z','Access restricted',
24         'u',$existing->subfield('u'));
25
26       ## create a new 856.
27       my $new = MARC::Field->new(
28          856', $existing->indicator(1),
29          $existing->indicator(2), @subfields
30       );
31
32       ## replace the existing subfield.
33       $existing->replace_with($new);
34
35     }
36
37     ## write out the record
38     print OUT $record->as_usmarc();
39
40   }

---

🌸 Updating subject subfield x to subfield v 🌸

as a somewhat more complicated example, you may find yourself wanting to update the last subfield x in a 650 field to be a subfield v instead. with the MARC::Field subfields() and replace_with() methods, plus some fancy footwork, this can be done relatively easily. 🧚

1  ## Example U9
2
3  use MARC::Batch;
4
5  my $file = shift;
6
7  my $batch = MARC::Batch->new('USMARC', $file);
8  while ( my $record = $batch->next() ) {
9
10    # go through all 6XX fields in the record.
11    foreach my $subject ( $record->field( '6..' ) ) {
12
13      # extract subfields as an array of array refs.
14      my @subfields = $subject->subfields();
15
16      # setup an array to store our new field.
17      my @newSubfields = ();
18
19      # a flag to indicate that we found an subfield x.
20      my $foundX = 0;
21
22      # use pop() to read the subfields backwards.
23      while ( my $subfield = pop( @subfields ) ) {
24
25        # for convenience, pull out the subfield
26        # code and data from  the array ref.
27        my ($code,$data) = @$subfield;
28
29        # if the subfield code is 'x' and
30        # we haven't already found one...
31        if ( $code eq 'x' and ! $foundX ) {
32
33          # change to a v.
34          $code = 'v';
35
36          # set flag so we know not to
37          # translate any more subfield x.
38          $foundX = 1;
39
40        }
41
42        # add our (potentially changed) subfield
43        # data to our new subfield data array.
44        unshift( @newSubfields, $code, $data );
45
46      }
47
48      # if we did find a subfield x, then create a new field using our
49      # new subfield data, and replace the old one with the new one.
50      if ( $foundX ) {
51        my $newSubject = MARC::Field->new(
52          $subject->tag(),
53          $subject->indicator(1),
54          $subject->indicator(2),
55          @newSubfields
56        );
57        $subject->replace_with( $newSubject );
58      }
59
60    }
61
62    # output the potentially changed record as MARC.
63    print $record->as_usmarc();
64
65  }

✨ VALIDATING ✨

Ok bestie, so you've got records. Now how do you know they're not a mess? 💅 Enter MARC::Lint, available on CPAN and in cvs on SourceForge — it's got some extra goodies that let you validate records. MARC::Lint comes with an extensive battery of tests, and it also hands you a whole framework for adding your own. So cute of it.

---

💖 Using MARC::Lint 💖

Here's an example of using MARC::Lint to generate a list of errors present in a batch of records in a file named 'file.dat':

1   ## Example V1
2
3   use MARC::Batch;
4   use MARC::Lint;
5
6   my $batch = MARC::Batch->new('USMARC','file.dat');
7   my $linter = MARC::Lint->new();
8   my $counter = 0;
9
10   while (my $record = $batch->next() ) {
11
12     $counter++;
13
14     ## feed the record to our linter object.
15     $linter->check_record($record);
16
17     ## get the warnings...
18     my @warnings = $linter->warnings();
19
20     ## output any warnings.
21     if (@warnings) {
22
23       print "RECORD $counter\n";
24       print join("\n",@warnings),"\n";
25
26     }
27
28   }

MARC::Lint is quite thorough, and will check the following when validating: presence of a 245 field, repeatability of fields and subfields, valid use of subfield within particular fields, presence of indicators and their values. Every single check is based on MARC21 bibliographic format. No cap. 🫶

---

🌸 Customizing MARC::Lint 🌸

Real talk: MARC::Lint makes no claim to check everything that might be wrong with a MARC record. In practice, individual libraries may have their own idea about what is valid or invalid. Like, a library might mandate that all MARC records with an 856 field should have a subfield z that reads "Connect to this resource". Very specific. Very them. 💁

Good news though — MARC::Lint does provide a framework for adding rules. You do it using the object oriented programming technique of inheritance. In short: you can create your own subclass of MARC::Lint, and then use it to validate your records. Here's an example:

1   ## Example V2
2
3   ## first, create our own subclass of MARC::Lint.
4   ## should be saved in a file called MyLint.pm.
5
6   package MyLint;
7   use base qw(MARC::Lint);
8
9   ## add a method to check that the 856
10   ## fields contain a correct subfield z.
11   sub check_856 {
12
13     ## your method is passed the MARC::Lint
14     ## and MARC::Field objects for the record.
15     my ($self,$field) = @_;
16
17     if ($field->subfield('z') ne 'Connect to this resource') {
18
19       ## add a warning to our lint object.
20       $self->warn("856 subfield z must read 'Connect to this resource'.");
21
22     }
23
24   }

Then create a separate program that uses your subclass to validate your MARC records. You'll need to make sure your program is able to find your module (in this case, MyLint.pm)... this can be achieved by putting both MyLint.pm and the following program in the same directory:

1   ## Example V3
2
3   use MARC::Batch;
4   use MyLint;
5
6   my $linter = MyLint->new();
7   my $batch = MARC::Batch->new('USMARC','file.marc');
8   my $counter = 0;
9
10   while (my $record = $batch->next()) {
11
12     $counter++;
13
14     ## check the record
15     $linter->check_record($record);
16
17     ## get the warnings, and print them out
18     my @warnings = $linter->warnings();
19     if (@warnings) {
20       print "RECORD $counter\n";
21       print join("\n",@warnings),"\n";
22     }
23
24   }

Now notice 👀 how the call to "check_record()" at line 15 automatically calls the "check_record" in MARC::Lint. The property of inheritance is literally what makes this happen. $linter is an instance of the MyLint class, and MyLint inherits from the MARC::Lint class, which allows $linter to inherit all the functionality of a normal MARC::Lint object plus the new functionality found in the "check_856" method. Iconic behaviour tbh. ✨

Notice also that we don't have to call "check_856()" directly. The call to "check_record()" automatically looks for any "check_XXX" methods that it can call to verify the record. Pretty neat stuff. And bestie — if you've added validation checks that you think could be of use to the general public, please share them on the perl4lib mailing list, or become a developer and add them to the source! 🎀

---

🦋 SWOLLEN APPENDICES 🦋

Brian Eno fans might catch this reference to his autobiography, which was comprised of a years worth of diary entries plus extra topics at the end, and was entitled "A Year With Swollen Appendices". The following section is a grab bag group of appendices. Many of them are not filled in yet; this is because they are just ideas... so perhaps the appendices aren't that swollen yet. Feel free to suggest new ones, or to fill these in. 🫶

Comparing Collections

Authority Records

URLs

ISBN/ISSNs

Call numbers

Subject headings

Suppose you have a batch of MARC records and you want to extract all the subject headings, generating a report of how many times each subject heading appeared in the batch:

1   use MARC::File::USMARC;
2   use constant MAX => 20;
3
4   my %counts;
5
6   my $filename = shift or die "Must specify filename\n";
7   my $file = MARC::File::USMARC->in( $filename );
8
9   while ( my $marc = $file->next() ) {
10       for my $field ( $marc->field("6..") ) {
11           my $heading = $field->subfield('a');
12
13           # trailing whitespace / punctuation.
14           $heading =~ s/[.,]?\s*$//;
15
16           # Now count it.
17           ++$counts{$heading};
18       }
19   }
20   $file->close();
21
22   # Sort the list of headings based on the count of each.
23   my @headings = reverse sort { $counts{$a} <=> $counts{$b} } keys %counts;
24
25   # Take the top N hits...
26   @headings = @headings[0..MAX-1];
27
28   # And print out the results.
29   for my $heading ( @headings ) {
30       printf( "%5d %s\n", $counts{$heading}, $heading );
31   }

Which will generate results like this:

 600 United States
 140 World War, 1939-1945
  78 Great Britain
  63 Afro-Americans
  61 Indians of North America
  58 American poetry
  55 France
  53 West (U.S.)
  53 Science fiction
  53 American literature
  50 Shakespeare, William
  48 Soviet Union
  46 Mystery and detective stories
  45 Presidents
  43 China
  40 Frontier and pioneer life
  38 English poetry
  37 Authors, American
  37 English language
  35 Japan

HTML

XML

MARCMaker

MARC::File::MARCMaker, available on CPAN and in cvs on SourceForge, is a subclass of MARC::File for working with MARC 21 data encoded in the format used by the Library of Congress MARCMaker and MARCBreaker programs (http://www.loc.gov/marc/makrbrkr.html) and MarcEdit (). 💾

An example of a brief record in this format:

 =LDR  00314nam  22001215a 4500
 =001  ctr00000123\
 =003  XX-XxUND
 =005  20000613133448.0
 =008  051029s2005\\\\xxua\\\\\\\\\\001\0\eng\\
 =040  \\$aXX-XxUND$cXX-XxUND
 =245  00$aSample of MARCMaker record.
 =260  \\$a[United States] :$b[S.n.],$c2005.
 =300  \\$a1 p. ;$c28 cm.

The following example converts an ISO2709 format record into MARCMaker format.

1    ## Example Maker1
2
3    use MARC::Batch;
4    use MARC::File::MARCMaker;
5
6    #mrc indicates ISO2709 format
7    my $mrc_in = 'in.mrc';
8    #mrk indicates MARCMaker format
9    my $mrk_out = 'out.mrk';
10
11   #initialize $batch_mrc as new MARC::Batch object
12   my $batch_mrc = MARC::Batch->new('USMARC', $mrc_in);
13
14   #open mrk (MARCMaker) format output file
15   open (OUTMRK, ">$mrk_out") || die "Cannot open $mrk_out, $!";
16
17   my $rec_count = 0;
18   while (my $record = $batch_mrc->next()) {
19      $rec_count++;
20
21      print OUTMRK MARC::File::MARCMaker->encode($record);
22
23   } # while
24
25   print "$rec_count records processed\n";

The following example shows conversion from MARCMaker format to ISO2709 format.

1    ## Example Maker2
2
3    use MARC::Batch;
4    use MARC::File::MARCMaker;
5
6    #mrk indicates MARCMaker format
7    my $mrk_in = 'in.mrk';
8    #mrc indicates ISO2709 format
9    my $mrc_out = 'out.mrc';
10
11   #initialize $batch_mrk as new MARC::Batch object
12   my $batch_mrk = MARC::Batch->new( 'MARCMaker', $mrk_in);
13
14   #open mrc (ISO2709) format output file
15   open (OUTMRC, ">$mrc_out") || die "Cannot open $mrc_out, $!";
16
17   my $rec_count = 0;
18   while (my $record = $batch_mrk->next()) {
19      $rec_count++;
20
21      print OUTMRC $record->as_usmarc();
22
23   } # while
24
25   print "$rec_count records processed\n";

Excel

Z39.50

Chris Biemesderfer was kind enough to contribute a short example of how to use MARC::Record in tandem with Net::Z3950. Net::Z3950 is a CPAN module which provides an easy to use interface to the Z39.50 protocol so that you can write programs that retrieve records from bibliographic database around the world. 🌍

Chris' program is a command line utility which you run like so:

./zm.pl 0596000278

where 0596000278 is an ISBN (for the 3rd edition of the Camel incidentally). The program will query the Library of Congress Z39.50 server for the ISBN, and dump out the retrieved MARC record on the screen. The program is designed to lookup multiple ISBNs if you separate them with a space. This is just an example showing what is possible. 💫

...and here's the rest of that ISBN-hunting script, bestie — it's giving very much "written in 2003 and still vibing":

 1   #!/usr/bin/perl -w
 2
 3   # GET-MARC-ISBN -- Get MARC records by ISBN from a Z39.50 server
 4
 5   use strict;
 6   use Carp;
 7   use Net::Z3950;
 8   use MARC::Record;
 9
10   exit if ($#ARGV < 0);
11
12   # We handle multiple ISBNs in the same query by assembling a
13   # (potentially very large) search string with Prefix Query Notation
14   # that ORs the ISBN-bearing attributes.
15   #
16   # For purposes of automation, we want to request batches of many MARC
17   # records.  I am not a Z39.50 weenie, though, and I don't know
18   # offhand if there is a limit on how big a PQN query can be...
19
20   my $zq = "\@attr 1=7 ". pop();
21   while (@ARGV) { $zq = '@or @attr 1=7 '. pop() ." $zq" }
22
23   ## HERE IS THE CODE FOR Z3950 REC RETRIEVAL
24   # Set up connection management structures, connect
25   # to the server, and submit the Z39.50 query.
26
27   my $mgr = Net::Z3950::Manager->new( databaseName => 'voyager' );
28   $mgr->option( elementSetName => "f" );
29   $mgr->option( preferredRecordSyntax => Net::Z3950::RecordSyntax::USMARC );
30
31   my $conn = $mgr->connect('z3950.loc.gov', '7090');
32   croak "Unable to connect to server" if !defined($conn);
33
34   my $rs = $conn->search($zq);
35
36   my $numrec = $rs->size();
37   print STDERR "$numrec record(s) found\n";
38
39   for (my $ii = 1; $ii <= $numrec; $ii++) {
40
41       # Extract MARC records from Z3950
42       # result set, and load MARC::Record.
43       my $zrec = $rs->record($ii);
44       my $mrec = MARC::Record->new_from_usmarc($zrec->rawdata());
45       print $mrec->as_formatted, "\n\n";
46
47   }

The way this works is a whole journey, fr fr: we build up a Prefix Query Notation search string by gluing ISBNs together with @or @attr 1=7, then we spin up a Net::Z3950::Manager for the voyager database, set elementSetName to "f" and preferredRecordSyntax to Net::Z3950::RecordSyntax::USMARC, connect to z3950.loc.gov on port 7090, and croak if the connection comes back undefined. Then we search, count the results with $rs->size(), and loop from 1 to $numrec, pulling each record, running it through MARC::Record->new_from_usmarc($zrec->rawdata()), and dumping $mrec->as_formatted. ✨

And the author of the script literally admits in a comment: they're not a Z39.50 weenie and don't know offhand if there's a limit on how big a PQN query can be. Iconic honesty tbh. 🙈

---

🍓 Databases 🍓

Here's a script that will do a Z39.50 query (using Chris Biemesderfer's zm.pl as a model), get a MARC record back, and store it as a binary blob in a MySQL table of this structure:

 +---------------+---------------+------+-----+---------+----------------+
 | Field         | Type          | Null | Key | Default | Extra          |
 +---------------+---------------+------+-----+---------+----------------+
 | TitleID       | int(7)        |      | PRI | NULL    | auto_increment |
 | RecLastMod    | timestamp(14) | YES  |     | NULL    |                |
 | ISSN          | text          | YES  |     | NULL    |                |
 | RawMARCRecord | blob          | YES  |     | NULL    |                |
 +---------------+---------------+------+-----+---------+----------------+

Look at that schema, she's structured, she's organized, she knows what she wants 💅

 1 #!/usr/bin/perl -w
 2
 3 # Script that reads in a file of ISSNs, queries a Z39.50 server,
 4 # and stores resulting records in a database. Limitations: Only
 5 # stores 1 records per ISSN.
 6 # Last updated 2004-09-08 Mark Jordan, mjordan@sfu.ca
 7
 8 use strict;
 9 use Carp;
10 use Net::Z3950;
11 use MARC::Record;
12 use DBI;
13
14 # DB connection settings
15 my $host = "somehost";
16 my $user = "someuser";
17 my $password = "somepass";
18 my $database = "somedb";
19
20 # Input file (one ISSS/line)
21 my $InputFile = $ARGV[0];
22
23 # Prepare list of ISSNs to search
24 my @ISSNs;
25 open (INPUT, "< $InputFile") or die "Can't find input file\n";
26 while (<INPUT>) { chomp $_; push (@ISSNs, $_); }
27 close INPUT;
28
29
30 # Set up connection management structures, connect to the server,
31 # and submit the Z39.50 query.
32 my $mgr = Net::Z3950::Manager->new( databaseName => 'voyager' );
33 $mgr->option( elementSetName => "f" );
34 $mgr->option( preferredRecordSyntax => Net::Z3950::RecordSyntax::USMARC );
35 my $conn = $mgr->connect('z3950.loc.gov', '7090');
36 croak "Unable to connect to server" if !defined($conn);
37
38
39 my $handle = DBI->connect("DBI:mysql:$database:$host","$user","$password")
40         or die $DBI::errstr;
41
42 foreach my $ISSN (@ISSNs) {
43         my $zq = "\@attr 1=8 ". $ISSN;
44         my $rs = $conn->search($zq);
45         my $numrec = $rs->size();
46         if ($numrec == 0) {
47             print "Record for ISSN $ISSN not found, moving to next ISSN...\n";
48             next;
49         } else {
50            # Extract MARC record from the result set, and invoke MARC::Record
51            my $zrec = $rs->record(1);
52            my $mrec = MARC::Record->new_from_usmarc($zrec->rawdata());
53            my $rawdata = $zrec->rawdata();
54            $rawdata = $handle->quote ($rawdata);
55            # Add to db
56            my $SQL = "insert into Titles values (NULL,NULL,'$ISSN',$rawdata)";
57            my $cursor = $handle->prepare($SQL);
58            $cursor->execute;
59            print "Record for ISSN $ISSN added to database...\n";
60            $cursor->finish;
61         }
62 }
63 $handle->disconnect;
64
65 __END__

Note the limitation up top: it only stores 1 records per ISSN. And Element: f, database voyager, z3950.loc.gov port 7090 again — same server bestie, different quest 🌸

If you want to pull records out of the same database and do something with them, here's a template script:

 1 #!/usr/bin/perl -w
 2
 3 # Script that gets MARC records (in blobs) from a database.
 4 # Last updated 2004-09-08 Mark Jordan, mjordan@sfu.ca
 5
 6 use strict;
 7 use MARC::Record;
 8 use DBI;
 9
10 # DB connection settings
11 my $mysql_host = "somehost";
12 my $mysql_user = "someuser";
13 my $mysql_password = "somepass*";
14 my $mysql_database = "somedb";
15
16
17 my $handle = DBI->connect("DBI:mysql:$mysql_database:$mysql_host",
18         "$mysql_user","$mysql_password") or die $DBI::errstr;
19
20 my $SQL = "select * from Titles";
21 my $cursor = $handle->prepare($SQL);
22 $cursor->execute;
23
24 while (my @Records = $cursor->fetchrow_array) {
25         my $RawMARC = $Records[3];
26         my $mrec = MARC::Record->new_from_usmarc($RawMARC);
27         # Print out the title
28         print $mrec->title , "\n";
29 }
30
31 $cursor->finish;
32 $handle->disconnect;
33
34 __END__

See how it just grabs $Records[3] (the RawMARCRecord blob, queen 👑), feeds it to MARC::Record->new_from_usmarc($RawMARC), and prints $mrec->title. Chef's kiss. 🎀

🌈 Procite/Endnote 🌈

...okay bestie, this heading is literally just a heading. Nothing underneath it. It said "Procite/Endnote" and then dipped. Respect the mysterious energy. 🦋

---

👑 CONTRIBUTORS 👑

Many thanks to all the contributors who have made this document possible. Like genuinely, this whole thing exists because of them and we're all just living in their legacy.

• Bryan Baldus <eijabb@cpan.org>
• Chris Biemesderfer <chris@seagoat.com>
• Morbus Iff <morbus@disobey.com>
• Mark Jordan <mjordan@sfu.ca>
• Andy Lester <andy@petdance.com>
• Christopher Morgan <morgan@acm.org>
• Shashi Pinheiro <SPinheiro@utsa.edu>
• Jackie Shieh <jshieh@umich.edu>
• Ed Summers <ehs@pobox.com>

Stand up and take a bow, every single one of you 💖✨🎀🦋🌸👑

---

And that's the whole entire manual page, bestie. No notes. We ate. 💅

This tutorial shipped with perl v5.36.0, dated 2022-10-13, and yes she's part of the MARC::...utorial(3pm) family — Perl's own user-contributed documentation, handcrafted with love. Go build something, go read some MARC, go be the main character. Love u, bye 🫶💜