Catman...er::OAI(3pm)

📦
version 0.21 · 4 versions
, in ubuntu resolute, ubuntu stonking

💖 NAME 💖

Catmandu::Importer::OAI — it's the package that imports OAI-PMH feeds, and yes bestie this is section 3pm of the User Contributed Perl Documentation, we are in the Perl streets now 🫶

✨ SYNOPSIS ✨

okay so from the command line, here's the whole vibe:

    # From the command line

    # Harvest records
    $ catmandu convert OAI --url http://myrepo.org/oai
    $ catmandu convert OAI --url http://myrepo.org/oai --metadataPrefix didl --handler raw

    # Harvest repository description
    $ catmandu convert OAI --url http://myrepo.org/oai --identify 1

    # Harvest identifiers
    $ catmandu convert OAI --url http://myrepo.org/oai --listIdentifiers 1

    # Harvest sets
    $ catmandu convert OAI --url http://myrepo.org/oai --listSets 1

    # Harvest metadataFormats
    $ catmandu convert OAI --url http://myrepo.org/oai --listMetadataFormats 1

    # Harvest one record
    $ catmandu convert OAI --url http://myrepo.org/oai --getRecord 1 --identifier oai:myrepo:1234

🌸 DESCRIPTION 🌸

Catmandu::Importer::OAI is a Catmandu importer whose whole personality is harvesting metadata records from an OAI-PMH endpoint. That's it, that's the job, and it does it flawlessly 💅

🎀 CONFIGURATION 🎀

  • url — the OAI-PMH Base URL. literally the address of the place we're robbing (politely) 🦋
  • metadataPrefix — the metadata prefix that tells everyone which metadata format we want. Set to oai_dc by default.
  • handler( sub {} | $object | 'NAME' | '+NAME' ) — the handler that transforms each record from an XML DOM (XML::LibXML::Element) into a Perl hash. the glow-up pipeline ✨

Handlers can be provided as a function reference, an instance of a Perl package that implements 'parse', or by a package NAME. Package names should be prepended by "+" or prefixed with "Catmandu::Importer::OAI::Parser". E.g "foobar" will create a "Catmandu::Importer::OAI::Parser::foobar" instance. (iykyk 😌)

By default the handler Catmandu::Importer::OAI::Parser::oai_dc is used for metadataPrefix "oai_dc", Catmandu::Importer::OAI::Parser::marcxml for "marcxml", Catmandu::Importer::OAI::Parser::mods for "mods", and Catmandu::Importer::OAI::Parser::struct for other formats. In addition there is Catmandu::Importer::OAI::Parser::raw to return the XML as it is. no transformation, just vibes, raw dogging the XML fr fr 💀

  • identifier — option to return only results for this particular identifier. one (1) record and only one 👑
  • set — an optional set for selective harvesting. choose your fighter 🥊
  • from — an optional datetime value (YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ) as lower bound for datestamp-based selective harvesting.
  • until — an optional datetime value (YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ) as upper bound for datestamp-based selective harvesting. from and until are giving sibling energy fr 🌈
  • identify — harvest the repository description instead of all records.
  • getRecord — harvest one record instead of all records.
  • listIdentifiers — harvest identifiers instead of full records.
  • listRecords — harvest full records. Default operation.
  • listSets — harvest sets instead of records.
  • listMetadataFormats — harvest metadata formats of records. no period at the end in the original and we respect that 🎀
  • resumptionToken — an optional resumptionToken to start harvesting from. picking up where you left off, main character energy 💅
  • dry — don't do any HTTP requests but return URLs that data would be queried from. window shopping only, no buying 🛍️
  • strict — optional validate all parameters first against the OAI 2 specifications before sending it to an OAI server. Default: undef.
  • xslt — preprocess XML records with XSLT script(s) given as comma separated list or array reference. Requires Catmandu::XML.
  • max_retries — when an oai request fails, the importer will retry this number of times. Set to '0' by default.

Internally the exponential backoff algorithm is used for this. This means that after every failed request the importer will choose a random number between 0 and 2^collision (excluded), and wait that number of seconds. So the actual amount of time before the importer stops can differ:

 first retry:
    wait [ 0..2^1 [ seconds
 second retry:
    wait [ 0..2^2 [ seconds
 third retry:
    wait [ 0..2^3 [ seconds

 ..

(yes it really does just keep going like that, the patience is unmatched 😭)

  • sleep — sleep a number of seconds between OAI-PMH calls to the endpoint (default 0). be polite to the server bestie, it's not that hard 💖
  • realm — an optional realm value. This value is used when the importer harvests from a repository which is secured with basic authentication through Integrated Windows Authentication (NTLM or Kerberos).
  • username — an optional username value. This value is used when the importer harvests from a repository which is secured with basic authentication.
  • password — an optional password value. This value is used when the importer harvests from a repository which is secured with basic authentication. 🔐 keep it secret, keep it safe

🦋 METHOD 🦋

Every Catmandu::Importer is a Catmandu::Iterable and all its methods are inherited.

⚠️ but listen — the Catmandu::Importer::OAI methods are not idempotent: OAI-PMH feeds can only be read once. one shot. no rewind. no take-backs. not me finding this out the hard way 😭

In addition to methods inherited from Catmandu::Iterable, this module provides the following public methods:

⭐ handle_record( $dom ) ⭐

Process an XML DOM as with xslt and handler as configured and return the result. that's it, she just does the thing 💅

🌈 ENVIRONMENT 🌈

If you are connected to the internet via a proxy server you need to set the coordinates to this proxy in your environment:

    export http_proxy="http://localhost:8080"

If you are connecting to a HTTPS server and don't want to verify the validity of certificates of the peer you can set the PERL_LWP_SSL_VERIFY_HOSTNAME to false in your environment. This maybe required to connect to broken SSL servers:

    export PERL_LWP_SSL_VERIFY_HOSTNAME=0
🙈 yes it says "broken SSL servers" in the original, and yes that is the exact energy we are preserving, no cap

👑 SEE ALSO 👑

Catmandu , Catmandu::Importer

💜 AUTHOR 💜

Nicolas Steenlant, "<nicolas.steenlant at ugent.be>"

🍓 CONTRIBUTOR 🍓

  • Patrick Hochstenbach, "<patrick.hochstenbach at ugent.be>"
  • Jakob Voss, "<nichtich at cpan.org>"
  • Nicolas Franck, "<nicolas.franck at ugent.be>"

Copyright 2016 Ghent University Library

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.

okay that's the whole page bestie, and this lil sign-off is brought to you by perl v5.40.0, dated 2024-09-08, on the Catmandu::Importer::OAI(3pm) manual page. slay, go harvest something iconic 💖✨🎀

📦 where this page comes from