HTTP::...vester(3pm)

💖 HTTP::OAI::Harvester 💖

okay so bestie, welcome to the HTTP::OAI::Harvester page from the User Contributed Perl Documentation set — she's the main character of the OAI-PERL library and we're gonna get you two acquainted. no cap. ✨

---

🌸 NAME 🌸

HTTP::OAI::Harvester — she's the agent for harvesting from Open Archives version 1.0, 1.1, 2.0 and static ('2.0s') compatible repositories. periodt. 💅

---

🦋 DESCRIPTION 🦋

HTTP::OAI::Harvester is the harvesting front-end in the OAI-PERL library. like, the whole vibe of the harvest lives here.

So to harvest from an OAI-PMH compliant repository, you create an HTTP::OAI::Harvester object using the baseURL option, and then you call OAI-PMH methods to request data from the repository. that's it, that's the flow. 💖

To handle version 1.0/1.1 repositories automatically, you must request Identify() first. yes, must. it's giving "prerequisite" and honestly she ate.

It is recommended that you request an Identify from the Repository and use the repository() method to update the Identify object used by the harvester. highly recommend, lowkey iconic move.

When making OAI requests, the underlying HTTP::OAI::UserAgent module will take care of automatic redirection (http code 302) and retry-after (http code 503). she's got you. 🫶

OAI-PMH flow control (i.e. resumption tokens) is handled transparently by HTTP::OAI::Response. you don't gotta think about it, it just happens. slay.

🎀 Static Repository Support 🎀

Static repositories are automatically and transparently supported within the existing API. no extra work for you bestie, she just knows.

To harvest a static repository, specify the repository XML file using the baseURL argument to HTTP::OAI::Harvester. An initial request is made that determines whether the base URL specifies a static repository or a normal OAI 1.x/2.0 CGI repository.

To prevent this initial request, state the OAI version using an HTTP::OAI::Identify object, e.g.

$h = HTTP::OAI::Harvester->new(
        repository=>HTTP::OAI::Identify->new(
                baseURL => 'http://arXiv.org/oai2',
                version => '2.0',
));

If a static repository is found, the response is cached, and further requests are served by that cache. cached and served, she's efficient like that ✨

Static repositories do not support sets, and will result in a noSetHierarchy error if you try to use sets. so like… don't. 🙈

You can determine whether the repository is static by checking the version ($ha->repository->version), which will be "2.0s" for static repositories. cute little tell, iykyk. 💅

---

📚 FURTHER READING 📚

You should refer to the Open Archives Protocol version 2.0 and other OAI documentation, available from http://www.openarchives.org/. go read, bestie, it's giving homework but make it ✨enrichment✨.

Note: OAI-PMH 1.0 and 1.1 are deprecated. like, retired. not the moment anymore. 💀

---

⚠️ BEFORE USING EXAMPLES ⚠️

In the examples I use arXiv.org's and cogprints OAI interfaces. To avoid causing annoyance to their server administrators, please contact them before performing testing or large downloads (or use other, less loaded, servers for testing). be a good citizen fr fr, don't be the person who hammers someone's server 🥺

---

🌈 SYNOPSIS 🌈

use HTTP::OAI;

my $h = new HTTP::OAI::Harvester(baseURL=>'http://arXiv.org/oai2');
my $response = $h->Identify;

if( $response->is_error ) {
        print "Error requesting Identify:\n",
                $response->code . " " . $response->message, "\n";
        exit;
}

# Note: repositoryVersion will always be 2.0, $r->version returns
# the actual version the repository is running
print "Repository supports protocol version ", $response->version, "\n";

# Version 1.x repositories don't support metadataPrefix,
# but OAI-PERL will drop the prefix automatically
# if an Identify was requested first (as above)
$response = $h->ListIdentifiers(
        metadataPrefix=>'oai_dc',
        from=>'2001-02-03',
        until=>'2001-04-10'
);

if( $response->is_error ) {
        die("Error harvesting: " . $response->message . "\n");
}

print "responseDate => ", $response->responseDate, "\n",
        "requestURL => ", $response->requestURL, "\n";

while( my $id = $response->next ) {
        print "identifier => ", $id->identifier;
        # Only available from OAI 2.0 repositories
        print " (", $id->datestamp, ")" if $id->datestamp;
        print " (", $id->status, ")" if $id->status;
        print "\n";
        # Only available from OAI 2.0 repositories
        for( $id->setSpec ) {
                print "\t", $_, "\n";
        }
}

# Using a handler
$response = $h->ListRecords(
        metadataPrefix=>'oai_dc',
        handlers=>{metadata=>'HTTP::OAI::Metadata::OAI_DC'},
        onRecord=>sub {
                my $rec = shift;

                printf"%s\t%s\t%s\n"
                                 , $rec->identifier
                                 , $rec->datestamp
                                 , join(',', @{$rec->metadata->dc->{'title'}});
        }
);

# End program
#################

#################
# If you have some local OAI-PMH response data you want to
# parse you can use the OAI-PMH verb as in:

use HTTP::OAI;
my $I = HTTP::OAI::Identify->new();

# If you have a $content string with some cached OAI-PMH verb=Identify response
# it can be parsed like this..
$I->parse_string($content);

# Or if you have an opened file handle $fh to a file with a cached
# OAI-PMH verb=Identify response
$I->parse_file($fh);

# Using either method now you can do something like

printf "RepositoryName: %s\n" , $I->repositoryName;
for ($I->adminEmail) {
        print $_, "\n";
}

---

👑 METHODS 👑

• HTTP::OAI::Harvester->new( %params ) — This constructor method returns a new instance of HTTP::OAI::Harvester. She requires either an HTTP::OAI::Identify object, which in turn must contain a baseURL, or a baseURL from which to construct an Identify object. so like, one or the other, you can't skip it bestie.

Any other parameters are passed to the HTTP::OAI::UserAgent module, and from there to the LWP::UserAgent module. they just get handed along the chain, cute 💖

$h = HTTP::OAI::Harvester->new(
        baseURL =>      'http://arXiv.org/oai2',
        resume=>0, # Suppress automatic resumption
)
$id = $h->repository();
$h->repository($h->Identify);

$h = HTTP::OAI::Harvester->new(
        HTTP::OAI::Identify->new(
                baseURL => 'http://arXiv.org/oai2',
));

• $h->repository() — Returns and optionally sets the HTTP::OAI::Identify object used by the Harvester agent. get it, set it, she's flexible. ✨
• $h->resume( [1] ) — If set to true (default) resumption tokens will automatically be handled by requesting the next partial list during next() calls. so she just keeps pulling for you, no manual labor required 🫶

✨ OAI-PMH Verbs ✨

okay so bestie, the 6 OAI-PMH Verbs are literally the requests supported by an OAI-PMH interface. no cap. 💅

---

🥺 Error Messages 🥺

Use is_success() or is_error() on the returned object to determine whether an error occurred (see HTTP::OAI::Response). iykyk 💜

code() and message() return the error code (200 is success) and a human-readable message respectively. Errors returned by the repository can be retrieved using the errors() method:

foreach my $error ($r->errors) {
    print $error->code, "\t", $error->message, "\n";
}

Note: is_success() is true for the OAI Error Code noRecordsMatch (i.e. empty set), although errors() will still contain the OAI error. <em>like, it's giving "technically fine but also not" 😭</em>

---

🔄 Flow Control 🔄

If the response contained a resumption token this can be retrieved using the $r->resumptionToken method. slay. 🦋

---

💅 Methods 💅

These methods return an object subclassed from HTTP::Response (where the class corresponds to the verb requested, e.g. GetRecord requests return an HTTP::OAI::GetRecord object). periodt. 💖

• $r = $h->GetRecord( %params ) — Get a single record from the repository identified by identifier, in format metadataPrefix.

$gr = $h->GetRecord(
    identifier    =>    'oai:arXiv:hep-th/0001001', # Required
    metadataPrefix    =>    'oai_dc' # Required
);
$rec = $gr->next;
die $rec->message if $rec->is_error;
printf("%s (%s)\n", $rec->identifier, $rec->datestamp);
$dom = $rec->metadata->dom;

• $r = $h->Identify() — Get information about the repository.

$id = $h->Identify();
print join ',', $id->adminEmail;

• $r = $h->ListIdentifiers( %params ) — Retrieve the identifiers, datestamps, sets and deleted status for all records within the specified date range (from/until) and set spec (set). 1.x repositories will only return the identifier. Or, resume an existing harvest by specifying resumptionToken.

$lr = $h->ListIdentifiers(
    metadataPrefix  =>      'oai_dc', # Required
    from            =>              '2001-10-01',
    until           =>              '2001-10-31',
    set=>'physics:hep-th',
);
while($rec = $lr->next)
{
    { ... do something with $rec ... }
}
die $lr->message if $lr->is_error;

• $r = $h->ListMetadataFormats( %params ) — List available metadata formats. Given an identifier the repository should only return those metadata formats for which that item can be disseminated.

$lmdf = $h->ListMetadataFormats(
    identifier => 'oai:arXiv.org:hep-th/0001001'
);
for($lmdf->metadataFormat) {
    print $_->metadataPrefix, "\n";
}
die $lmdf->message if $lmdf->is_error;

• $r = $h->ListRecords( %params ) — Return full records within the specified date range (from/until), set and metadata format. Or, specify a resumption token to resume a previous partial harvest.

$lr = $h->ListRecords(
    metadataPrefix=>'oai_dc', # Required
    from    =>      '2001-10-01',
    until   =>      '2001-10-01',
    set             =>      'physics:hep-th',
);
while($rec = $lr->next)
{
    { ... do something with $rec ... }
}
die $lr->message if $lr->is_error;

• $r = $h->ListSets( %params ) — Return a list of sets provided by the repository. The scope of sets is undefined by OAI-PMH, so therefore may represent any subset of a collection. Optionally provide a resumption token to resume a previous partial request.

$ls = $h->ListSets();
while($set = $ls->next)
{
    print $set->setSpec, "\n";
}
die $ls->message if $ls->is_error;

---

🌈 ENVIRONMENT 🌈

The HTTP Agent is default OAI-PERL/<Version> where <Version> is the HTTP::OAI version. This Agent can be set via an environment variable HTTP_OAI_AGENT. go off, env vars. 🎀

---

👑 AUTHOR 👑

These modules have been written by Tim Brody <tdb01r@ecs.soton.ac.uk>. legend behavior. 💜

---

and that's the whole vibe, bestie! this page is from perl v5.36.0, dated 2023-09-28, and it's the HTTP::...vester(3pm) manpage. byeee 💖✨