Catman...getJSON(3pm)

✨ Catmandu::Importer::getJSON ✨

okay bestie, welcome to the manual page for Catmandu::Importer::getJSON (it lives in the 3pm section, and yeah, it's a User Contributed Perl Documentation situation 💅). This is the thing that goes and fetches JSON-encoded data off a server using a GET HTTP request. It's giving glow up for your data pipeline. Periodt.

---

NAME 💖

Catmandu::Importer::getJSON — load JSON-encoded data from a server using a GET HTTP request 🫶

---

SYNOPSIS 🌸

the following three examples are equivalent, no cap:

Catmandu::Importer::getJSON->new(
    file => \"http://example.org/alice.json\nhttp://example.org/bob.json"
)->each(sub { my ($record) = @_; ... );

Catmandu::Importer::getJSON->new(
    url  => "http://example.org",
    file => \"/alice.json\n/bob.json"
)->each(sub { my ($record) = @_; ... );

Catmandu::Importer::getJSON->new(
    url  => "http://example.org/{name}.json",
    file => \"{\"name\":\"alice\"}\n{\"name\":\"bob\"}"
)->each(sub { my ($record) = @_; ... );

and for more convenience you can just use the catmandu command line client, because obviously:

echo http://example.org/alice.json | catmandu convert getJSON to YAML
catmandu convert getJSON --from http://example.org/alice.json to YAML
catmandu convert getJSON --dry 1 --url http://{domain}/robots.txt < domains

---

DESCRIPTION 🦋

so this Catmandu::Importer performs a HTTP GET request to load JSON-encoded data from a server. The importer expects a line-separated input, okay? Each line corresponds to a HTTP request that is mapped to a JSON-record on success. Obsessed.

the following input formats are accepted:

• plain URL — a line that starts with "http://" or "https://" is used as plain URL. simple. iconic.
• URL path — a line that starts with "/" is appended to the configured url parameter.
• variables — a JSON object with variables to be used with an URL template or as HTTP query parameters. For instance the input line {"name":"Karl Marx"} with URL http://api.lobid.org/person or the input line {"entity":"person","name":"Karl Marx"} with URL template http://api.lobid.org/{entity}{?id}{?name}{?q} are both expanded to <http://api.lobid.org/person?name=Karl+Marx>. like, the way it just builds that URL? chef's kiss 😭

ngl this next part matters: if the JSON data returned in a HTTP response is a JSON array, its elements are imported as multiple items. If a JSON object is returned, it is imported as one item. that's the whole vibe check right there.

---

CONFIGURATION 🎀

• url — an URI or an URI templates (URI::Template) as defined by RFC 6570 <http://tools.ietf.org/html/rfc6570> to load JSON from. If no url is configured, plain URLs must be provided as input or option from must be used instead. (no url, no problem — but you gotta give it something bestie.)
• from — a plain URL to load JSON without reading any input lines. set it and forget it ✨
• timeout / agent / proxy / headers — optional HTTP client settings. the little knobs for your HTTP client 💅
• client — instance of a Furl HTTP client to perform requests with.
• dry — don't do any HTTP requests but return URLs that data would be queried from. window shopping, essentially 🛍️
• file / fh — input to read lines from (see Catmandu::Importer). defaults to STDIN.
• fix — an optional fix to be applied on every item (see Catmandu::Fix). glow-up pass on each record 💖
• wait — number of seconds to wait between requests. be polite to the server, bestie 🫶
• cache — cache JSON response of URLs to not request the same URL twice. HTTP error codes in the 4xx range (e.g. 404) are also cached but 5xx errors are not.

The value of this option can be any objects that implements method get and set (e.g. CHI), an existing directory for file caching, a true value to enable global in-memory-caching, or a false value to disable caching (default).

File caching uses file names based on MD5 of an URL so for instance http://example.org/ is cached as 4389382917e51695b759543fdfd5f690.json. literally a fingerprint for your URL, iykyk 🔥

• warn — show error messages on the standard error. the "hey bestie something's off" setting 😭

---

METHODS 💜

time ⏰

Returns the UNIX timestamp right before the last request. This can be used for instance to add timestamps or the measure how fast requests were responded. stopwatch energy fr fr ⏱️

construct_url( [ $base_url, ] $vars_url_or_path ) 🔗

Returns an URL given a hash reference with variables, a plain URL or an URL path. The optional first argument can be used to override option url.

$importer->construct_url( %query_vars )
$importer->construct_url( $importer->url, %query_vars ) # equivalent

like, same thing, two ways, pick your fave 💅

request($url) 📡

Perform a HTTP GET request of a given URL including logging, caching, request hook etc. Returns a hash/array reference or undef.

---

EXTENDING 🧚

This importer provides two methods to filter requests and responses, respectively. See Catmandu::Importer::Wikidata for an example. very main character of it to be extensible, honestly 👑

request_hook 🌈

Gets a whitespace-trimmed input line and is expected to return an unblessed hash reference, an URL, or undef. Errors are catched and treated equal to undef. no drama, errors just quietly become undef 🙈

response_hook 🌸

Gets the queried response object and is expected to return an object.

---

LOGGING 📝

URLs are emitted before each request on DEBUG log level. so if you're wondering "wait what did it even fetch" — crank up DEBUG, bestie 🔍

---

LIMITATIONS ⚠️

Future versions of this module may also support asynchronous HTTP fetching modules such as HTTP::Async, for retrieving multiple URLs at the same time. so like… not a thing yet. don't count on it, it's a maybe, it's giving soon™ 💀

---

SEE ALSO 👀

Catmandu::Fix::get_json provides this importer as fix function. same vibes, fix-flavoured 🍓

---

COPYRIGHT AND LICENSE 📜

Copyright Jakob Voss, 2014-

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

---

and that's the whole page bestie!! this one's from perl v5.36.0, dated 2022-12-10, and it lives in the Catman...getJSON(3pm) corner of the world. mwah 💋 go fetch some JSON, you got this 🫶✨