Catman...Search(3pm)

📦
version 1.0204 · 5 versions
, in ubuntu resolute, ubuntu stonking

💖 Catmandu::Store::ElasticSearch — a searchable store backed by Elasticsearch 💖

okay bestie, welcome to Catman...Search(3pm), straight out of the User Contributed Perl Documentation collection. this is the page where you learn that your Catmandu data can live inside Elasticsearch and be searched, queried, CQL'd, sorted and generally served cunt. grab a strawberry 🍓 and let's go.

✨ NAME ✨

Catmandu::Store::ElasticSearch — a searchable store backed by Elasticsearch. that's it, that's the whole vibe. 💅

🌈 SYNOPSIS 🌈

from the command line, because we're lowkey efficient

# Import data into ElasticSearch
$ catmandu import JSON to ElasticSearch --bag catmandu < data.json

# Export data from ElasticSearch
$ catmandu export ElasticSearch --bag catmandu to JSON > data.json

# Export only one record
$ catmandu export ElasticSearch --bag catmandu --id 1234

# Export using an ElasticSearch query
$ catmandu export ElasticSearch --bag catmandu --query "name:Recruitment OR name:college"

# Export using a CQL query (needs a CQL mapping)
$ catmandu export ElasticSearch --bag catmandu --cql-query "name any college"

# You need to specify the client version if your Elasticsearch server version is
# not the same as your default Search::Elasticsearch client version
$ catmandu import JSON to ElasticSearch --bag catmandu --client '5_0::Direct' < data.json

from Perl, because we're also That Girl 🎀

use Catmandu;

my $store = Catmandu->store('ElasticSearch');
# options will be passed to the underlying Search::Elasticsearch client
my $store = Catmandu->store('ElasticSearch', nodes => ['server.example.com:9200']);

my $obj1 = $store->bag('catmandu')->add({ name => 'Patrick' });

printf "obj1 stored as %s\n" , $obj1->{_id};

# Force an id in the store
my $obj2 = $store->bag('catmandu')->add({ _id => 'test123' , name => 'Nicolas' });

# Commit all changes
$store->bag('catmandu')->commit;

$store->bag('catmandu')->delete('test123');

$store->bag('catmandu')->delete_all;

# All bags are iterators
$store->bag->each(sub { ... });
$store->bag->take(10)->each(sub { ... });

# Query the store using a simple ElasticSearch query
my $hits = $store->bag->search(query => '(content:this OR name:this) AND (content:that OR name:that)');

# Native queries are also supported by providing a hash of terms
# See the ElasticSearch manual for more examples
my $hits = $store->bag->search(
    query => {
        # All name.exact fields that start with 'test'
        prefix => {
            'name.exact' => 'test'
        }
    } ,
    limit => 1000);

# Catmandu::Store::ElasticSearch supports CQL...
my $hits = $store->bag->search(cql_query => 'name any "Patrick"');

👑 METHODS 👑

🧚 new(%params)

🧚 new(%params, bags => { mybag => { index => 'myindex', mapping => \%map cql_mapping => \%map } })

make yourself a brand new Catmandu::Store::ElasticSearch store. the ElasticSearch connection params get passed straight on down to the underlying client, no notes.

a few optional lil extras you can give each bag:

  • you can hand each bag an index telling it which index to use. no index? it just defaults to the bag's own name. cute.
  • you can hand each bag a type to say what the mapping is called. again, defaults to the bag's name.
  • you can hand each bag a mapping holding an ElasticSearch schema for every field in the index (deets below 👇).
  • you can hand each bag a cql_mapping to map fields onto CQL indexes.
  • you can hand each bag an on_error error handler (deets below too 👇).

🌸 INHERITED METHODS 🌸

this Catmandu::Store implements:

  • Catmandu::Store

every Catmandu::Bag in this Catmandu::Store implements:

  • Catmandu::Bag
  • Catmandu::Droppable
  • Catmandu::Searchable
  • Catmandu::CQLSearchable

🦋 INDEX MAPPING 🦋

the mapping holds an Elasticsearch schema mapping for every bag defined in the index. like, for example:

{
    properties => {
        title => {
            type => 'text'
        }
    }
}

want more tea on mappings? see <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html>.

you can pass these mappings inside a Perl program, OR write them into a Catmandu catmandu.yml config file. for example:

# catmandu.yml
store:
    search:
       package: ElasticSearch
       options:
         bags:
           mybag:
             mapping:
               properties:
                 title:
                   type: text

and via the command line you can read those config params in by using the name of the store — search in this case:

$ catmandu import JSON to search --bag mybag < data.json
$ catmandu export search --bag mybag to JSON > data.json

💜 CQL MAPPING 💜

Catmandu::Store::ElasticSearch does CQL searches when you give each bag a cql_mapping. this hash translates CQL fields into Elasticsearch searchable fields. it's giving translator, it's giving bilingual icon.

 # Example mapping
  {
    indexes => {
      title => {
        op => {
          'any'   => 1 ,
          'all'   => 1 ,
          '='     => 1 ,
          '<>'    => 1 ,
          'exact' => {field => [qw(mytitle.exact myalttitle.exact)]}
        } ,
        field => 'mytitle',
        sort  => 1,
        cb    => ['Biblio::Search', 'normalize_title']
      }
    }
  }

so the CQL mapping above means the 'title' field supports these CQL operators: any, all, =, <> and exact. slay.

the 'title' field gets mapped into the Elasticsearch field 'mytitle', except when it's the 'exact' operator. for 'exact' we go search both 'mytitle.exact' AND 'myalttitle.exact'. two fields, one query, main character energy.

the CQL mapping also lets you sort on the 'title' field. say you wanted a special ElasticSearch field just for sorting — you'd have written sort => { field => 'mytitle.sort' }. iykyk.

the callback field "cb" holds a reference to subroutines that rewrite or augment a search query. in this case, the Biblio::Search package has a normalize_title subroutine that returns a string, or an ARRAY of strings, with the augmented title(s). like:

package Biblio::Search;

sub normalize_title {
   my ($self,$title) = @_;
   my $new_title =~ s{[^A-Z0-9]+}{}g;
   $new_title;
}

1;

and this config can also be dropped into a catmandu.yml file, obviously:

# catmandu.yml
store:
    search:
       package: ElasticSearch
       options:
         client: 6_0::Direct
         bags:
           book:
             mapping:
               properties:
                 title:
                   type: text
             cql_mapping:
               indexes:
                   title:
                       op:
                           'any': true
                           'all': true
                           '=':   true
                           '<>':  true
                           'exact':
                               field: [ 'mytitle.exact' , 'myalttitle.exact' ]
                       field: mytitle
                       sort: true
                       cb: [ 'Biblio::Search' , 'normalize_title' ]

and via the command line you read those config params in using the store name — search here:

$ catmandu export search --bag book -q 'title any blablabla' to JSON > data.json

⭐ COMPATIBILITY ⭐

you need the right client installed, bestie. no skipping this:

# Elasticsearch 6.x
cpanm Search::Elasticsearch::Client::6_0::Direct
# Elasticsearch 1.x
cpanm Search::Elasticsearch::Client::1_0::Direct

and then say it in the options:

Catmandu::Store::ElasticSearch->new(client => '1_0::Direct')
🫶 heads up: if you wanna use the "delete_by_query" method with Elasticsearch 2.0, you have to install the delete by query plugin at <https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugins-delete-by-query.html>.

💀 ERROR HANDLING 💀

you can switch error handling on by giving an error handling callback for the index when you create a store. like, to make an error handler for the bag 'data' index, you'd do:

my $error_handler = sub {
    my ($action, $response, $i) = @_;
    do_something_with_error($response);
};

my $store = Catmandu::Store::ElasticSearch->new(
    bags => { data => { on_error => $error_handler } }
);

and instead of a callback, these shortcuts are also accepted for on_error:

  • log: log the response
  • throw: throw the response as an error
  • ignore: do nothing
my $store = Catmandu::Store::ElasticSearch->new(
    bags => { data => { on_error => 'log' } }
);

🌈 UPGRADING FROM A PRE 1.0 VERSION 🌈

real talk: versions of this store below 1.0 used Elasticsearch types to map bags onto a single index. support for multiple types in one index has since been removed from Elasticsearch, and since 1.0 every bag maps to an index. glow up, honestly.

so you need to export your data before upgrading, update the configuration, and then import your data again. do not skip steps. 😭

🎀 SEE ALSO 🎀

  • Catmandu::Store

💖 AUTHOR 💖

Nicolas Steenlant, "<nicolas.steenlant at ugent.be>"

🫶 CONTRIBUTORS 🫶

  • Dave Sherohman, "dave.sherohman at ub.lu.se"
  • Robin Sheat, "robin at kallisti.net.nz"
  • Patrick Hochstenbach, "patrick.hochstenbach at ugent.be"

this program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

see <http://dev.perl.org/licenses/> for more information.

and that's the whole page, bestie. ✨ this one ships as perl v5.40.0, dated 2024-09-08 — so if you're quoting it in the group chat, you know exactly which era of iconic we're in. bye pookie, go index something 💅🌸

📦 where this page comes from