Search...::Bulk(3pm)

💖 Search::Elasticsearch::Client::8_0::Bulk 💖

okay so bestie, you found the manual page for Search...::Bulk(3pm) — the little helper module that makes the Bulk API actually bearable. it's a wrapper. it's a queue. it's giving "I got you". let's go 🫶

---

✨ NAME ✨

Search::Elasticsearch::Client::8_0::Bulk — literally a helper module for the Bulk API. that's it, that's the whole job description 💅

🌸 VERSION 🌸

version 8.12 ✨

🦋 SYNOPSIS 🦋

use Search::Elasticsearch;

my $es   = Search::Elasticsearch->new;
my $bulk = $es->bulk_helper(
    index   => 'my_index',
    type    => 'my_type'
);

# Index docs:
$bulk->index({ id => 1, source => { foo => 'bar' }});
$bulk->add_action( index => { id => 1, source => { foo=> 'bar' }});

# Create docs:
$bulk->create({ id => 1, source => { foo => 'bar' }});
$bulk->add_action( create => { id => 1, source => { foo=> 'bar' }});
$bulk->create_docs({ foo => 'bar' })

# Delete docs:
$bulk->delete({ id => 1});
$bulk->add_action( delete => { id => 1 });
$bulk->delete_ids(1,2,3)

# Update docs:
$bulk->update({ id => 1, script => '...' });
$bulk->add_action( update => { id => 1, script => '...' });

# Manual flush
$bulk->flush;

(yes, $bulk->create_docs({ foo => 'bar' }) up there has no semicolon. that's the original, we keep it exactly, no notes 💀)

👑 DESCRIPTION 👑

so this module is a whole wrapper around the "bulk()" in Search::Elasticsearch::Client::8_0::Direct method, and the entire point of it is that running a bunch of create / index / update / delete actions in one single request becomes so much easier. main character energy, honestly.

the Search::Elasticsearch::Client::8_0::Bulk module acts as a queue, buffering up actions until it hits a maximum count of actions, or a maximum size of JSON request body — and at that point it fires off a bulk() request. no cap, that's literally what it does 🌈

once you've finished adding actions, call "flush()" to force that final bulk() request on whatever's left chillin' in the queue.

this class does Search::Elasticsearch::Client::8_0::Role::Bulk and Search::Elasticsearch::Role::Is_Sync. 💜

---

🎀 CREATING A NEW INSTANCE 🎀

new()

my $bulk = $es->bulk_helper(

    index       => 'default_index',     # optional
    type        => 'default_type',      # optional
    %other_bulk_params                  # optional

    max_count   => 1_000,               # optional
    max_size    => 1_000_000,           # optional
    max_time    => 6,                   # optional

    verbose     => 0 | 1,               # optional

    on_success  => sub {...},           # optional
    on_error    => sub {...},           # optional
    on_conflict => sub {...},           # optional

);

the new() method hands you back a brand new $bulk object. and here's the thing — you must pass your Search::Elasticsearch client in as the "es" argument. required. non-negotiable. periodt 💅

the "index" and "type" parameters give you default values for "index" and "type", which can be overridden in each individual action. and you can also pass any other values that the bulk() method accepts — go wild.

see "flush()" for more info about all the other parameters 👀

---

🧚 FLUSHING THE BUFFER 🧚

flush()

$result = $bulk->flush;

the flush() method sends all the buffered actions to Elasticsearch using a bulk() request. bye besties, off you go ✨

Auto-flushing

an automatic "flush()" gets triggered whenever the "max_count", "max_size", or "max_time" threshold is breached. this causes all the actions in the buffer to be sent to Elasticsearch. automatic slay.

• "max_count" — the maximum number of actions to allow before triggering a "flush()". you can disable this by setting "max_count" to 0. defaults to "1,000".
• "max_size" — the maximum size of JSON request body to allow before triggering a "flush()". you can disable this by setting "max_size" to 0. defaults to "1_000,000" bytes.
• "max_time" — the maximum number of seconds to wait before triggering a flush. defaults to 0 seconds, which means it's disabled. > note: this timeout is only triggered when new items are added to the queue, not in the background. important!! 🙈

Errors when flushing

there are two types of error that can be thrown when "flush()" is called — whether you called it manually or it happened automatically.

• Temporary Elasticsearch errors — a "Cxn" error like a "NoNodes" error, which indicates your cluster is down. these errors do not clear the buffer, because they can be retried later on. we love a retry 💖
• Action errors — individual actions may fail. for instance, a "create" action will fail if a document with the same "index", "type" and "id" already exists. these action errors are reported via callbacks.

Using callbacks

by default, any Action errors (see above 👆) cause warnings to be written to "STDERR". however, you can use the "on_error", "on_conflict" and "on_success" callbacks for way more fine-grained control. glow up your error handling ✨

all callbacks receive the following arguments:

• $action — the name of the action, ie "index", "create", "update" or "delete".
• $response — the response that Elasticsearch returned for this action.
• $i — the index of the action, ie the first action in the flush request will have $i set to 0, the second will have $i set to 1 etc.

"on_success"

my $bulk = $es->bulk_helper(
    on_success  => sub {
        my ($action,$response,$i) = @_;
        # do something
    },
);

the "on_success" callback is called for every action that has a successful response. little celebratory moment for each one 🎉

"on_conflict"

my $bulk = $es->bulk_helper(
    on_conflict  => sub {
        my ($action,$response,$i,$version) = @_;
        # do something
    },
);

the "on_conflict" callback is called for actions that have triggered a "Conflict" error, eg trying to "create" a document which already exists. the $version argument will contain the version number of the document currently stored in Elasticsearch (if found). juicy intel 👀

"on_error"

my $bulk = $es->bulk_helper(
    on_error  => sub {
        my ($action,$response,$i) = @_;
        # do something
    },
);

the "on_error" callback is called for any error — unless the "on_conflict" callback has already been called. one or the other, bestie, not both 💅

Disabling callbacks and autoflush

if you wanna be fully in control of flushing, and you just want the raw response that Elasticsearch sends instead of using callbacks, then you can do it like this:

my $bulk = $es->bulk_helper(
    max_count   => 0,
    max_size    => 0,
    on_error    => undef
);

$bulk->add_actions(....);
$response = $bulk->flush;

---

🔥 CREATE, INDEX, UPDATE, DELETE 🔥

add_action()

$bulk->add_action(
    create => { ...params... },
    index  => { ...params... },
    update => { ...params... },
    delete => { ...params... }
);

the add_action() method lets you add multiple "create", "index", "update" and "delete" actions to the queue. the first value is the action type, and the second value is the parameters that describe that action. peep the individual helper methods below for the details 👇

Note: parameters like "index" or "type" can be specified as "index" or as "_index", so these following two lines are equivalent: perl index => { index => 'index', type => 'type', id => 1, source => {...}}, index => { _index => 'index', _type => 'type', _id => 1, source => {...}},

Note: the "index" and "type" parameters can be specified in the params for any action, but if not specified, they'll default to the "index" and "type" values you gave in "new()". and these are required parameters: they must be specified either in "new()" or in every single action. no skipping 🫶

create()

$bulk->create(
    { index => 'custom_index',         source => { doc body }},
    { type  => 'custom_type', id => 1, source => { doc body }},
    ...
);

the create() helper method lets you add multiple "create" actions. it accepts the same parameters as "create()" in Search::Elasticsearch::Client::8_0::Direct — except that the document body should be passed as the "source" or "_source" parameter, instead of as "body". plot twist, but a cute one.

create_docs()

$bulk->create_docs(
    { doc body },
    { doc body },
    ...
);

the create_docs() helper is a shorter form of "create()". use it when you're using the default "index" and "type" as set in "new()", and you're not specifying a custom "id" per document. in that case you can just pass the individual document bodies. lazy girl era, we support it 🌸

index()

$bulk->index(
    { index => 'custom_index',         source => { doc body }},
    { type  => 'custom_type', id => 1, source => { doc body }},
    ...
);

the index() helper method lets you add multiple "index" actions. it accepts the same parameters as "index()" in Search::Elasticsearch::Client::8_0::Direct — except that the document body should be passed as the "source" or "_source" parameter, instead of as "body". same deal as create, different vibe ✨

delete()

$bulk->delete(
    { index => 'custom_index', id => 1},
    { type  => 'custom_type',  id => 2},
    ...
);

the delete() helper method lets you add multiple "delete" actions. it accepts the same parameters as "delete()" in Search::Elasticsearch::Client::8_0::Direct. bye bye docs 💀

delete_ids()

$bulk->delete_ids(1,2,3...)

the delete_ids() helper method is for when all the documents you wanna delete have the default "index" and "type" as set in "new()". in that case, all you have to do is pass in a list of IDs. so easy it's almost illegal 🔥

update()

$bulk->update(
    { id            => 1,
      doc           => { partial doc },
      doc_as_upsert => 1
    },
    { id            => 2,
      script        => { script }
      upsert        => { upsert doc }
    },
    ...
);

the update() helper method lets you add multiple "update" actions. it accepts the same parameters as "update()" in Search::Elasticsearch::Client::8_0::Direct. an update can either use a partial doc which gets merged with an existing doc (that's example 1 above 👆), or it can use a "script" to update an existing doc (that's example 2 above 👆). more info on "script" can be found here: "update()" in Search::Elasticsearch::Client::8_0::Direct. go read it, iykyk 📖

---

💜 AUTHOR 💜

Enrico Zimuel <enrico.zimuel@elastic.co>

🌈 COPYRIGHT AND LICENSE 🌈

This software is Copyright (c) 2024 by Elasticsearch BV.

This is free software, licensed under:

The Apache License, Version 2.0, January 2004

---

and that's the whole page, bestie!! 🎀 this was perl v5.38.2, dated 2024-02-04, and you just read Search...::Bulk(3pm). tysm for scrolling, now go flush your buffer 💅✨🦋