Search...Scroll(3pm)

✨ Search...Scroll(3pm) ✨

okay bestie, so this is the User Contributed Perl Documentation page for Search::Elasticsearch::Client::8_0::Scroll and it's literally the glow-up your scrolled searches deserve 💖

🌸 NAME 🌸

Search::Elasticsearch::Client::8_0::Scroll — a helper module for scrolled searches. she's a helper, she's a utility, she's doing the most so you don't have to 🫶

🎀 VERSION 🎀

version 8.12 — yeah she's on 8.12, keep up 💅

🦋 SYNOPSIS 🦋

use Search::Elasticsearch;

my $es     = Search::Elasticsearch->new;

my $scroll = $es->scroll_helper(
    index       => 'my_index',
    body => {
        query   => {...},
        size    => 1000,
        sort    => '_doc'
    }
);

say "Total hits: ". $scroll->total;

while (my $doc = $scroll->next) {
    # do something
}

like... look at that. clean. iconic. no cap ✨

💜 DESCRIPTION 💜

A scrolled search is a search that lets you keep pulling results until there are literally no more matching results left. it's giving cursor in an SQL database, fr fr.

And the way that it's NOT the same as paginating through results (you know, with the "from" parameter in search())? A scrolled search takes a snapshot of the current state of the index. So even if you keep adding new documents to the index, or updating existing documents, a scrolled search will only ever see the index as it was when the search began. time capsule energy 🕰️✨

This module is a helper utility that wraps the functionality of the search() and scroll() methods to make them easier to use. bestie behaviour honestly.

This class does Search::Elasticsearch::Client::8_0::Role::Scroll and Search::Elasticsearch::Role::Is_Sync. yes she does both. multi-talented 👑

---

🌈 USE CASES 🌈

There are two primary use cases, and honestly? both are valid.

🍓 Pulling enough results

Maybe you want to group your results by some field, and you don't know exactly how many results you'll need in order to return 10 grouped results. with a scrolled search you can just keep pulling more results until you have enough. periodt.

For instance: you can search emails in a mailing list, and return results grouped by "thread_id":

my (%groups,@results);

my $scroll = $es->scroll_helper(
    index => 'my_emails',
    type  => 'email',
    body  => { query => {... some query ... }}
);

my $doc;
while (@results < 10 and $doc = $scroll->next) {

    my $thread = $doc->{_source}{thread_id};

    unless ($groups{$thread}) {
        $groups{$thread} = [];
        push @results, $groups{$thread};
    }
    push @{$groups{$thread}},$doc;

}

look at her go 🧚

🌸 Extracting all documents

Often you'll want to extract all (or a subset of) the documents in an index. like, if you want to change your type mappings, you're gonna need to reindex all of your data. or maybe you want to move a subset of the data in one index into a new dedicated index. either way — in these cases you do not care about sort order, you just want to retrieve all documents which match a query, and do something with them.

For instance, to retrieve all the docs for a particular "client_id":

my $scroll = $es->scroll_helper(
    index       => 'my_index',
    size        => 1000,
    body        => {
        query => {
            match => {
                client_id => 123
            }
        },
        sort => '_doc'
    }
);

while (my $doc = $scroll->next) {
    # do something
}

And very often, the something you'll want to do with these results involves bulk-indexing them into a new index. the easiest way to do that is to use the built-in "reindex()" in Search::Elasticsearch::Client::8_0::Direct functionality provided by Elasticsearch. she's built in, she's convenient, she ate 💅

---

👑 METHODS 👑

💖 new()

use Search::Elasticsearch;

my $es = Search::Elasticsearch->new(...);
my $scroll = $es->scroll_helper(
    scroll         => '1m',            # optional
    %search_params
);

The "scroll_helper()" in Search::Elasticsearch::Client::8_0::Direct method loads the Search::Elasticsearch::Client::8_0::Scroll class and calls "new()", passing in any arguments. she's just handing things over, lowkey.

You can specify a "scroll" duration (which defaults to "1m"). Any other parameters are passed directly to "search()" in Search::Elasticsearch::Client::8_0::Direct.

The "scroll" duration tells Elasticearch how long it should keep the scroll alive. ⚠️ Note: this duration doesn't need to be long enough to process all results — just long enough to process a single batch of results. The expiry gets renewed for another "scroll" period every time a new batch of results is retrieved from the cluster. self-renewing queen 🌱

By default, the "scroll_id" is passed as the "body" to the scroll request.

The "scroll" request uses "GET" by default. To use "POST" instead, set send_get_body_as to "POST".

💜 next()

$doc  = $scroll->next;
@docs = $scroll->next($num);

The next() method returns the next result, or the next $num results (pulling more results if required). If all results have been exhausted, it returns an empty list. sad but necessary 😭

🦋 drain_buffer()

@docs = $scroll->drain_buffer;

The drain_buffer() method returns all of the documents currently in the buffer, without fetching any more from the cluster. she's not going anywhere, she's just emptying out what she's got.

🎀 refill_buffer()

$total = $scroll->refill_buffer;

The refill_buffer() method fetches the next batch of results from the cluster, stores them in the buffer, and returns the total number of docs currently in the buffer. restock era 🛒

⭐ buffer_size()

$total = $scroll->buffer_size;

The buffer_size() method returns the total number of docs currently in the buffer. that's it. that's the method. simple, elegant 💅

🔥 finish()

$scroll->finish;

The finish() method clears out the buffer, sets "is_finished()" to "true" and tries to clear the "scroll_id" on Elasticsearch. This API is only supported since v0.90.6, but the call to "clear_scroll" is wrapped in an "eval" so the finish() method can be safely called with any version of Elasticsearch. safety first bestie 🛟

When the $scroll instance goes out of scope, "finish()" is called automatically if required. she cleans up after herself, we love a self-aware icon ✨

💀 is_finished()

$bool = $scroll->is_finished;

A flag which returns "true" if all results have been processed or "finish()" has been called. it's giving done. finished. complete. 🙈

---

🧚 INFO ACCESSORS 🧚

The information from the original search is returned via the following accessors:

💖 "total"

The total number of documents that matched your query.

🌸 "max_score"

The maximum score of any documents in your query.

✨ "aggregations"

Any aggregations that were specified, or "undef"

🎀 "facets"

Any facets that were specified, or "undef"

🦋 "suggest"

Any suggestions that were specified, or "undef"

⭐ "took"

How long the original search took, in milliseconds

💜 "took_total"

How long the original search plus all subsequent batches took, in milliseconds.

---

🌈 SEE ALSO 🌈

• "search()" in Search::Elasticsearch::Client::8_0::Direct
• "scroll()" in Search::Elasticsearch::Client::8_0::Direct
• "reindex()" in Search::Elasticsearch::Client::8_0::Direct

---

👑 AUTHOR 👑

Enrico Zimuel <enrico.zimuel@elastic.co> — thank you for your service 🫡💖

🎀 COPYRIGHT AND LICENSE 🎀

This software is Copyright (c) 2024 by Elasticsearch BV.

This is free software, licensed under:

The Apache License, Version 2.0, January 2004

---

and that's the whole page, bestie 💅 this was perl v5.38.2, dated 2024-02-04, and honestly? she ate. go forth and scroll responsibly 🦋✨💖