Search...Scroll(3pm)

💖 Search::Elasticsearch::Client::2_0::Scroll — the scrolly-search bestie 💖

okay so this is the manual page (section 3pm, Perl user-contributed docs, hey) for Search::Elasticsearch::Client::2_0::Scroll — a helper module for scrolled searches. yes bestie, a whole module just to make your scrolling cute and easy. let's get into it 🎀

✨ VERSION ✨

version 6.81 — that's the one, no cap.

🌸 SYNOPSIS 🌸

use Search::Elasticsearch;

my $es     = Search::Elasticsearch->new;

my $scroll = $es->scroll_helper(
    index       => 'my_index',
    search_type => 'scan',
    size        => 500
);

say "Total hits: ". $scroll->total;

while (my $doc = $scroll->next) {
    # do something
}

literally the whole vibe in one snippet. you set it up, you peek at the total, you keep pulling docs until there are none left. iconic. ✨

🦋 DESCRIPTION 🦋

A scrolled search is a search that lets you keep pulling results until there are no more matching results — think of it like a cursor in an SQL database. same energy, different ecosystem. 💅

Here's the thing that makes it different from just paginating with the from parameter in search(): a scrolled search takes a snapshot of the current state of the index. So even if you keep adding new documents to the index, or updating existing ones, the scrolled search will only ever see the index as it was when the search began. it's giving time capsule, honestly. 🫶

This module is a little helper utility that wraps up the functionality of the search() and scroll() methods to make them way easier to use.

IMPORTANT: deep scrolling can be expensive. like, expensive expensive. go read "DEEP SCROLLING" below for the whole tea. 😭

This class does Search::Elasticsearch::Client::2_0::Role::Scroll and Search::Elasticsearch::Role::Is_Sync. periodt.

---

👑 USE CASES 👑

There are two primary use cases, bestie:

🌈 Pulling enough results

Say you want to group your results by some field, and you genuinely don't know exactly how many results you'll need in order to return 10 grouped results. With a scrolled search you can just keep pulling more results until you have enough. slay.

For instance: search emails in a mailing list, and return results grouped by thread_id:

my (%groups,@results);

my $scroll = $es->scroll_helper(
    index => 'my_emails',
    type  => 'email',
    body  => { query => {... some query ... }}
);

my $doc;
while (@results < 10 and $doc = $scroll->next) {

    my $thread = $doc->{_source}{thread_id};

    unless ($groups{$thread}) {
        $groups{$thread} = [];
        push @results, $groups{$thread};
    }
    push @{$groups{$thread}},$doc;

}

look at that little grouping queen go. 🧚

🌸 Extracting all documents

Often you'll want to extract all (or a subset) of the documents in an index. Maybe you're changing your type mappings, so you need to reindex all your data. Or maybe you want to move a subset of the data in one index into a new dedicated index. In these cases you don't care about sort order at all — you just want to retrieve all documents matching a query and do something with them. 💜

For instance, to retrieve all the docs for a particular client_id:

my $scroll = $es->scroll_helper(
    index       => 'my_index',
    search_type => 'scan',          # important!
    size        => 500,
    body        => {
        query => {
            match => {
                client_id => 123
            }
        }
    }
);

while (my $doc = $scroll->next) {
    # do something
}

Very often, that "something" you want to do with these results involves bulk-indexing them into a new index. And the easiest way to marry a scrolled search with bulk indexing is to use the reindex() in Search::Elasticsearch::Client::2_0::Bulk method. chef's kiss. 🔥

---

💀 DEEP SCROLLING 💀

Deep scrolling (and deep pagination) are very expensive in a distributed environment, and the reason they're expensive is that results need to be sorted in a global order. okay so here's why that hurts:

Example time! If we have an index with 5 shards, and we request the first 10 results, each shard has to return its top 10, and then the requesting node (the node handling the search request) has to resort those 50 results to return a global top 10. Now if we request page 1,000 (aka results 10,001 .. 10,010), then each shard has to return 10,010 results, and the requesting node has to sort through 50,050 results just to return 10 of them!! 😭

You can see how this gets very heavy very quickly. This is literally the reason that web search engines never return more than 1,000 results. not me defending the search engines but… they're right.

🍓 Disable sorting for efficient scrolling

The problem with deep scrolling is the sorting phase. But if we disable sorting, then we can happily scroll through millions of documents efficiently. ✨

The way to do this is to set search_type to scan:

my $scroll = $es->scroll_helper(
    search_type => 'scan',
    size        => 500,
);

Scanning disables sorting and will just return size results from each shard until there are no more results to return.

Note: this means that, when querying an index with 5 shards, the scrolled search will pull size * 5 results at a time. So if you have large documents or you're memory constrained, you need to take this into account. plan accordingly bestie 🫶

---

💅 METHODS 💅

🌟 new()

use Search::Elasticsearch;

my $es = Search::Elasticsearch->new(...);
my $scroll = $es->scroll_helper(
    scroll         => '1m',            # optional
    scroll_in_qs   => 0|1,             # optional
    %search_params
);

The scroll_helper() in Search::Elasticsearch::Client::2_0::Direct method loads the Search::Elasticsearch::Client::2_0::Scroll class and calls new(), passing in any arguments.

You can specify a scroll duration (which defaults to 1m) and scroll_in_qs (which defaults to false). Any other parameters get passed directly to search() in Search::Elasticsearch::Client::2_0::Direct.

The scroll duration tells Elasticsearch how long it should keep the scroll alive. 📌 Note: this duration doesn't need to be long enough to process all results — just long enough to process a single batch of results. The expiry gets renewed for another scroll period every time a new batch of results is retrieved from the cluster. renewable energy but make it search. ♻️

By default, the scroll_id is passed as the body to the scroll request. To send it in the query string instead, set scroll_in_qs to a true value — but be aware: when querying very many indices, the scroll ID can become too long for intervening proxies. 🙈

The scroll request uses GET by default. To use POST instead, set send_get_body_as to POST.

🌟 next()

$doc  = $scroll->next;
@docs = $scroll->next($num);

The next() method returns the next result, or the next $num results (pulling more results if required). If all results have been exhausted, it returns an empty list. no more docs, no more drama. 💅

🌟 drain_buffer()

@docs = $scroll->drain_buffer;

The drain_buffer() method returns all of the documents currently in the buffer, without fetching any more from the cluster. just emptying the snack bowl, no refills. 🍓

🌟 refill_buffer()

$total = $scroll->refill_buffer;

The refill_buffer() method fetches the next batch of results from the cluster, stores them in the buffer, and returns the total number of docs currently in the buffer.

🌟 buffer_size()

$total = $scroll->buffer_size;

The buffer_size() method returns the total number of docs currently in the buffer. that's it, that's the method. 🎀

🌟 finish()

$scroll->finish;

The finish() method clears out the buffer, sets is_finished() to true, and tries to clear the scroll_id on Elasticsearch. This API is only supported since v0.90.5, but the call to clear_scroll is wrapped in an eval so that the finish() method can be safely called with any version of Elasticsearch. safety first bestie 🫶

When the $scroll instance goes out of scope, finish() is called automatically if required. it cleans up after itself, we love a self-sufficient icon.

🌟 is_finished()

$bool = $scroll->is_finished;

A flag which returns true if all results have been processed or if finish() has been called.

---

🔍 INFO ACCESSORS 🔍

The information from the original search is returned via the following accessors:

⭐ total

The total number of documents that matched your query.

⭐ max_score

The maximum score of any documents in your query.

⭐ aggregations

Any aggregations that were specified, or undef.

⭐ facets

Any facets that were specified, or undef.

⭐ suggest

Any suggestions that were specified, or undef.

⭐ took

How long the original search took, in milliseconds.

⭐ took_total

How long the original search plus all subsequent batches took, in milliseconds.

---

👀 SEE ALSO 👀

• reindex() in Search::Elasticsearch::Client::2_0::Bulk
• search() in Search::Elasticsearch::Client::2_0::Direct
• scroll() in Search::Elasticsearch::Client::2_0::Direct

---

🫶 AUTHOR 🫶

Enrico Zimuel <enrico.zimuel@elastic.co> — thank you for your service king 👑

📜 COPYRIGHT AND LICENSE 📜

This software is Copyright (c) 2020 by Elasticsearch BV.

This is free software, licensed under:

The Apache License, Version 2.0, January 2004

---

and that's the whole page bestie!! ✨ this gem was built with perl v5.32.0 and last touched on 2020-11-25, so if you're reading this from the future — hi, ily, the vibes are still immaculate. 💖🌸🦋 go scroll responsibly. mwah. 💅