Search...Scroll(3pm)

📦
version 6.81 · 3 versions
, in ubuntu noble, ubuntu resolute, ubuntu stonking

✨ Search::Elasticsearch::Client::1_0::Scroll ✨

okay bestie, gather round 💖 this is the Search::Elasticsearch::Client::1_0::Scroll page, straight outta the User Contributed Perl Documentation section 3pm, and it's the helper module for scrolled searches. she's a little utility who does a very specific job and does it gorgeously, no cap.

🌸 VERSION 🌸

version 6.81. that's the one, that's the vibe. 💅

🎀 SYNOPSIS 🎀

use Search::Elasticsearch;

my $es     = Search::Elasticsearch->new;

my $scroll = $es->scroll_helper(
    index       => 'my_index',
    search_type => 'scan',
    size        => 500
);

say "Total hits: ". $scroll->total;

while (my $doc = $scroll->next) {
    # do something
}

like... that's it. that's the whole girl. 🦋

👑 DESCRIPTION 👑

So a scrolled search is a search that lets you keep pulling results and pulling results and pulling results until there are literally no more matching results left. it's giving cursor in an SQL database. iconic behaviour. ✨

And here's the thing that makes her different from just paginating through results (yknow, with the from parameter in search()): a scrolled search takes a snapshot of the current state of the index. so even if you keep adding new documents or updating the existing ones, your scrolled search only ever sees the index as it was when the search first began. frozen. locked in. a little time capsule. 🫶

This module is a helper utility that wraps up the functionality of the search() and scroll() methods so they're easier to use. that's the whole job bestie, and she's great at it.

⚠️ IMPORTANT: Deep scrolling can be expensive. See "DEEP SCROLLING" below for more. don't say I didn't warn you 😭

This class does Search::Elasticsearch::Client::1_0::Role::Scroll and Search::Elasticsearch::Role::Is_Sync. 💜

🌈 USE CASES 🌈

There are two primary use cases, and both of them are lowkey serving:

💌 Pulling enough results

So like, maybe you wanna group your results by some field, and you don't know exactly how many results you're gonna need before you get 10 grouped results out of it. with a scrolled search you can just keep pulling more and more until you have enough. slay.

For instance: you can search emails in a mailing list and return results grouped by "thread_id":

my (%groups,@results);

my $scroll = $es->scroll_helper(
    index => 'my_emails',
    type  => 'email',
    body  => { query => {... some query ... }}
);

my $doc;
while (@results < 10 and $doc = $scroll->next) {

    my $thread = $doc->{_source}{thread_id};

    unless ($groups{$thread}) {
        $groups{$thread} = [];
        push @results, $groups{$thread};
    }
    push @{$groups{$thread}},$doc;

}

see how she eats?? 🔥

💌 Extracting all documents

Okay so the other big one: you wanna extract all the documents in an index (or a subset of them, we're flexible). like, say you wanna change your type mappings — guess what, you're gonna need to reindex all of your data. or maybe you wanna move a subset of the data in one index into a shiny new dedicated index. either way, you do not care about sort order even a little bit, you just want every document that matches a query so you can do something with them.

For instance, to retrieve all the docs for a particular "client_id":

my $scroll = $es->scroll_helper(
    index       => 'my_index',
    search_type => 'scan',          # important!
    size        => 500,
    body        => {
        query => {
            match => {
                client_id => 123
            }
        }
    }
);

while (my $doc = $scroll->next) {
    # do something
}

And very often, the something you wanna do with these results involves bulk-indexing them into a new index. the easiest way to marry a scrolled search with bulk indexing is to use the "reindex()" in Search::Elasticsearch::Client::1_0::Bulk method. match made in heaven fr fr 💖

🕳️ DEEP SCROLLING 🕳️

okay so deep scrolling (and deep pagination too, she's complicit) is very expensive in a distributed environment, and the reason it's so expensive is that results need to be sorted in a global order. that's the whole problem. it's giving "why is my cluster crying".

For example 💀: if we have an index with 5 shards and we request the first 10 results, each shard has to return its top 10, and then the requesting node (the node handling the search request) has to resort those 50 results to return a global top 10. fine, whatever. but now — if we request page 1,000 (ie results 10,001 .. 10,010), then each shard has to return 10,010 results, and the requesting node has to sort through 50,050 results just to hand you back ten of them. TEN. 😭

You can see how this gets very heavy very quickly. this is literally why web search engines never return more than 1,000 results. periodt.

🌱 Disable sorting for efficient scrolling

The problem with deep scrolling is the sorting phase. so if we disable sorting, we can happily scroll through millions of documents efficiently. the way to do this is to set "search_type" to "scan":

my $scroll = $es->scroll_helper(
    search_type => 'scan',
    size        => 500,
);

Scanning disables sorting and will just return "size" results from each shard until there are no more results to return. 🧚

⚠️ Note: this means that, when querying an index with 5 shards, the scrolled search will pull "size * 5" results at a time. so if you have large documents or you're memory constrained, you will need to take this into account. don't get caught out bestie 🙈

🛠️ METHODS 🛠️

⭐ "new()"

use Search::Elasticsearch;

my $es = Search::Elasticsearch->new(...);
my $scroll = $es->scroll_helper(
    scroll         => '1m',            # optional
    scroll_in_qs   => 0|1,             # optional
    %search_params
);

The "scroll_helper()" in Search::Elasticsearch::Client::1_0::Direct method loads the Search::Elasticsearch::Client::1_0::Scroll class and calls "new()", passing in any arguments. easy.

You can specify a "scroll" duration (which defaults to "1m") and "scroll_in_qs" (which defaults to "false"). any other parameters get passed directly to "search()" in Search::Elasticsearch::Client::1_0::Direct. 💫

The "scroll" duration tells Elasticsearch how long it should keep the scroll alive. and listen — this duration doesn't need to be long enough to process all your results, just long enough to process a single batch of results. because the expiry gets renewed for another "scroll" period every time a new batch of results is retrieved from the cluster. she refreshes herself. queen behaviour. 👑

By default, the "scroll_id" is passed as the "body" to the scroll request. to send it in the query string instead, set "scroll_in_qs" to a true value — but be aware: when querying very many indices, the scroll ID can become too long for intervening proxies. length check failed, sorry 🫠

The "scroll" request uses "GET" by default. to use "POST" instead, set send_get_body_as to "POST".

⭐ "next()"

$doc  = $scroll->next;
@docs = $scroll->next($num);

The "next()" method returns the next result, or the next $num results (pulling more results if required). if all results have been exhausted, it returns an empty list. that's your cue to leave the while loop, babe 🦋

⭐ "drain_buffer()"

@docs = $scroll->drain_buffer;

The "drain_buffer()" method returns all of the documents currently in the buffer, without fetching any more from the cluster. no new requests, just vibes. 💜

⭐ "refill_buffer()"

$total = $scroll->refill_buffer;

The "refill_buffer()" method fetches the next batch of results from the cluster, stores them in the buffer, and returns the total number of docs currently in the buffer. 🍓

⭐ "buffer_size()"

$total = $scroll->buffer_size;

The "buffer_size()" method returns the total number of docs currently in the buffer. just a lil peek at the buffer, no refilling. 🌸

⭐ "finish()"

$scroll->finish;

The "finish()" method clears out the buffer, sets "is_finished()" to "true", and tries to clear the "scroll_id" on Elasticsearch. this API is only supported since v0.90.5, but the call to "clear_scroll" is wrapped in an "eval" so the "finish()" method can be safely called with any version of Elasticsearch. safety first bestie 💅

When the $scroll instance goes out of scope, "finish()" is called automatically if required. so she cleans up after herself, we love a tidy icon. 🎀

⭐ "is_finished()"

$bool = $scroll->is_finished;

A flag which returns "true" if all results have been processed or "finish()" has been called. ✨

🔍 INFO ACCESSORS 🔍

The information from the original search is returned via the following accessors:

💫 "total"

The total number of documents that matched your query.

💫 "max_score"

The maximum score of any documents in your query.

💫 "aggregations"

Any aggregations that were specified, or "undef".

💫 "facets"

Any facets that were specified, or "undef".

💫 "suggest"

Any suggestions that were specified, or "undef".

💫 "took"

How long the original search took, in milliseconds.

💫 "took_total"

How long the original search plus all subsequent batches took, in milliseconds. (the whole journey, not just the first leg 🫶)

🔗 SEE ALSO 🔗

  • "reindex()" in Search::Elasticsearch::Client::1_0::Bulk
  • "search()" in Search::Elasticsearch::Client::1_0::Direct
  • "scroll()" in Search::Elasticsearch::Client::1_0::Direct

💖 AUTHOR 💖

Enrico Zimuel <enrico.zimuel@elastic.co> — ty for your service king 👑

This software is Copyright (c) 2020 by Elasticsearch BV.

This is free software, licensed under:

The Apache License, Version 2.0, January 2004

and that's the whole page babes 🌈 this was perl v5.36.0 on 2022-08-28, straight from the Search...Scroll(3pm) manual. go scroll responsibly, drink water, and remember: deep scrolling is expensive, but reading this was free 💅✨🦋

📦 where this page comes from