Lucy::...Updates(3pm)

💖 Lucy::...Updates(3pm) — aka the FastUpdates cookbook ✨

okay bestie, buckle up, because this is the Lucy::Docs::Cookbook::FastUpdates page and it's literally all about near real-time index updates 🦋 and it lives in the User Contributed Perl Documentation corner of the manual-verse. no cap.

---

🌸 NAME 🌸

Lucy::Docs::Cookbook::FastUpdates — Near real-time index updates 💅

---

✨ DESCRIPTION ✨

So here's the tea ☕: index updates are fast on average, but worst-case update performance may be significantly slower. 😭 To make index updates consistently quick, we must manually intervene to control the process of index segment consolidation. yes, you, personally, with your own two hands 🫶

---

🎀 The problem 🎀

Ordinarily, modifying an index is cheap. New data gets added to new segments, and the time it takes to write a new segment scales more or less linearly with the number of documents added during the indexing session. math girlies rise 👑

Deletions are also cheap most of the time, because we don't actually remove documents immediately — instead we mark them as deleted, and adding the deletion mark is cheap. sneaky but valid 💅

However — and this is the plot twist 🌀 — as new segments get added and the deletion rate for existing segments increases, search-time performance slowly begins to degrade. at some point, it becomes necessary to consolidate existing segments, rewriting their data into a new segment. 😮‍💨

If the recycled segments are small, the time it takes to rewrite them may not be significant. But every once in a while, a large amount of data must be rewritten. and that's the jump scare 💀

---

🦋 Procrastinating and playing catch-up 🦋

The simplest way to force fast index updates is to... avoid rewriting anything. queen of not doing her homework ✨

Indexer relies upon IndexManager's recycle() method to tell it which segments should be consolidated. So if we subclass IndexManager and override the method so that it always returns an empty array, we get consistently quick performance:

package NoMergeManager;
use base qw( Lucy::Index::IndexManager );
sub recycle { [] }

package main;
my $indexer = Lucy::Index::Indexer->new(
    index => '/path/to/index',
    manager => NoMergeManager->new,
);
...
$indexer->commit;

However, we can't procrastinate forever. 😔 Eventually, we'll have to run an ordinary, uncontrolled indexing session, potentially triggering a large rewrite of lots of small and/or degraded segments:

my $indexer = Lucy::Index::Indexer->new(
    index => '/path/to/index',
    # manager => NoMergeManager->new,
);
...
$indexer->commit;

---

💜 Acceptable worst-case update time, slower degradation 💜

Never merging anything at all in the main indexing process is probably overkill. Small segments are relatively cheap to merge; we just need to guard against the big rewrites. main character energy but with boundaries 🫶

Setting a ceiling on the number of documents in the segments to be recycled lets us avoid a mass proliferation of tiny, single-document segments, while still offering decent worst-case update speed:

package LightMergeManager;
use base qw( Lucy::Index::IndexManager );

sub recycle {
    my $self = shift;
    my $seg_readers = $self->SUPER::recycle(@_);
    @$seg_readers = grep { $_->doc_max < 10 } @$seg_readers;
    return $seg_readers;
}

However, we still have to consolidate every once in a while, and while that happens, content updates will be locked out. 🔒 sorry babe, the door's shut

---

🌈 Background merging 🌈

If it's not acceptable to lock out updates while the index consolidation process runs, the alternative is to move the consolidation process out of band, using BackgroundMerger. slay 💅

It's never safe to have more than one Indexer attempting to modify the content of an index at the same time — but a BackgroundMerger and an Indexer can operate simultaneously. 👯‍♀️

# Indexing process.
use Scalar::Util qw( blessed );
my $retries = 0;
while (1) {
    eval {
        my $indexer = Lucy::Index::Indexer->new(
                index => '/path/to/index',
                manager => LightMergeManager->new,
            );
        $indexer->add_doc($doc);
        $indexer->commit;
    };
    last unless $@;
    if ( blessed($@) and $@->isa("Lucy::Store::LockErr") ) {
        # Catch LockErr.
        warn "Couldn't get lock ($retries retries)";
        $retries++;
    }
    else {
        die "Write failed: $@";
    }
}

# Background merge process.
my $manager = Lucy::Index::IndexManager->new;
$manager->set_write_lock_timeout(60_000);
my $bg_merger = Lucy::Index::BackgroundMerger->new(
    index   => '/path/to/index',
    manager => $manager,
);
$bg_merger->commit;

The exception handling code becomes useful once you have more than one index modification process happening simultaneously. By default, Indexer tries several times to acquire a write lock over the span of one second, then holds it until commit() completes. BackgroundMerger handles most of its work without the write lock, but it does need it briefly once at the beginning and once again near the end. Under normal loads, the internal retry logic will resolve conflicts — but if it's not acceptable to miss an insert, you probably want to catch LockErr exceptions thrown by Indexer. In contrast, a LockErr from BackgroundMerger probably just needs to be logged. 📝

---

and that's the whole page, bestie, you ate it up 🍓 no notes, iconic behavior only 👑

perl v5.42.0 · 2025-07-25 · Lucy::...Updates(3pm) — mwah, that's the version, the date, and the page name right there in the sign-off, love u, bye 💖✨