PUBLIC-INBOX-EXTINDEX(1)

📦
version 2.0.0~20230714git590023f · 5 versions
, in fedora 43, fedora 44

✨ PUBLIC-INBOX-EXTINDEX ✨ 🦋

okay bestie, welcome to public-inbox-extindex — it's part of the public-inbox user manual and it's literally the vibe you need if you're juggling a million inboxes and want them all searchable. sit down, hydrate, let's go 💖

NAME 🎀

public-inbox-extindex — create and update external search indices

SYNOPSIS 🌸

public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...

public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all

DESCRIPTION 💅

so like, public-inbox-extindex creates and updates an external search and overview database that the read-only public-inbox PSGI (HTTP), NNTP, and IMAP interfaces use. it's giving "one index to rule them all" ✨

it needs either the Search::Xapian XS bindings OR the Xapian SWIG bindings, plus the DBD::SQLite and DBI Perl modules. no modules, no magic, sorry bestie 😭

OPTIONS 👑

fr fr, for all of those up there, just go read public-inbox-index(1) — they work exactly the same way, no cap 💯
  • --all — index all publicinbox entries in PI_CONFIG.

publicinbox entries indexed by public-inbox-extindex can have full Xapian searching abilities with the per-publicinbox indexlevel set to basic and their respective Xapian (xap15 or xapian15) directories removed. for multiple public-inboxes where cross-posting is common, this allows significant space savings on Xapian indices. 🫶 like, imagine deleting a whole chunk of Xapian and still keeping full search — iconic behaviour, honestly.

  • --gc — perform garbage collection instead of indexing. use this if inboxes are removed from the extindex, or if messages are purged or removed from some inboxes. it's the "spring cleaning" option, she's tidy 🧹✨
  • --reindex — forces a re-index of all messages in the extindex. this can be used for in-place upgrades and bugfixes while read-only server processes are utilizing the index.
⚠️ keep in mind this roughly doubles the size of the already-large Xapian database. yes, DOUBLE. your disk said "bestie no" 😭

the extindex locks will be released roughly every 10s to allow public-inbox-mda(1) and public-inbox-watch(1) processes to write to the extindex. thoughtful queen 💅

  • --fast — used with --reindex, it will only look for new and stale entries and not touch already-indexed messages. lowkey the speedrun version 🔥

FILES 📁

public-inbox-extindex-format(5)

CONFIGURATION 🛠️

public-inbox-extindex does not write to the public-inbox-config(5) file — you have to enter it manually, bestie. no autopilot here 🙈

the extindex name of all is a special case which corresponds to indexing --all inboxes. an example for --all is as follows:

[extindex "all"]
    topdir = /path/to/extindex_dir
    url = all
    coderepo = foo
    coderepo = bar

putting an extindex entry in the config allows PublicInbox::WWW. you can have any number of extentry.$NAME sections where $NAME is something other than all to display a union of several inboxes. so many little vibes, one big index 🌈

see public-inbox-config(5) for more details.

ENVIRONMENT 🌍

  • PI_CONFIG — used to override the default ~/.public-inbox/config value.
  • XAPIAN_FLUSH_THRESHOLD — the number of documents to update before committing changes to disk. this environment is handled directly by Xapian, refer to Xapian API documentation for more details.

setting XAPIAN_FLUSH_THRESHOLD or publicinbox.indexBatchSize for a large --reindex may cause public-inbox-mda(1), public-inbox-learn(1) and public-inbox-watch(1) tasks to wait long and unpredictable periods of time during --reindex. 😭 unpredictable wait times? not the moment.

default: none, uses publicinbox.indexBatchSize

UPGRADING 🚀

occasionally, public-inbox will update it's schema version and require a full index by running this command. so like, when the version bumps, you run the thing, you don't ask questions 💖

CONTACT 💌

feedback welcome via plain-text mail to <mailto:meta@public-inbox.org>

the mail archives are hosted at <https://public-inbox.org/meta/> and <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>

copyright all contributors <mailto:meta@public-inbox.org>

license: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>

SEE ALSO 👀

Search::Xapian, DBD::SQLite

and that's the whole page, bestie 🫶 this was public-inbox.git, dated 1993-10-02 — yeah she's got history, she's a legend, she's PUBLIC-INBOX-EXTINDEX(1) and she ate 💅✨

📦 where this page comes from