PUBLIC-INBOX-INDEX(1)

📦
version 2.0.0~20230714git590023f · 6 versions
, in fedora 43, fedora 44

💖 PUBLIC-INBOX-INDEX 💖

okay bestie, welcome to public-inbox-index(1) — the little command from the public-inbox user manual that builds and feeds the search brains of your inbox. it's giving librarian energy but make it fast. let's go 🧚

✨ NAME ✨

public-inbox-index — create and update search indices

🌸 SYNOPSIS 🌸

public-inbox-index [OPTIONS] INBOX_DIR...

public-inbox-index [OPTIONS] --all

🦋 DESCRIPTION 🦋

so like, public-inbox-index creates and updates the search, overview and NNTP article number database that the read-only public-inbox HTTP and NNTP interfaces use. 💅 rn, this needs the DBD::SQLite and DBI Perl modules. Search::Xapian is optional — it's only there to support the PSGI search interface.

once the initial indices are created by public-inbox-index, besties public-inbox-mda(1) and public-inbox-watch(1) will automatically maintain them for you. no cap. 🫶

running this by hand to update indices is only required if you're relying on git-fetch(1) to mirror an existing public-inbox, or if you're upgrading to a new version of public-inbox using the --reindex option.

having the overview and article number database is essential to running the NNTP interface, and strongly recommended for the HTTP interface, because it provides thread grouping on top of normal search functionality. iconic behavior tbh 👑

🎀 OPTIONS 🎀

see --jobs in public-inbox-init(1) for a full description of sharding.

--jobs=0 is accepted as of public-inbox 1.6.0 to disable parallel indexing regardless of the number of pre-existing shards.

if the inbox has not been indexed or initialized, JOBS - 1 shards will be created (one job is always needed for indexing the overview and article number mapping).

Default: the number of existing Xapian shards

  • -c, --compact — compacts the Xapian DBs after indexing. this is recommended when using --reindex so you don't run out of disk space while indexing multiple inboxes.

while this option takes a negligible amount of time compared to --reindex, it requires temporarily duplicating the entire contents of the Xapian DB. 💀

this switch may be specified twice, in which case compaction happens both before and after indexing to minimize the temporal footprint of the (re)indexing operation.

Available since public-inbox 1.4.0.

  • --reindex — forces a re-index of all messages in the inbox. this can be used for in-place upgrades and bugfixes while NNTP/HTTP server processes are utilizing the index. keep in mind this roughly doubles the size of the already-large Xapian database. 😭 using this with --compact or running public-inbox-compact(1) afterwards is recommended to release free space.

public-inbox protects writes to various indices with flock(2), so it is safe to reindex (and rethread) while public-inbox-watch(1), public-inbox-mda(1) or public-inbox-learn(1) run.

this does not touch the NNTP article number database. it does not affect threading unless --rethread is used.

  • --all — index all inboxes configured in ~/.public-inbox/config. this is an alternative to specifying individual inboxes directories on the command-line.
  • --rethread — regenerate internal THREADID and message thread associations when reindexing.

this fixes some bugs in older versions of public-inbox. while it is possible to use this without --reindex, it makes little sense to do so.

Available in public-inbox 1.6.0+.

Available since public-inbox 1.2.0.

  • --max-size SIZE — sets or overrides publicinbox.indexMaxSize on a per-invocation basis. see publicinbox.indexMaxSize below.

Available since public-inbox 1.5.0.

  • --batch-size SIZE — sets or overrides publicinbox.indexBatchSize on a per-invocation basis. see publicinbox.indexBatchSize below.

when using rotational storage but abundant RAM, using a large value (e.g. 500m) with --sequential-shard can significantly speed up and reduce fragmentation during the initial index and full --reindex invocations (but not incremental updates).

Available in public-inbox 1.6.0+.

  • --no-fsync — disables fsync(2) and fdatasync(2) operations on SQLite and Xapian. this is only effective with Xapian 1.4+. this is primarily intended for systems with low RAM and the small (default) --batch-size=1m. users of large --batch-size may even find disabling fdatasync(2) causes too much dirty data to accumulate, resulting on latency spikes from writeback. 🙈

Available in public-inbox 1.6.0+.

  • --dangerous — speed up initial index by using in-place updates and denying support for concurrent readers. this is only effective with Xapian 1.4+.
okay so it's literally named dangerous bestie, proceed with caution 💅

Available in public-inbox 1.8.0+

  • --sequential-shard — sets or overrides publicinbox.indexSequentialShard on a per-invocation basis. see publicinbox.indexSequentialShard below.

Available in public-inbox 1.6.0+.

  • --skip-docdata — stop storing document data in Xapian on an existing inbox.

see --skip-docdata in public-inbox-init(1) for description and caveats.

Available in public-inbox 1.6.0+.

  • -E EXTINDEX, --update-extindex=EXTINDEX — update the given external index (public-inbox-extindex-format(5)). either the configured section name (e.g. all) or a directory name may be specified.

defaults to all if [extindex "all"] is configured, otherwise no external indices are updated.

may be specified multiple times in rare cases where multiple external indices are configured.

  • --no-update-extindex — do not update the all external index by default. this negates all uses of -E / --update-extindex= on the command-line.
  • --since=DATESTRING, --after=DATESTRING, --until=DATESTRING, --before=DATESTRING — passed directly to git-log(1) to limit changes for --reindex

🌈 FILES 🌈

for v1 (ssoma) repositories described in public-inbox-v1-format(5). all public-inbox-specific files are contained within the $GIT_DIR/public-inbox/ directory.

v2 inboxes are described in public-inbox-v2-format(5).

💜 CONFIGURATION 💜

publicinbox.indexMaxSize

  • prevents indexing of messages larger than the specified size value. a single suffix modifier of k, m or g is supported, thus the value of 1m prevents indexing of messages larger than one megabyte.
  • this is useful for avoiding memory exhaustion in mirrors via git. it does not prevent public-inbox-mda(1) or public-inbox-watch(1) from importing (and indexing) a message.
  • this option is only available in public-inbox 1.5 or later.
  • Default: none

publicinbox.indexBatchSize

  • flushes changes to the filesystem and releases locks after indexing the given number of bytes. the default value of 1m (one megabyte) is low to minimize memory use and reduce contention with parallel invocations of public-inbox-mda(1), public-inbox-learn(1), and public-inbox-watch(1).
  • increase this value on powerful systems to improve throughput at the expense of memory use. the reduction of lock granularity may not be noticeable on fast systems. with SSDs, values above 4m have little benefit.
  • for public-inbox-v2-format(5) inboxes, this value is multiplied by the number of Xapian shards. thus a typical v2 inbox with 3 shards will flush every 3 megabytes by default unless parallelism is disabled via --sequential-shard or --jobs=0.
  • this influences memory usage of Xapian, but it is not exact. the actual memory used by Xapian and Perl has been observed in excess of 10x this value. 😭
  • this option is available in public-inbox 1.6 or later. public-inbox 1.5 and earlier used the current default, 1m.
  • Default: 1m (one megabyte)

publicinbox.indexSequentialShard

  • for public-inbox-v2-format(5) inboxes, setting this to true allows indexing Xapian shards in multiple passes. this speeds up indexing on rotational storage with high seek latency by allowing individual shards to fit into the kernel page cache.
  • using a higher-than-normal number of --jobs with public-inbox-init(1) may be required to ensure individual shards are small enough to fit into cache.
  • > Warning: interrupting public-inbox-index(1) while this option is in use may leave the search indices out-of-date with respect to SQLite databases. WWW and IMAP users may notice incomplete search results, but it is otherwise non-fatal. using --reindex will bring everything back up-to-date. 🫶
  • Available in public-inbox 1.6.0+.
  • this is ignored on public-inbox-v1-format(5) inboxes.
  • Default: false, shards are indexed in parallel

publicinbox.<name>.indexSequentialShard

  • identical to publicinbox.indexSequentialShard, but only affects the inbox matching <name>.

⭐ ENVIRONMENT ⭐

  • PI_CONFIG — used to override the default ~/.public-inbox/config value.
  • XAPIAN_FLUSH_THRESHOLD — the number of documents to update before committing changes to disk. this environment is handled directly by Xapian, refer to Xapian API documentation for more details.

for public-inbox 1.6 and later, use publicinbox.indexBatchSize instead.

setting XAPIAN_FLUSH_THRESHOLD or publicinbox.indexBatchSize for a large --reindex may cause public-inbox-mda(1), public-inbox-learn(1) and public-inbox-watch(1) tasks to wait long and unpredictable periods of time during --reindex. 😭

Default: none, uses publicinbox.indexBatchSize

🦋 UPGRADING 🦋

occasionally, public-inbox will update it's schema version and require a full index by running this command. bestie you just gotta do it, no shortcuts 💅

🍓 CONTACT 🍓

feedback welcome via plain-text mail to <mailto:meta@public-inbox.org>

the mail archives are hosted at <https://public-inbox.org/meta/> and <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>

copyright all contributors <mailto:meta@public-inbox.org>

license: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>

🌸 SEE ALSO 🌸

Search::Xapian, DBD::SQLite, public-inbox-extindex-format(5)

and that's the whole page, bestie!! 🎀 this came from public-inbox.git, dated 1993-10-02 — mwah, go index something iconic. slay 💖✨

📦 where this page comes from