PUBLIC-INBOX-XCPDB(1)

📦
version 2.0.0~20230714git590023f · 6 versions
, in fedora 43, fedora 44

💖 public-inbox-xcpdb 💖

okay bestie, you found the page for public-inbox-xcpdb, straight out of the public-inbox user manual ✨ it's the little tool that upgrades your Xapian DB formats, and honestly? it's giving "rare special occasion" energy. let's get into it 🫶

🌸 NAME 🌸

public-inbox-xcpdb — upgrade Xapian DB formats 💅

✨ SYNOPSIS ✨

public-inbox-xcpdb [OPTIONS] INBOX_DIR

public-inbox-xcpdb [OPTIONS] --all

🦋 DESCRIPTION 🦋

so, public-inbox-xcpdb is lowkey the cousin of copydatabase(1) — it's here to upgrade you to the latest database format that Xapian supports (think "glass" or "honey" ✨) — but with a glow up: it's designed to tolerate and accept parallel Xapian database modifications coming from public-inbox-watch(1), public-inbox-mda(1), public-inbox-learn(1), and public-inbox-index(1). like, it doesn't freak out when other processes are touching the DB at the same time. main character behavior honestly 👑

ngl though, this command is rarely used — because Xapian DB formats rarely change. it's the "emergency cute outfit in the back of the closet" of public-inbox tools 🎀

💅 OPTIONS 💅

  • --all — copies every inbox configured in ~/.public-inbox/config. it's the alternative to listing individual inbox directories on the command-line, so you don't have to type them all out like a menace 💖
  • -c, --compact — on top of doing the copy operation, it runs xapian-compact(1) on each Xapian shard after copying but before finalizing it. and here's the tea: compared to the cost of copying a Xapian database, compacting one only takes around 5% of the time required to copy. so cheap, so worth it 😌

compared to public-inbox-compact(1), this option is preferable for gigantic inboxes — because the coarse-grained lock currently required for public-inbox-compact(1) can make the compaction take hours at-a-time 😭 yeah. hours. no cap.

  • -R N, --reshard=N — reshard the Xapian database on a v2 inbox to "N" shards. since xapian-compact(1) is not suitable for merging, you can rely on this switch to reshard the existing Xapian database(s) to any positive value of "N" 🧚

this is useful in case the Xapian DB was created with too few or too many shards given the capabilities of the current hardware — aka your hardware changed and the old shard count is not it anymore 💀

available in public-inbox 1.6.0+.

  • --sequential-shard — copy each shard sequentially, ignoring --jobs. this also affects indexing done at the end of a run. one at a time, patient queen 🌸
  • --batch-size=BYTES, --max-size=BYTES — see public-inbox-index(1) for a description of these options. 👀

these indexing options affect indexing at the end of a run. public-inbox-xcpdb may run in parallel with public-inbox-index(1), and public-inbox-xcpdb needs to reindex changes made to the old Xapian DBs by public-inbox-index(1) while it was running. so yeah, the parallel stuff is handled, but it's handled deliberately 🫶

🌈 ENVIRONMENT 🌈

  • PI_CONFIG — the default config file, normally ~/.public-inbox/config. see public-inbox-config(5)
  • XAPIAN_FLUSH_THRESHOLD — the number of documents to update before committing changes to disk. this environment is handled directly by Xapian, so refer to the Xapian API documentation for more details 📚

Default: 10000

👑 UPGRADING 👑

this tool is intended for admins upgrading Xapian search databases used by public-inbox — NOT users upgrading public-inbox itself. two different quests, bestie, don't mix them up 💅

in particular, it DOES NOT upgrade the schema used by the PSGI search interface (see public-inbox-index(1)). that's a different job entirely, ok? 🙈

⚠️ LIMITATIONS ⚠️

do NOT use public-inbox-purge(1) or public-inbox-edit(1) while this is running — old (purged or edited data) may show up. like, the ghost of data past. we do not want that 💀

normal invocations of public-inbox-index(1) can safely run while this is running, too. however, reindexing via the --reindex in public-inbox-index(1) switch will be a waste of computing resources — it's just burning CPU for no reason, and that's not slay, that's just wasteful 😭

💌 CONTACT 💌

feedback welcome via plain-text mail to <mailto:meta@public-inbox.org> 🫶

the mail archives are hosted at:

Copyright 2019-2021 all contributors <mailto:meta@public-inbox.org>

License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>

🔥 SEE ALSO 🔥

copydatabase(1), xapian-compact(1), public-inbox-index(1)

and that's the whole page, bestie!! ✨ this one came from public-inbox.git, dated 1993-10-02, and it lives in the manual as PUBLIC-INBOX-XCPDB(1) — so if you ever need to reshard or upgrade your Xapian DB, you know exactly where to look 💖 mwah, go forth and copy responsibly 🦋

📦 where this page comes from