PUBLIC-INB...DEX-FORMAT(5)

💖 public-inbox-extindex-format(5) 💖

okay bestie, welcome to the extindex format page — it's manual section 5, so yes we're in the "file formats" girlie zone, and this one is genuinely iconic. let's get into it 🫶

---

✨ NAME ✨

public-inbox-extindex-format — the whole vibe of the external index format, explained.

---

🌸 DESCRIPTION 🌸

so the extindex is basically the glow-up evolution of the per-inbox SQLite and Xapian indices that public-inbox-v2-format(5) and public-inbox-v1-format(5) use. it's index-only, no cap. and it exists for two reasons, both of which are slay:

• so you can search across multiple inboxes at once, and
• so you don't waste index space when messages get cross-posted to several existing inboxes. because bestie, duplicating that is not the move.

it transparently indexes messages across any combination of v1 and v2 inboxes, plus data about the inboxes themselves. it just does it. no drama.

---

🎀 DIRECTORY LAYOUT 🎀

so it's inspired by v2, but — and this is important — there is no git blob storage and no msgmap.sqlite3 DB. we left those at home.

instead there's an ALL.git (all caps, do not lowercase her) git repo, and it treats every indexed v1 inbox or v2 epoch as a git alternate. cute.

and just like with v2 inboxes, it uses over.sqlite3 and Xapian shards for WWW and IMAP use. on top of that, several exclusive new tables get added to handle XREF3 DEDUPLICATION and metadata.

💀 and unlike v1 and v2 inboxes, it is NOT designed to map to an NNTP newsgroup. so it lacks msgmap.sqlite3, which is the thing that enforces NNTP's unique Message-ID requirement. different species, honestly.

🌈 INDEX OVERVIEW AND DEFINITIONS

here's the map, bestie. learn her:

$SCHEMA_VERSION - DB schema version (for Xapian)
$SHARD - Integer starting with 0 based on parallelism

foo/                              # "foo" is the name of the index
- ei.lock                         # lock file to protect global state
- ALL.git                         # empty, alternates for inboxes
- ei$SCHEMA_VERSION/$SHARD        # per-shard Xapian DB
- ei$SCHEMA_VERSION/over.sqlite3  # overview DB for WWW, IMAP
- ei$SCHEMA_VERSION/misc          # misc Xapian DB

and the file and directory names are intentionally different from the analogous v2 names — on purpose, periodt — so that extindex and v2 inboxes can be told apart easily. no mistaken identity in this house.

🦋 XREF3 DEDUPLICATION

okay so context: in the big Linux kernel development community, cross-posted messages are literally the norm, and Xapian indices are the primary consumer of storage. so it just makes sense to deduplicate the indexing as much as possible. we're being efficient, we're being mindful, we're being thrifty 💅

the internal storage format is based on the NNTP Xref tuple, but with a lil third element added: the git blob OID. so the triple in string form looks like:

$NEWSGROUP_NAME:$ARTICLE_NUM:$OID

and if there's no newsgroup configured for an inbox? then the inboxdir of that inbox gets used instead. simple as that.

this data lives in the xref3 table of over.sqlite3. she has a home. 🏠

🧚 misc XAPIAN DB

besides the numeric Xapian shards for indexing messages, there's a new, in-development Xapian index for storing data about inboxes themselves and other non-message data.

why do we care? because this index lets us speed up operations involving hundreds or thousands of inboxes. it's giving scalability. it's giving main character energy. 👑

---

🔥 BENEFITS 🔥

on top of the cross-inbox search capabilities (already amazing), it can also replace per-inbox Xapian shards — but not per-inbox over.sqlite3, okay, don't get it twisted.

and what does that get you?

• less disk space used
• fewer open file handles
• less associated memory use

we love a resource-efficient queen. 🌸

---

⚠️ CAVEATS ⚠️

listen. relocating v1 and v2 inboxes on the filesystem will require extindex to be garbage-collected and/or reindexed. it's not optional, bestie. plan for it.

and here's the pro tip that saves you pain later: configuring and maintaining stable newsgroup names before any messages are indexed from every inbox can avoid expensive reindexing entirely, and let you rely exclusively on GC. future you will be so grateful. 🙈

---

🔒 LOCKING 🔒

flock(2) locking exclusively locks the empty ei.lock file for all non-atomic operations. that's the whole lock situation. she's exclusive, she's simple, she's doing her job. 💜

---

💖 THANKS 💖

thanks to the Linux Foundation for sponsoring the development and testing. absolute patrons, honestly. 🫶

---

©️ COPYRIGHT ©️

Copyright 2020-2021 all contributors <mailto:meta@public-inbox.org>

License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>

---

👀 SEE ALSO 👀

• public-inbox-v2-format(5)

---

and that's the whole page, bestie!! this one came from public-inbox.git, dated 1993-10-02 — yes, the date on this footer is genuinely older than the internet's entire personality, don't ask, just accept it ✨ mwah, go read your indices 💅💖