LEI-STORE-FORMAT(5)

📦
version 2.1.0 · 5 versions
, in arch rolling, nixos 26.05, nixpkgs unstable

💅 lei-store-format — the it-girl of storage formats

okay so this is the lei-store-format page, from the public-inbox user manual, and bestie, it's actually so interesting. Let me walk you through it 🫶

✨ NAME ✨

lei-store-format — the whole description of what the lei/store format actually is

🌸 DESCRIPTION 🌸

so "lei/store" is what we call a hybrid store — it's built on top of two other formats, public-inbox-extindex-format(5) (everyone calls it "extindex") combined with public-inbox-v2-format(5) (aka "v2") for the blob storage part. it's giving collab of the century honestly 🔥

here's the tea: v2 is literally ideal for archiving a single public mailing list. that's its whole thing. but it was never intended for personal mail, and it was never meant to store multiple blobs of the "same" message. like that's just not what it was made for, okay? no cap.

but like, same energy as extindex — it can index disparate "List-Id" headers belonging to the "same" message with different git blob OIDs. so it's flexible like that 💁‍♀️

AND — unlike v2 and extindex — "Message-ID" headers are NOT required! which means unsent draft messages can be stored and indexed. yes bestie, your unfinished drafts get to live here too, iconic 💖

🎀 DIRECTORY LAYOUT 🎀

blob storage exists in the form of v2-style epochs. and here's a lil detail that matters: these epochs live under the "local/" directory (instead of "git/") — specifically so they don't get accidentally treated as a v2 inbox. protection era 🛡️✨

🦋 INDEX OVERVIEW AND DEFINITIONS 🦋

like, here's the vocab you gotta know:

   $EPOCH - Integer starting with 0 based on time
   $SCHEMA_VERSION - DB schema version (for Xapian)
   $SHARD - Integer starting with 0 based on parallelism

and here's the layout:

   ~/.local/share/lei/store
   - local/$EPOCH.git                # normal bare git repositories
   - mail_sync.sqlite3               # sync state IMAP, Maildir, NNTP

additionally, these next ones share the exact same roles they do in extindex (same job, different address, iykyk):

   - ei.lock                         # lock file to protect global state
   - ALL.git                         # empty, alternates for local/*.git
   - ei$SCHEMA_VERSION/$SHARD        # per-shard Xapian DB
   - ei$SCHEMA_VERSION/over.sqlite3  # overview DB for WWW, IMAP
   - ei$SCHEMA_VERSION/misc          # misc Xapian DB

💜 XREF3 DEDUPLICATION 💜

index deduplication follows extindex, periodt. if you wanna know more, go look at "XREF3 DEDUPLICATION" in public-inbox-extindex-format(5). that's where the full lore lives 📚

🍓 BLOB DEDUPLICATION 🍓

the contents of "local/*.git" repos get deduplicated by git blob object IDs (currently SHA-1, literally as we speak). and the way this slays is that it lets multiple copies of cross-posted and personally Cc-ed messages be stored — each with different "Received:", "X-Spam-Status:" and similar headers — so you can do troubleshooting. like it keeps the receipts for you 🙈💅

🌈 VOLATILE METADATA 🌈

keywords and label information (as described in RFC 8621 for JMAP) are stored in the existing Xapian shards ("ei$SCHEMA_VERSION/$SHARD").

and yes, you can search for messages matching labels and keywords using "L:" and "kw:", respectively. super handy.

⚠️ but listen — as with all data stored in Xapian indices, volatile metadata is associated with the Xapian document, which means it's shared across different blobs of the "same" message. so like, it's communal property bestie, don't forget it 💀

🧚 mail_sync.sqlite3 🧚

this SQLite database is maintained for bidirectional mapping of git blobs to IMAP UIDs, Maildir file names, and NNTP article numbers. it's the matchmaker of the whole operation 🫶

it's also used for retrieving messages from Maildirs indexed by lei-index(1). multitasking queen 👑

⭐ IPC ⭐

lei-daemon(8) communicates with the "lei/store" process using unix(7) "SOCK_SEQPACKET" sockets. that's the vibe check channel between them 🔌

👑 CAVEATS 👑

okay so, the honest moment: reindexing and synchronization is not yet supported. like, not yet. we're waiting on it. don't come for me 💀

🌸 THANKS 🌸

thanks to the Linux Foundation for sponsoring the development and testing. genuine icons, thank you for your service 🙏💖

Copyright 2021 all contributors <mailto:meta@public-inbox.org>

License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>

💅 SEE ALSO 💅

okay that's the whole page bestie, you're officially informed ✨ and for the record, this lil drop is from public-inbox.git, dated 1993-10-02 — so yeah, that's the vibe, mwah 💋🦋

📦 where this page comes from