PUBLIC-INBOX-V1-FORMAT(5)

💖 public-inbox-v1-format — the git repo & tree situation (aka "ssoma") 💖

ok so bestie, you found the <cyan>public-inbox-v1-format</cyan> page from the public-inbox user manual. sit down, hydrate, we're gonna talk about how v1 repos are built. 🫶

---

✨ NAME ✨

public-inbox-v1-format — git repository and tree description (aka "ssoma")

---

🌸 DESCRIPTION 🌸

WARNING: this does NOT describe the scalable v2 format used by public-inbox. Use of ssoma is not recommended for new installations due to scalability problems. like, don't do it, babe. 😭

so. ssoma uses a git repository to store each email as a git blob. the tree filename of that blob is based on the SHA1 hexdigest of the first Message-ID header. a commit gets made for each message delivered, and that commit's SHA-1 identifier is what ssoma clients use to track their synchronization state. that's the whole system, it's giving very "git as a mail spool" and honestly? iconic for its time. 💅

---

🦋 PATHNAMES IN TREES 🦋

ok so a Message-ID can be extremely long and can also contain slashes, so using them directly as a path name is... challenging. not the vibe. 🙈

instead we use the SHA-1 hexdigest of the Message-ID (excluding the leading < and trailing >) to generate the path name. and yes — leading and trailing white space in the Message-ID header is ignored for hashing. no cap.

a message with Message-ID of:

<20131106023245.GA20224@dcvr.yhbt.net>

would be stored as:

f2/8c6cfd2b0a65f994c3e1be266105413b3d3f63

so it's super easy to look up the contents of a message matching a given Message-ID. lookup slays. ⭐

---

💀 MESSAGE-ID CONFLICTS 💀

public-inbox v1 repositories currently do not resolve conflicting Message-IDs or messages with multiple Message-IDs. they just... don't. it's a known thing. 🤷♀️

---

🎀 HEADERS 🎀

the Message-ID header is required. periodt.

the Bytes, Lines and Content-Length headers are stripped and not allowed — they can interfere with further processing. bye. 👋

and when you're using ssoma with public-inbox-mda, the Status mbox header is also stripped, because that header makes no sense in a public archive. correct. 🫡

---

🔒 LOCKING 🔒

flock(2) locking exclusively locks the empty $GIT_DIR/ssoma.lock file for all non-atomic operations. that's the lock, that's the whole lock. 💜

---

🔥 EXAMPLE INPUT FLOW (SERVER-SIDE MDA) 🔥

1. Message is delivered to a mail transport agent (MTA)

1a. (optional) reject/discard spam, this should run before ssoma-mda

1b. (optional) reject/strip unwanted attachments

ssoma-mda handles all steps once invoked. she's a multitasker, she ate. 👑

2. Mail transport agent invokes ssoma-mda
3. reads message via stdin, extracting Message-ID
4. acquires exclusive flock lock on $GIT_DIR/ssoma.lock
5. creates or updates the blob of associated 2/38 SHA-1 path
6. updates the index and commits
7. releases $GIT_DIR/ssoma.lock

ssoma-mda can also be used as an inotify(7) trigger to monitor maildirs, and the ability to monitor IMAP mailboxes using IDLE will be available in the future. future plans, we love to see it. 🌈

---

🌟 GIT REPOSITORIES (SERVERS) 🌟

ssoma uses bare git repositories on both servers and clients. no working tree, no drama.

using the git-init(1) command with --bare is the recommended method of creating a git repository on a server:

git init --bare /path/to/wherever/you/want.git

there are no standardized paths for servers — administrators make all the choices regarding git repository locations. total main character energy, honestly. ✨

special files in $GIT_DIR on the server:

• $GIT_DIR/ssoma.lock — an empty file for flock(2) locking. this is necessary to ensure the index and commits are updated consistently and that multiple processes running MDA do not step on each other. teamwork makes the dream work. 🤝
• $GIT_DIR/public-inbox/msgmap.sqlite3 — a SQLite3 database maintaining a stable mapping of Message-IDs to NNTP article numbers. used by public-inbox-nntpd(1) and created and updated by public-inbox-index(1).

users of the PublicInbox::WWW interface will find it useful for attempting recovery from copy-paste truncations of URLs containing long Message-IDs.

automatically updated by public-inbox-mda(1), public-inbox-learn(1) and public-inbox-watch(1).

losing or damaging this file will cause synchronization problems for NNTP clients. this file is expected to be stable and require no updates to its schema.

requires DBD::SQLite.

• $GIT_DIR/public-inbox/xapian$N/ — Xapian database for search indices in the PSGI web UI.

$N is the value of PublicInbox::Search::SCHEMA_VERSION, and installations may have parallel versions on disk during upgrades or to roll-back upgrades.

this is created and updated by public-inbox-index(1).

automatically updated by public-inbox-mda(1), public-inbox-learn(1) and public-inbox-watch(1).

this directory can always be regenerated with public-inbox-index(1). if lost or damaged, there is no need to back it up unless the CPU/memory cost of regenerating it outweighs the storage/transfer cost. free yourself bestie, let it go. 🧚

since SCHEMA_VERSION 15 and the development of the v2 format, the "overview" DB also exists in the xapian directory for v1 repositories. see "OVERVIEW DB" in public-inbox-v2-format(5)

our use of the "OVERVIEW DB" requires Xapian document IDs to remain stable. using public-inbox-compact(1) and public-inbox-xcpdb(1) wrappers are recommended over tools provided by Xapian.

this directory is large, often two to three times the size of the objects stored in a packed git repository. she's thick and she's valid. 💅

• $GIT_DIR/ssoma.index — this file is no longer used or created by public-inbox, but it is updated if it exists, to remain compatible with ssoma installations.

a git index file used for MDA updates. the normal git index (in $GIT_DIR/index) is not used at all, as there is typically no working tree.

each client $GIT_DIR may have multiple mbox/maildir/command targets. it is possible for a client to extract the mail stored in the git repository to multiple mboxes for compatibility with a variety of different tools. flexible queen. 🌸

---

🙈 CAVEATS 🙈

• it is NOT recommended to check out the working directory of a git. there may be many files. like, so many. don't.
• it is impossible to completely expunge messages, even spam, as git retains full history. projects may (with adequate notice) cycle to new repositories/branches with history cleaned up via git-filter-repo(1) or git-filter-branch(1). this is up to the administrators. once it's in git it's in git, bestie. 💀

---

💜 COPYRIGHT 💜

Copyright 2013-2021 all contributors <mailto:meta@public-inbox.org>

License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>

---

🫶 SEE ALSO 🫶

• gitrepository-layout(5)
• ssoma(1)

---

and that's the whole page, bestie!! 🎀 thanks for reading, mwah. this was public-inbox.git, dated 1993-10-02, from the PUBLIC-INBOX-V1-FORMAT(5) manual. now go outside and touch grass, or at least touch a bare git repo. 💖✨