PUBLIC-INBOX-CINDEX(1)

📦 version 2.1.0, in nixos 26.05, nixpkgs unstable

💖 PUBLIC-INBOX-CINDEX(1) 💖

okay so welcome to the public-inbox user manual page for public-inbox-cindex — yes bestie, we're talking coderepo search indices today and it's lowkey iconic. 💅

✨ NAME ✨

public-inbox-cindex — create and update coderepo search indices. That's the whole name, no cap. 🎀

🦋 SYNOPSIS 🦋

public-inbox-cindex -d EXTDIR [OPTIONS] --join

public-inbox-cindex -d EXTDIR [OPTIONS] --update

public-inbox-cindex [OPTIONS] -g GIT_DIR [-g GIT_DIR]...

🌸 DESCRIPTION 🌸

okay so public-inbox-cindex creates and updates the Xapian search index for git code repository — aka "coderepo" — search. It can associate (fuzzy join) coderepos with Xapian-indexed inboxes to enable blob reconstruction in the "$INBOX_URL/$BLOB_OID/s/" WWW endpoint. It only indexes commit messages and diffs as they would show up in an email. It does not currently index the contents of blobs directly. 😭

Like inbox indices, coderepo indices can either be internal or external to a coderepo. Either way, they're both created and updated through public-inbox-cindex. External indices via -d EXTDIR are recommended for sites hosting multiple coderepos with common history. 🫶

Currently, public-inbox-cindex exists mainly to save WWW admins the trouble of associating hundreds/thousands of inboxes and coderepos with each other via publicinbox.*.coderepo and coderepo.*.dir directives in public-inbox-config(5)). Eventually, it will allow lei-rediff(1) functionality to be ported to the WWW UI and allow searching commits in coderepos directly via WWW interface. 🌈

Once the initial indices are created by public-inbox-cindex, the --update switch will incrementally update them. Slay. ✨

💅 OPTIONS 💅

  • -d EXTDIR — use the given directory as an external index. External indices are generally recommended to internal indices since they do not need write access to any coderepos themselves. They are highly recommended when many coderepos share a common history or if there is an M:N relationship between inboxes and coderepos. 💖
  • -g GIT_DIR, --git-dir=GIT_DIR — when not using -d EXTDIR, the cindex will be written to $GIT_DIR/public-inbox-cindex. May also be combined with -d EXTDIR to index a single (or subset of) git coderepos. May be specified multiple times. 🫶
  • -j JOBS, --jobs=JOBS — influences the number of Xapian indexing shards. If the repo has not been indexed or initialized, JOBS - 1 shards will be created. Default: the number of existing Xapian shards. ✨
  • --join — attempt a fuzzy association of all inboxes and coderepos to enhance the WWW interface. See EXAMPLE below. A C++ compiler, xapian-delve(1) and Xapian development files (e.g. libxapian-dev or xapian-core-devel) will make this operation orders of magnitude faster. This operation should be rerun whenever inboxes or coderepos are added or removed, or when one project merges with another. Web servers running PublicInbox::WWW (e.g. public-inbox-netd(1) or public-inbox-httpd(1)) currently need to be restarted to pick up new (or expire old) associations. 💅
  • --reindex — forces a re-index of all commits. This can be used for in-place upgrades and bugfixes while read-only processes are utilizing the index. 🔥
  • --update, -u — incrementally index all previously-indexed coderepos without checking for new ones. 🦋
  • --prune — unindexes commits which are no longer accessible via git. Use this after git-gc(1) (or git-prune(1)), or if coderepos are removed. 💀
  • --no-fsync — affects the coderepo index the same way it affects inbox indices. See public-inbox-index(1). 🎀
  • --dangerous — affects the coderepo index the same way it affects inbox indices. See public-inbox-index(1). ⚠️
  • --max-size SIZE — affects the coderepo index the same way it affects inbox indices. See public-inbox-index(1). A smaller value of --max-size (e.g. --max-size=10m) is highly recommended to limit memory usage for gigantic commits. 🫶
  • --batch-size SIZE — affects the coderepo index the same way it affects inbox indices. See public-inbox-index(1). ✨
  • --project-list=FILE — the same project list used by cgit, gitweb, grokmirror and public-inbox-clone(1). Requires --project-root=DIRECTORY. 🌸
  • --project-root=DIRECTORY, -r DIRECTORY — specifies the top-level directory for projects in --project-list=FILE. 💜
  • --exclude (GLOB|PATH) — exclude given coderepos when using --project-list=FILE. May be specified multiple times. 🦋

📁 FILES 📁

  • For internal indices, the Xapian DB is stored in $GIT_DIR/public-inbox-cindex.
  • External indices are stored wherever -d EXTDIR points.

⚙️ CONFIGURATION ⚙️

publicinbox.indexMaxSize / publicinbox.indexBatchSize

These configuration knobs affect the coderepo index the same way they affect inbox indices. See public-inbox-index(1). ✨

cindex.$NAME.topdir

Directory where an external coderepo index was created (with -d EXTDIR). 💖

$NAME is the URL prefix (without leading/trailing slashes) for all coderepos in the WWW interface, an empty string ("") is allowed when repos are stored at the toplevel. Combined with cindex.$NAME.localprefix, this allows admins to avoid specifying a separate coderepo.$NICK.dir entry for every coderepo indexed. 🫶

cindex.$NAME.localprefix

The local directory name prefix of all coderepos to be displayed in the WWW interface. This is typically a subdirectory in --project-root=DIRECTORY. 🌸

🌿 ENVIRONMENT 🌿

  • PI_CONFIG — used to override the default ~/.public-inbox/config value. 🫶
  • XAPIAN_FLUSH_THRESHOLD — the number of documents to update before committing changes to disk. This variable is handled directly by Xapian, refer to Xapian API documentation for more details. Use publicinbox.indexBatchSize instead. ⚠️

⬆️ UPGRADING ⬆️

Occasionally, public-inbox will update its schema version and require a full reindex by running this command with --reindex. 💅

🌈 EXAMPLE 🌈

Assuming you have an all extindex for your inboxes and store coderepos in /path/to/repos, the contents of your PI_CONFIG file should include something like this:

        [extindex "all"]
                topdir = /path/to/eidx-all
        [cindex "pub"]
                localprefix = /path/to/repos/pub
                topdir = /path/to/cidx-all

Assuming you're using a cgit/gitweb/grokmirror/public-inbox-clone(1) compatible --project-list, you can periodically use --join when new coderepos are added, deleted, or when one project merges with another:

        public-inbox-cindex -d /path/to/cidx-all --max-size=10m \
                -L medium --join \
                --exclude='**/uninteresting-a.git' \
                --exclude='**/uninteresting-b.git' \
                --project-list=/path/to/repos/projects.list \
                --project-root=/path/to/repos

After this (and restarting the webserver), a project in /path/to/pub/project.git should be visible via https://$HOSTNAME/pub/project.git from a WWW instance and the bottom of the project.git HTML page should display a list of "associated public inboxes". When viewing patch emails in any associated inbox, diff hunk headers (those @@ -123,4 +123,4 @@ lines) will link to $INBOX_URL/$BLOB_OID/s/ URLs which attempt to display the blob (reconstructing blobs from patch emails if necessary). 🦋

Since --join is expensive and (coderepo|inbox) additions/removals are rare, incrementally updating the index can be done more quickly with --update:

        public-inbox-cindex -d /path/to/cidx-all --max-size=10m --update

📬 CONTACT 📬

Feedback welcome via plain-text mail to <mailto:meta@public-inbox.org> 🫶

The mail archives are hosted at <https://public-inbox.org/meta/> and <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>. ✨

Copyright all contributors <mailto:meta@public-inbox.org>.

License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>.

👀 SEE ALSO 👀

k bye bestie! This page came from public-inbox.git, dated 1993-10-02, and it's the PUBLIC-INBOX-CINDEX(1) manual. Stay indexed, stay iconic. 💖✨

📦 where this page comes from