SQUATTER(8)

📦
version 3.12.3 · 6 versions
, in ubuntu stonking

💖 SQUATTER(8) 💖

squatter — straight outta the Cyrus IMAP docs, section 8 (the admin commands, bestie 💅)

✨ NAME ✨

squatter — Cyrus IMAP documentation

Create SQUAT and Xapian indexes for mailboxes 🦋

🦋 SYNOPSIS 🦋

general:
squatter [ -C config-file ] [mode] [options] [source]

i.e.:
squatter [ -C config-file ] [ -v ] [ -a ] [ -S seconds ] [ -Z ]
squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] mailbox...
squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] -u user...
squatter [ -C config-file ] [ -v ] [ -a ] -R [ -n channel ] [ -d ] [ -S seconds ] [ -Z ]
squatter [ -C config-file ] [ -v ] [ -a ] -f synclogfile [ -S seconds ] [ -Z ]
squatter [ -C config-file ] [ -v ] -t srctier(s)... -z desttier [ -B ] [ -F ] [ -U ] [ -T reindextiers ] [ -X ] [ -o ] [ -S seconds ] [ -u user... ]

🌸 DESCRIPTION 🌸

NOTE: the name "squatter" once referred both to the SQUAT indexing engine and to the command used to create indexes. Now that Cyrus supports more than one index type — SQUAT and Xapian, as of this writing — the name "squatter" refers to the command used to control index creation. The terms "SQUAT" or "SQUAT index(es)" refers to the indexes used by the older SQUAT indexing engine. Post v3 the search_engine setting in imapd.conf determines which search engine is used. 💅

Okay so real talk, squatter builds a brand-new text index for one or more IMAP mailboxes. The index is literally a unified index of all of the header and body text of each message in a given mailbox. And the whole point? This index is used to significantly reduce IMAP SEARCH times on a mailbox. Slay. 🔥

mode is one of indexer, search, rolling, synclog, compact or audit. Pick your fighter. ✨

By default, squatter creates an index of ALL messages in the mailbox — not just those since the last time that it was run. The -i option is used to select incremental updates instead. And heads up bestie: any messages appended to the mailbox after squatter is run will NOT be included in the index. To include new messages in the index, squatter must be run again, or on a regular basis via crontab, an entry in the EVENTS section of cyrus.conf(5), or you can just use rolling mode (-R). 😌

Here's the synopsis breakdown, one at a time, fr fr:

  • In the first synopsis, squatter indexes all mailboxes. Every single one. Iconic.
  • In the second synopsis, squatter indexes the specified mailbox(es). The mailboxes are space-separated. 🎀
  • In the third synopsis, squatter indexes the specified user(s) mailbox(es). 🫶

For the latter two index modes (mailbox, user) one may optionally specify -r to recurse from the specified start, or -a to limit action only to mailboxes which have the shared /vendor/cmu/cyrus-imapd/squat annotation set to "true". No cap. ✅

In the fourth synopsis, squatter runs in rolling mode. In this mode squatter backgrounds itself and runs as a daemon (unless -d is set), listening to a sync log channel chosen using the -n option, and set up using the sync_log_channels setting in imapd.conf(5). Very soon after messages are delivered or uploaded to mailboxes, squatter will incrementally index the affected mailbox (see notes, below). It's giving always-on main character energy. 👑

In the fifth synopsis, squatter reads a single sync log file and performs incremental indexing on the mailbox(es) listed therein. This is sometimes useful for cleaning up after problems with rolling mode. We don't talk about what happened. 🙈

In the sixth synopsis, squatter will compact indices from srctier(s) to desttier, optionally reindexing (-X) or filtering expunged records (-F) in the process. The optional -T flag may be used to specify members of srctiers which must be reindexed. These files are eventually copied with rsync -a and then removed by rm. 🗑️

Watch out: rsync can increase the load average of the system, especially when the temporary directory is on tmpfs. To throttle rsync it is possible to modify the call in imap/search_xapian.c and pass --bwlimit=<number> as further parameter. 💀

The -o flag may be used to direct that a single index be copied, rather than compacted, from srctier to desttier. The -u flag may be used to restrict operation to the specified user(s). 💖

For all modes, the -S option may be specified, causing squatter to pause seconds seconds after each mailbox, to smooth loads. Self-care for your server, honestly. 🌸

When using the Xapian engine the -Z option may be specified, for the indexing modes. This tells squatter to consult the Xapian internally indexed GUIDs, rather than relying on what's stored in cyrus.indexed.db, allowing for recovery from broken cyrus.indexed.db at the sacrifice of efficiency. (Trading efficiency for survival? That's the vibe check.) ⭐

NOTE: Incremental updates are very inefficient with the SQUAT search engine. If using SQUAT for large and active mailboxes, you should run squatter periodically as an EVENT in cyrus.conf(5). Do not skip this, bestie, I'm begging. 😭
NOTE: Messages and mailboxes that have not been indexed CAN still be SEARCHed, just not as quickly as those with an index. So it's not the end of the world, it's just... not as cute. 🐢

Finally, squatter reads its configuration options out of the imapd.conf(5) file unless specified otherwise by -C. 🦋

✨ OPTIONS ✨

okay bestie, this is the part where we go flag by flag, no skipping, no skimming, we read ALL of them 💅

  • -C config-file — use config-file instead of the default imapd.conf(5). yes queen, custom config file, live your truth 🎀
  • -a, --squat-annot — only build indexes for mailboxes that have the shared /vendor/cmu/cyrus-imapd/squat annotation set to "true".

heads up: the value of the /vendor/cmu/cyrus-imapd/squat annotation gets inherited by every child of that mailbox. so you can index (or not index) a whole mailbox tree by setting one single annotation on the root of that tree to "true" (or "false"). literally one annotation, whole tree handled, iconic 🌈

and if a mailbox does not have a /vendor/cmu/cyrus-imapd/squat annotation on it (or doesn't inherit one), then it is not indexed. so basically the implicit value of /vendor/cmu/cyrus-imapd/squat is "false". no annotation, no index, that's the vibe 🙈

  • -A, --audit — audits the mailboxes you specified (or all of them) and reports any unindexed messages. 👀 this feature is only available on the master branch.
  • -d, --nodaemon — in rolling mode, don't background, and do emit log messages on standard error. chef's kiss for debugging 🐛 this feature was introduced in version 3.0.
  • -B, --skip-locked — in compact mode, use a non-blocking lock to start, and skip any users who have their xapianactive file locked at that moment (i.e. another reindex task is on it). no waiting around, we move 🏃♀️ this feature is only available on the master branch.
  • -F, --filter — in compact mode, filter the resulting database so it only includes messages that are not expunged in mailboxes with an existing name/uidvalidity. this feature was introduced in version 3.0.
  • -f synclogfile, --synclog=synclogfile — read synclogfile, incrementally index all the mailboxes listed in it, then exit. read, index, bye 👋 this feature was introduced in version 3.0.
  • -h, --help — display this usage information. classic, we love a help flag 🫶
  • -i, --incremental — incremental updates, but only where indexes already exist. no starting from scratch here, we're just topping up ✨
  • -N name, --name=name — only index mailboxes beginning with name while iterating through the mailbox list derived from your other options. filters within the list, not instead of it 💖
  • -n channel, --channel=channel — in rolling mode, specify the name of the sync log channel that squatter will listen to. the default is "squatter". this channel must be defined in imapd.conf(5) before you use it, no cap. this feature was introduced in version 3.0.
  • -o, --copydb — in compact mode, if only one source database is selected, just copy it to the destination instead of compacting. work smarter bestie 💅 this feature was introduced in version 3.0.
  • -p, --allow-partials — when indexing, allow messages to be partially indexed. this can happen if attachment indexing is enabled but indexing failed for one or more attachment body parts. if this flag is set, the message is partially indexed and squatter continues. otherwise squatter aborts with an error 😭 also see -P. Xapian only. this feature is only available on the master branch.
  • -P, --reindex-partials — when reindexing, attempt to reindex any partially indexed messages (see -p). setting this flag implies -Z. Xapian only. this feature is only available on the master branch.
  • -L, --reindex-minlevel=level — when reindexing, index all messages that have an index level less than level. rn Cyrus only supports two index levels: a message for which attachment indexing was never attempted has index level 1. a message that has indexed attachments, or does not contain attachments, has index level 3. so running squatter with minlevel set to 3 will make it try to reindex all messages for which attachment indexing never was attempted. future Cyrus versions may introduce additional levels. setting this flag implies -Z. Xapian only. this feature is only available on the master branch.
  • -R, --rolling — run in rolling mode: squatter runs as a daemon listening to a sync log channel, continuously incrementally indexing mailboxes. see also -d and -n. main character daemon energy 👑 this feature was introduced in version 3.0.
  • -r, --recursive — recursively create indexes for all sub-mailboxes of the user, mailboxes or mailbox prefixes you gave as arguments. all the way down, we go deep 🌀
  • -s delta, --squat-skip=delta — skip mailboxes that haven't been modified since the last index. how? by comparing the last modification time of a mailbox to the last time the squat index of that mailbox got updated. if the mailbox modification time plus delta is less than the squat index modification time, the mailbox gets skipped. delta is defined in seconds and must be greater than or equal to zero. the historical default delta was 60, and honestly that's still a great general choice, but for technical reasons it now must be specified explicitly. Squat only.
  • -S seconds, --sleep=seconds — after processing each mailbox, sleep for seconds before continuing. great for some load balancing 🌸 accepts fractional amounts. this feature was introduced in version 3.0.
  • -T reindextiers, --reindex-tier=reindextiers — in compact mode, a comma-separated subset of the source tiers (see -t) to be reindexed. similar to -X but lets you limit which tiers get reindexed. this feature was introduced in version 3.0.
  • -t srctiers, --srctier=srctiers — in compact mode, the comma-separated source tier(s) for the compacted indices. at least one source tier must be specified in compact mode. Xapian only. this feature was introduced in version 3.0.
  • -u name, --user=name — extra options refer to usernames (e.g. foo@bar.com) rather than mailbox names. usernames are space-separated. this feature was introduced in version 3.0.
  • -U, --only-upgrade — in compact mode, only compact if re-indexing. Xapian only. this feature is only available on the master branch.
  • -v, --verbose — increase the verbosity of progress/status messages. sometimes extra messages get emitted on the terminal with this option, and the messages are unconditionally sent to syslog. sometimes messages are sent to syslog only if -v is provided. in rolling and synclog modes, -vv sends even more messages to syslog. more words, more insight, feed me information 🔥
  • -X, --reindex — reindex all the messages before compacting. this mode reads all the lists of messages indexed by the listed tiers, and re-indexes them into a temporary database before compacting that into place. Xapian only. this feature was introduced in version 3.0.
  • -z desttier, --compact=desttier — in compact mode, the destination tier for the compacted indices. this must be specified in compact mode. Xapian only. this feature was introduced in version 3.0.
  • -Z, --internalindex — when indexing messages, use the Xapian internal cyrusid rather than referencing the ranges of already indexed messages to figure out if a particular message is indexed. super useful if the ranges get out of sync with the actual messages (e.g. if files on a tier are lost) 💀 Xapian only. this feature is only available on the master branch.

🌈 EXAMPLES 🌈

squatter is typically deployed via entries in cyrus.conf(5), in either the DAEMON or EVENTS sections.

for the older SQUAT search engine — which offers poor performance in rolling mode (-R), ngl — we recommend triggering periodic runs via entries in the EVENTS section, like this:

sample entries from the EVENTS section of cyrus.conf(5) for periodic squatter runs:

EVENTS {
    # reindex changed mailboxes (fulltext) approximately every three hours
    squatter1   cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180

    # reindex all mailboxes (fulltext) daily
    squattera   cmd="/usr/lib/cyrus/bin/squatter" at=0117
}

for the newer Xapian search engine, and with sufficiently fast storage, rolling mode (-R) offers advantages. using rolling mode requires that squatter be invoked in the DAEMON section.

sample entries for the DAEMON section of cyrus.conf(5) for rolling squatter operation:

DAEMON {
  # run a rolling squatter using the default sync_log channel "squatter"
  squatter cmd="squatter -R"

  # run a rolling squatter using a specific sync_log channel
  squatter cmd="squatter -R -n indexer"
}
NOTE: when you're using -R rolling mode, you MUST enable sync_log operation in imapd.conf(5) via the sync_log: on setting, and you MUST define a sync_log channel via the sync_log_channels: setting. if you're also using replication, you must either explicitly specify your replication sync_log channel via the sync_log_channels directive with a name, or specify the default empty name with "" (the two-character string U+22 U+22). [please see imapd.conf(5) for details]. fr fr, don't skip this one 🫠
NOTE: when configuring rolling search indexing on a replica, you gotta think about whether sync_logs will be written at all. in that case, please consider the setting sync_log_unsuppressable_channels to make sure that the sync_log channel your squatter instance depends on keeps getting written. see imapd.conf(5) for details. 🔍
NOTE: when using the Xapian search engine, you must define various settings in imapd.conf(5). please read all relevant Xapian documentation in this release before using Xapian. read first, then index bestie 📚

[NB: More examples needed] 💀 yeah, they said it, not me.

📜 HISTORY 📜

support for additional search engines was added in version 3.0.

the following command-line switches were added in version 3.0:

-F -R -X -d -f -o -u

the following command-line settings were added in version 3.0:

-S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier

📂 FILES 📂

  • /etc/imapd.conf, /etc/cyrus.conf

👀 SEE ALSO 👀

👑 AUTHOR 👑

the Cyrus Team, Nic Bernstein (Onlight)

1993–2026, The Cyrus Team

and that's the whole page bestie, we made it 🎀 this was version 3.12.3, dated Jul 15, 2026, from the manual section SQUATTER(8). go index something and slay ✨💖

📦 where this page comes from