SPAMORACLE(1)

💖 SPAMORACLE 💖

okay bestie, gather round, because spamoracle is literally the manual page we're doing today — it's in the General Commands Manual and it's a spam classification tool and honestly? it's giving main character energy in your mail pipeline. 🦋

---

✨ NAME ✨

spamoracle — a spam classification tool 🌸

---

✨ SYNOPSIS ✨

no cap, here's every single way you can call it. read them like a menu, they're all real:

spamoracle [-config conf] [-f database] mark [ mailbox ...  ]

spamoracle [-config conf] [-f database] add [-v] -spam spambox ...
-good goodbox ...

spamoracle [-config conf] [-f database] test [-min prob] [-max prob] [
mailbox ...  ]

spamoracle [-config conf] [-f database] stat [ mailbox ...  ]

spamoracle [-config conf] [-f database] list regexp ...

spamoracle [-config conf] [-f database] backup > backupfile

spamoracle [-config conf] [-f database] restore < backupfile

spamoracle [-config conf] [-f database] upgrade

spamoracle [-config conf] [-f database] words [ mailbox ...  ]

---

✨ DESCRIPTION ✨

SpamOracle is a tool that helps you detect and filter away "spam" (that unsolicited commercial e-mail that keeps showing up like an ex who won't take the hint 🙈). The way it does this is statistical analysis of the words that appear in your e-mail — it compares how often words show up against the frequencies found in a corpus you provide, made of known spam and known legitimate e-mail. The classification algorithm is based on Bayes' formula, and it's described in Paul Graham's paper, A plan for spam, at http://www.paulgraham.com/spam.html. 📚

This program is designed to work in conjunction with procmail(1). The result of the analysis gets output as an additional message header, X-Spam:, followed by yes, no or unknown, plus some extra details. Then a procmail rule can test that X-Spam: header and deliver the e-mail to the right mailbox. 💅

On top of that, SpamOracle also analyses MIME attachments, pulling out relevant info like the MIME type, the character encoding and the attached file name, and it summarizes all of that in an additional X-Attachments: header. That's what lets procmail easily reject e-mails with suspicious attachments — like Windows executables, which often indicate a virus. 💀

---

✨ REQUIREMENTS AND LIMITATIONS ✨

To use SpamOracle, your mail has to be delivered to a Unix machine where you've got a shell account. That machine needs procmail(1) installed (peep http://www.procmail.org/). Your ~/.forward file has to be set up so all incoming e-mail runs through procmail(1). And if your mail server supports the POP or IMAP protocols, you can also use fetchmail(1) to fetch your mail from the server and have it delivered to your local machine. 🫶

To provide the corpus of messages SpamOracle "learns" from, you need an archive of about 1000 of your e-mails. That archive has to be manually or semi-automatically split into known spams and known good messages. Heads up: mis-classified messages in the corpus (like spams that got mistakenly stored among the good messages) will decrease the efficiency of the classification. 😭 The archive must be in Unix mailbox format, or in "one message per file" format (a la MH). Other formats, such as Emacs' Babyl, are not supported.

the vibe check: the notion of "word" that SpamOracle uses is slanted towards Western European languages, i.e. the ISO Latin-1 and Latin-9 character sets. Preliminary support for JIS-encoded Japanese can be selected at compile-time. But SpamOracle will not work well if you receive many legitimate e-mails written in other character sets, such as Chinese or Korean sets.

---

✨ INITIALIZATION ✨

To build the database of word frequencies from the corpus, do this:

rm ~/.spamoracle.db
spamoracle add -v -good goodmails -spam spammails

By default, the database is stored in the file .spamoracle.db in your home directory. You can override that with the -f option: spamoracle -f mydatabase add .... And the -v option prints progress information while it's chewing through the corpus. ✨

This assumes the good, non-spam messages from the corpus live in the file goodmails, and the known spam messages live in the file spammails. You can also fetch corpus messages from several files, and/or process them via several invocations of SpamOracle:

spamoracle add -good goodmails1 ... goodmailsN
spamoracle add -spam spammails1 ... spammailsP

---

✨ TESTING THE DATABASE ✨

To check that the database was built correctly — and to familiarize yourself with the statistical analysis SpamOracle performs — invoke the "test" mode on the mailboxes you just used for building the corpus:

spamoracle test goodmails | more
spamoracle test spammails | more

For each message in the given mailboxes, you'll see a summary like this one:

From: bbo <midhack@ureach.com>
Subject: Check This Out
Score: 1.00 -- 15
Details: refid:98 $$$$:98 surfing:98 asp:95 click:93 cable:92
  instantly:90 https:88 internet:87 www:86 U4:85 isn't:14 month:81
  com:75 surf:75
Attachments: cset="GB2312" type="application/octet-stream"
  name="Guangwen4.zip"
File: inbox/314

The first two lines are just the From: and Subject: fields of the original message.

The Score: line summarizes the result of the analysis. The first number (between 0.0 and 1.0) is the probability that the message is actually spam — or, equivalently, the degree of similarity of the message with the spam messages in the corpus. The second number (an integer between 0 and 15) is the number of "interesting" words found in the message. "Interesting" words are those that occur at least 5 times in the corpus. In the example, we've got 15 interesting words (the max!) and a score of 1.00, indicating a spam with high certainty. 🔥

The Details: line provides an explanation of the score. It lists the 15 most interesting words found in the message — that is, the 15 interesting words whose probability of denoting a spam is farthest away from the neutral 0.5. Each word is given with its individual score, written as a percentage (between 01 and 99) rather than as a probability, so as to save space. Here we see a bunch of very "spammish" words such as $$$$ or click, with probability 0.98 and 0.93 respectively, plus a few "innocent" words such as isn't (probability 0.14). The U4 word with probability 0.85 is actually a pseudo-word representing a 4-letter word all in uppercase — something spammers are fond of. 🙈

The Attachments: line summarizes some information about MIME attachments for this message. Here we've got one attachment of type application/octect-stream, file name Guangwen4.zip, and character set GB2312 (an encoding for Chinese).

The File: line shows the file that's being tested.

Normally, when running spamoracle test goodmails, most messages should come out with a low score (0.2 or less), and when running spamoracle test spammails, most messages should come out with a high score (0.8 or more). If not, your corpus isn't very good, or not well classified into spam and non-spam. To quickly see the outliers, you can reduce the interval of scores for which message summaries are displayed, like this:

spamoracle test -min 0.2 goodmails | more
       # Shows only good mails with score >= 0.2
spamoracle test -max 0.8 spammails | more
       # Shows only spam mails with score <= 0.8

Now, for a more challenging test, take a mailbox that contains unfiltered e-mails — i.e. a mixture of spam and legitimate e-mails — and run it through SpamOracle:

spamoracle test mymailbox | less

Marvel at how well the oracle recognizes spam from the rest! 🌈 If the result isn't that marvelous to you, keep in mind that certain spams are just too short to be recognized (not enough significant words). Also, perhaps your corpus was too small, or not well categorized... 😭

---

✨ MARKING AND FILTERING INCOMING E-MAIL ✨

Once the database is built, you're ready to run incoming e-mails through SpamOracle. The command spamoracle mark reads one e-mail from standard input, and copies it to standard output, with two headers inserted: X-Spam: and X-Attachments:. The X-Spam: header has one of the following formats:

X-Spam: yes; score; details

or

X-Spam: no; score; details

or

X-Spam: unknown; score; details

The score and details are as described for spamoracle test.

The yes/no/unknown tag synthesizes the results of the analysis: yes means the score is >= 0.8 and at least 5 interesting words were found; no means the score is <= 0.2 and at least 5 interesting words were found; unknown is returned otherwise. The unknown case generally occurs for very short messages, where not enough interesting words were found. 💅

The X-Attachments: header contains the same information as the Attachments: output of spamoracle test, that is, a summary of the message attachments.

To automatically process your incoming e-mail through SpamOracle and act upon the results of the analysis, just insert the following "recipes" in the file ~/.procmailrc:

:0fw
| /usr/local/bin/spamoracle mark

:0
* ^X-Spam: yes;
spambox

What these cryptic commands mean is:

• Run every mail through the spamoracle mark command. (If spamoracle wasn't installed in /usr/local/bin, adjust the path as necessary.) This adds two headers to the message: X-Spam: and X-Attachments:, describing the results of the spam analysis and the attachment analysis.
• If we have an X-Spam: yes header, deliver the message to the file spambox rather than to your regular mailbox. Presumably, you'll read spambox once in a while, but less often than your regular mailbox.

bestie, listen to me: daring users can put /dev/null instead of spambox to just throw away the message, but please don't do that until you've used SpamOracle for a while and are happy with the results. SpamOracle's false positive rate (i.e. legitimate mails classified as spam) is low (0.1%) but not null. So, better save the presumed spams somewhere, and scan them quickly from time to time. 🫶

If you'd like to enjoy a bit of attachment-based filtering, here are some procmail rules for that:

:0
* ^X-Attachments:.*name=".*\.(pif|scr|exe|bat|com)"
spambox

:0
* ^X-Attachments:.*type="audio/(x-wav|x-midi)
spambox

:0
* ^(Content-type:.*|X-Attachments:.*cset="|^Subject:.*=\?)(ks_c|gb2312|iso-2|euc-|big5|windows-1251)
spambox

The first rule treats as spam every mail that has a Windows executable as an attachment. These mails are typically sent by viruses. 👑 The second rule does the same with attachments of type x-wav or x-midi. I never normally receive music by e-mail, however some popular e-mail viruses seem fond of these attachment types. The third rule treats as spam every mail that uses character encodings corresponding to Korean, Chinese, Japanese, and Cyrillic. 🌸

💾 UPDATING THE DATABASE 💾

okay so bestie, the database is never done growing. Any time you want, you can feed it more known spams or more known legit messages with the spamoracle add command. it's giving glow-up, but for your filter ✨

So like, say a spam message slips through and does NOT get classified as spam? Rude. Run it through spamoracle add -spam so SpamOracle can literally learn from its own mistake 🫶 (and heads up: with no extra arguments, this command reads a single message from standard input and records it as spam.)

Under mutt(1) for example, you just highlight the spam message and type:

|spamoracle add -spam

Same energy the other way — if you're scrolling your spam box and find a legitimate message that got banished unfairly, run it through spamoracle add -good.

Another option: collect a bunch more known spams or known good messages into mailbox files, and every once in a while go:

spamoracle add -good new_good_mails
spamoracle add -spam new_spam_mails

---

🔍 QUERYING THE DATABASE 🔍

For your edification and your entertainment (yes queen, both), the contents of the database can be queried by regular expressions. The spamoracle list regexp command lists all words in the database that match regexp — an Emacs-style regular expression — along with their number of occurrences in spam mail and in good mail. Obsessed with the transparency ngl.

For instance:

spamoracle list '.*' # show all words -- big list!
spamoracle list 'sex.*'
spamoracle list 'linux.*'

(fr fr that first one is a big list, brace yourself 💀)

---

🗄️ DATABASE BACKUPS AND UPGRADES 🗄️

The database SpamOracle uses is stored in a compact, binary format that is not human-readable. And plot twist: this format is subject to change in later versions of SpamOracle. So to make backups and upgrades not a nightmare, the database contents can also be manipulated in a portable, text format. iconic behaviour honestly 👑

• The spamoracle backup command dumps the contents of the database to standard output, in a textual, portable format.
• The spamoracle restore command reads such a dump from standard input and rebuilds the database with this data.

The recommended procedure for upgrading to a newer version of SpamOracle is:

# Before the upgrade:
spamoracle backup > backupfile
# Upgrade SpamOracle
# Restore the database
spamoracle restore < backupfile

Alternatively, the spamoracle upgrade command converts a database created with an earlier version of SpamOracle to the native database format of the current version of SpamOracle. Main character energy: out with the old, in with the new 💅

---

⚙️ CONFIGURING FILTERING PARAMETERS ⚙️

Many of the parameters that govern message classification can be configured via a configuration file. By default, the config is read from the file .spamoracle.conf in the user's home directory. Want a different config file? Specify it on the command line with the -config option:

spamoracle -config myconfigfile ...

The list of configurable parameters and the format of the configuration file are described in spamoracle.conf(5).

All parameters have reasonable defaults, but bestie you can try to improve the quality of classification further by tweaking them 🎀 To figure out the impact of your changes, use either the test or stat commands to spamoracle. The spamoracle stat command prints a one-line summary of how many spam, non-spam, and unknown messages were found in the mailboxes given as arguments. receipts, literally 📊

---

🧠 TECHNICAL DETAILS 🧠

SpamOracle's notion of "word" is any run of 3 to 12 of the following characters: letters, single quotes, and dashes (-). If support for non-English european languages was compiled in, word characters also include the relevant accented letters for the languages in question. All words are mapped to lowercase, and accented letters are mapped to the corresponding non-accented letters.

A run of 3 to 12 of the following characters also constitutes a word: digits, dots, commas, and dollar, Euro and percent signs.

In addition, a run of three or more uppercase letters generates a pseudo-word Un where n is the length of the run. Similarly, a run of three or more non-ASCII characters (code >= 128) generates a pseudo-word Wn where n is the length of the run. so mysterious, so powerful ✨

For instance, the following text:

SUMMER in English is written "ete" in French

is processed into the following words, assuming French support was selected at compile-time:

U5 summer english written ete french W3

and if French support was not selected:

U5 summer english written french W3

To see the words that are extracted from a message, issue the spamoracle words command. It reads either a single message from standard input, or all messages from the mailbox files given as arguments, decomposes the messages into words and prints the words. peek behind the curtain, slay 🔮

---

🎲 RANDOM NOTES 🎲

The database file can be compressed with gzip(1) to save disk space, at the expense of slower spamoracle operations. If the database file specified with the -f option has the extension .gz, spamoracle will automatically uncompress it on start-up, and re-compress it after updates. set it and forget it, honestly 🦋

If your mail is stored in MH format, you may run into "command line too long" errors while trying to process a lot of small files with the spamoracle add command, e.g. when doing:

spamoracle add -good archives/*/* -spam spam/*

Instead, do something like:

find archives -type f -print | xargs spamoracle add -good
find spam -type f -print | xargs spamoracle add -spam

lowkey a lifesaver, highkey just good shell hygiene 🐚

---

👑 AUTHOR 👑

Xavier Leroy <Xavier.Leroy@inria.fr>

---

🌈 SEE ALSO 🌈

spamoracle.conf(5); procmail(1); fetchmail(1)

http://spamoracle.forge.ocamlcore.org/ (SpamOracle distribution site)

http://www.paulgraham.com/spam.html (Paul Graham's seminal paper)

---

okay that's the whole page bestie, we ate and left absolutely no crumbs 💅 this was the SPAMORACLE(1) page and it lives in the usual man page spot — go forth and filter, you've got this 🫶✨