SA-LEARN(1p)

📦
version 4.0.2+svn1935691 · 9 versions
, in ubuntu stonking

👑 sa-learn(1p) — from User Contributed Perl Documentation 🎀

✨ NAME ✨

sa-learn — train SpamAssassin's Bayesian classifier 🧠💖

🌸 SYNOPSIS 🌸

sa-learn [options] [file]...

sa-learn [options] --dump [ all | data | magic ]

💫 Options 💫

  • --ham — learn the messages that come next as ham (aka not-spam, the good stuff) 🫶
  • --spam — learn the messages that come next as spam (booo 🍅)
  • --forget — forget the messages that come next
  • --use-ignores — actually respect bayes_ignore_from and bayes_ignore_to
  • --sync — sync up the database and the journal if needed
  • --force-expire — force a database sync and an expiry run, no waiting
  • --dbpath <path> — lets you override from the command line (in bayes_path form) where the Bayes DB gets read from
  • --dump [all|data|magic] — show what's inside the Bayes database; takes an optional argument for which bit to display
  • --regexp <re> — for dump only; says which tokens get dumped, based on a regular expression
  • -f file, --folders=file — read the list of files/directories out of file
  • --dir — ignored; it's here for historical compatibility only
  • --file — ignored; also historical compatibility, bestie
  • --mbox — the input sources are in mbox format
  • --mbx — the input sources are in mbx format
  • --max-size <b> — skip messages bigger than b bytes; defaults to 500 KB, and 0 means no limit at all
  • --showdots — show progress with little dots 🔴
  • --progress — show progress with a progress bar
  • --no-sync — skip synchronizing the database and journal after learning
  • -L, --local — operate locally, no network accesses. Using this is recommended, check the docs 📖
  • --import — migrate data over from older-version / non-DB_File-based databases
  • --clear — wipe out the existing database 💀
  • --backup — back up the existing database, out to STDOUT
  • --restore <filename> — restore a database from filename
  • -u username, --username=username — override the username taken from the runtime environment; used with SQL
  • -C path, --configpath=path, --config-file=path — path to the standard configuration dir
  • -p prefs, --prefspath=file, --prefs-file=file — set the user preferences file
  • --siteconfigpath=path — path for site configs (default: /usr/etc/mail/spamassassin)
  • --cf='config line' — one extra line of configuration, right here on the command line
  • -D, --debug [area,...] — print debugging messages
  • -V, --version — print the version
  • -h, --help — print the usage message

🌈 DESCRIPTION 🌈

Okay so picture this: you've got a big old pile of incoming mail, some of it classified as spam and some as ham (ham = non-spam, the mail you actually want 💖). Hand all that to this tool and it feeds each mail to SpamAssassin so the thing can learn which signs tend to mean spam and which signs tend to mean ham. It's giving glow-up, but for your classifier.

Just run this command once for every mail folder you've got, and it will learn from all the mail sitting in there. One folder, one run. Periodt.

Lowkey important: csh-style globbing in the mail folder names is supported. Like, if you list a folder name as *, it'll scan every folder that matches. See Mail::SpamAssassin::ArchiveIterator for more details. 🔍

If your mail boxes are in some format that isn't maildir, you should be reaching for the --mbox or --mbx parameters.

Files compressed with gzip/bzip2/xz/lz4/lzip/lzo get uncompressed automatically, no cap ✨. Again, Mail::SpamAssassin::ArchiveIterator has the details.

SpamAssassin remembers which mail messages it's already learnt, so it will not re-learn those messages again — unless you hit it with the --forget option. And messages learnt as spam get their SpamAssassin markup removed, on the fly. Iconic behaviour honestly.

Made a mistake and scanned a mail as ham when it's actually spam, or the other way round? No stress 🙈. Just rerun this command with the correct classification and the mistake gets corrected. SpamAssassin will automatically forget the previous indications for you.

Heads up to all the spamd girlies who wanna do their training remotely, over a network: go investigate the spamc -L switch. 👀

✨ OPTIONS ✨

okay bestie, buckle up, this is the big one 💅

  • --ham — learns the input message(s) in the files that follow this option as ham. If you've already taught SpamAssassin any of those messages as spam before, it will forget them first and then re-learn them as ham. But if you already taught them as ham, it just skips them this time around. And if the messages have already been run through SpamAssassin at some point, the learner ignores any modifications SpamAssassin may have made. 🫶
  • --spam — same energy, opposite direction: learns the input message(s) in the files that follow the option as spam. If you've previously learnt any of them as ham, SpamAssassin forgets them first, then re-learns them as spam. If you already learnt them as spam, it skips them this time. And again, if the messages have already been filtered through SpamAssassin, the learner ignores any modifications SpamAssassin may have made. 😌
  • --folders=filename, -f filename — sa-learn reads a list of folders out of the file you name, one folder per line. If a folder is prefixed with "ham:type:" or "spam:type:", then sa-learn learns that folder the way you said. Otherwise the folders are assumed to be whatever type you set with --ham or --spam.

That "type" bit is optional, but it's the same standard used for ArchiveIterator: mbox, mbx, dir, file, or detect (and detect is the default if you don't specify anything). 🌈

  • --mbox — sa-learn reads the file(s) holding the emails to be learnt and processes them in mbox format (that's one or more emails per file, iykyk).
  • --mbx — sa-learn reads the file(s) holding the emails to be learnt and processes them in mbx format (again, one or more emails per file).
  • --use-ignores — don't learn the message if a from address matches the config item bayes_ignore_from, or a to address matches bayes_ignore_to. This one's a lifesaver when you're learning from a giant pile of messages where nobody bothered to pull out the hammy spam messages or the spammy ham messages. 🙈
  • --sync — synchronizes the journal and databases. Once the database has been successfully synced up with everything in the journal, the journal file gets removed. bye girl 💨
  • --force-expire — forces an expiry attempt, regardless of whether it's actually necessary or not. Note: this does not mean any tokens will actually expire. go read the EXPIRATION section down below for the real tea.

Note: --force-expire also causes the journal data to be synchronized into the Bayes databases.

  • --forget — forgets the input message(s) in the files following the option, as in, the ones you previously learnt. ✌️
  • --dbpath — lets you override the bayes_path configuration option straight from the command line.
  • --dump option — displays the contents of the Bayes database. With no option, or with the all option, all magic tokens and data tokens get displayed. magic shows only the magic tokens, and data shows only the data tokens.

You can also throw in the --regexp RE option to pick which tokens get displayed based on a regular expression. ⭐

  • --clear — clears an existing Bayes database by removing all traces of it.
WARNING: this is destructive and should be used with care. not a drill, bestie 😭
  • --backup — performs a dump of the Bayes database in a machine/human readable format.

The dump includes token and seen data. It's suitable for feeding right back into the --restore command. 🎀

  • --restore=filename — performs a restore of the Bayes database defined by filename.
WARNING: this is a destructive operation — your previous Bayes data will be wiped out. 💀
  • -h, --help — prints the help message and exits. classic.
  • -u username, --username=username — if you specify this, the username overrides whatever username got taken from the runtime environment. You can use this to name users in a virtual user configuration when you're using SQL as the Bayes backend.

NOTE: this option does not switch to the given username — it only attempts to act on behalf of that user. Because of that, you need proper permissions to be able to change files owned by username. In the case of SQL, this generally isn't a problem. 👑

  • -C path, --configpath=path, --config-file=path — use the specified path for locating the distributed configuration files. Ignores the default directories (usually "/usr/share/spamassassin" or similar).
  • --siteconfigpath=path — use the specified path for locating site-specific configuration files. Ignores the default directories (usually "/etc/spamassassin" or similar).
  • --cf='config line' — adds extra lines of configuration directly from the command line, parsed after the configuration files are read. You can use multiple --cf arguments, and each one counts as its own separate line of configuration. ✨
  • -p prefs, --prefspath=prefs, --prefs-file=prefs — reads user score preferences from prefs (usually "$HOME/.spamassassin/user_prefs").
  • --progress — prints a progress bar (to STDERR) showing you the current progress. If no valid terminal is found, this option behaves very much like the --showdots option. we love a fallback 🫶
  • -D [area,...], --debug [area,...] — produces debugging output. If you list no areas, all debugging info gets printed. You can also enable diagnostic output for each area individually — area is the area of the code you want to instrument. For example, to get diagnostic output on bayes, learn, and dns, do:
spamassassin -D bayes,learn,dns

Use an empty string (-D '') to indicate no areas when the next item on the command line is a path — that stops the path from being parsed as an area. lowkey a sneaky little gotcha, bestie.

For more info on which areas (also known as channels) are available, please see the documentation at:

https://wiki.apache.org/spamassassin/DebugChannels

Higher priority informational messages that are suitable for logging in normal circumstances are available with an area of "info".

  • --no-sync — skips the slow synchronization step that normally happens after changing database entries. If you're planning to learn from a whole bunch of folders in one batch, or learn a ton of individual messages one-by-one, it's faster to use this switch and then run "sa-learn --sync" once all the folders have been scanned. ⚡

Clarification: the state of --no-sync overrides the bayes_learn_to_journal configuration option. If not specified, sa-learn learns to the database directly. If specified, sa-learn learns to the journal file.

Note: --sync and --no-sync can both be specified on the same command line, which is slightly confusing. In that case, the --no-sync option is ignored, since there's no learn operation happening anyway.

  • -L, --local — do not perform any network accesses while learning details about the mail messages. You should normally use this one, because there really isn't anything Bayes can learn from network lookup results. Official SpamAssassin plugins don't currently do any network lookups when learning, but third party ones might. 😬
  • --import — if you previously used SpamAssassin's Bayesian learner without the "DB_File" module installed, it will have created files in other formats, like "GDBM_File", "NDBM_File", or "SDBM_File". This switch lets you migrate that old data into the "DB_File" format. It will overwrite any data currently in "DB_File".

Can also be used with the --dbpath path option to specify the location of the Bayes files to use. 🌸

🦋 MIGRATION 🦋

okay so there are now multiple backend storage modules for storing a user's bayesian data, which means sometimes you're gonna want to move from one backend to another. here's the simple procedure for migrating from one backend to another. slay 💅

Just a heads up: if you have individual user databases, you'll have to do a similar procedure for each one of them.

  • sa-learn --sync — this will sync any outstanding journal entries.
  • sa-learn --backup > backup.txt — this will save all your Bayes data to a plain text file.
  • sa-learn --clear — this is optional, but good to do, to clear out the old database.
  • Repeat! — at this point, if you have multiple databases, you should run the procedure above for each of them. (i.e. each user's database needs to be backed up before continuing.)
  • Switch backends — once you've backed up all the databases, you can update your configuration for the new database backend. This will involve at least the bayes_store_module config option, and may involve some additional config options depending on what the module requires. (For example, you may need to configure an SQL database.)
  • sa-learn --restore backup.txt — again, you need to do this for every database.

If you're migrating to SQL, you can make use of the -u username option in sa-learn to populate each user's database. Otherwise, you must run sa-learn as the user whose database you're restoring. 👑

💅 INTRODUCTION TO BAYESIAN FILTERING 🧠✨

(Shoutout to Michael Bell for this section! King behavior honestly.)

So if you want the full ✨deluxe✨ explanation of how this all works, go hit up http://www.paulgraham.com/ and read "A Plan for Spam". It's actually reasonably readable, even if statistics make me break out in hives. 😭

The short, semi-inaccurate version: once you give it some training, a spam heuristics engine can take the most "spammy" and "hammy" words and run probabilistic analysis on them. And here's the slay part — once it's got a basis for the analysis, the engine can keep learning iteratively by running both the non-Bayesian and Bayesian rulesets together to build evolving "intelligence". It's giving glow-up. 🌱

SpamAssassin 2.50 and later supports Bayesian spam analysis, in the form of the BAYES rules. This is a new feature, it's quite powerful, and it's disabled until enough messages have been learnt. No vibes-based spam detection until you feed it, bestie.

The pros of Bayesian spam analysis, bestie:

  • Can greatly reduce false positives and false negatives. It learns from your mail, so it's tailored to your unique e-mail flow. Custom fit, no cap. 👑
  • Once it starts learning, it can continue to learn from SpamAssassin and improve over time.

And the cons, because we keep it real:

  • A decent number of messages are required before the results are even useful for ham/spam determination.
  • It's hard to explain why a message is or isn't marked as spam. Like — a straightforward rule that matches, say, VIAGRA is easy to understand. If it generates a false positive or a false negative, it is fairly easy to understand why. But with Bayesian analysis, it's all probabilities — "because the past says it is likely as this falls into a probabilistic distribution common to past spam in your systems". Tell that to your users! Tell that to the client when he asks "what can I do to change this". (By the way, the answer in this case is "use welcomelisting". You're welcome. 🙈)
  • It will take disk space and memory. The databases it maintains take quite a lot of resources to store and use. She's high-maintenance, we accept her as she is. 💅

🚀 GETTING STARTED 🌈

Still interested? Ok periodt, here's the guidelines for getting this working.

First, a high-level overview:

  • Build a significant sample of both ham and spam. I suggest several thousand of each, placed in SPAM and HAM directories or mailboxes. Yes, you MUST hand-sort this — otherwise the results won't be much better than SpamAssassin on its own. Verify the spamminess/haminess of EVERY message. You're urged to avoid using a publicly available corpus (sample) — this must be taken from YOUR mail server, if it's going to be statistically useful. Otherwise, the results may be pretty skewed. 😬
  • Use this tool to teach SpamAssassin about these samples, like so:
sa-learn --spam /path/to/spam/folder
sa-learn --ham /path/to/ham/folder
sa-learn --ham hampath1 hampath2 --spam spampath1 spampath2
...
  • Let SpamAssassin proceed, learning stuff. When it finds ham and spam it will add the "interesting tokens" to the database. It's collecting little lore nuggets. 🦋
  • If you need SpamAssassin to forget about specific messages, use the --forget option. This can be applied to either ham or spam that has run through the sa-learn processes. It's a bit of a hammer, really — it lowers the weighting of the specific tokens in that message (only if that message has been processed before).
  • Learning from single messages uses a command like this:
sa-learn --ham --no-sync mailmessage

This is handy for binding to a key in your mail user agent. It's very fast, as all the time-consuming stuff is deferred until you run with the "--sync" option. Lazily-eager, we love a deferred queen. ⭐

  • Autolearning is enabled by default. If you don't have a corpus of mail saved to learn, you can let SpamAssassin automatically learn the mail that you receive. If you are autolearning from scratch, the amount of mail you receive will determine how long until the BAYES_* rules are activated. Patience, bestie, she's marinating. 🍓

📚 EFFECTIVE TRAINING 💖

Learning filters require training to be effective. If you don't train them, they won't work. That's it, that's the tweet. And on top of that, you need to train them with new messages regularly to keep them up-to-date, or their data will become stale and impact accuracy. Stale data is not the vibe check we want. 💀

You need to train with both spam and ham mails. One type of mail alone will not have any effect. Both or nothing, fr fr.

Heads up: if your mail folders contain things like forwarded spam, discussions of spam-catching rules, etc., this will cause trouble. You should avoid scanning those messages if possible. (An easy way to do this is to move them aside, into a folder which is not scanned.) 🙈

If the messages you are learning from have already been filtered through SpamAssassin, the learner will compensate for this. In effect, it learns what each message would look like if you had run spamassassin -d over it in advance. She's psychic like that. ✨

Another thing to be aware of: typically you should aim to train with at least 1000 messages of spam, and 1000 ham messages, if possible. More is better, but anything over about 5000 messages does not improve accuracy significantly in our tests. Diminishing returns, she said. 📉

Be careful that you train from the same source — for example, if you train on old spam, but new ham mail, then the classifier will think that a mail with an old date stamp is likely to be spam. Correlated data = chaos. 😭

It's also worth noting that training with a very small quantity of ham will produce atrocious results. You should aim to train with at least the same amount (or more if possible!) of ham data than spam. Balance the diet, bestie.

On an on-going basis, it is best to keep training the filter to make sure it has fresh data to work from. There are various ways to do this:

  1. Supervised learning This means keeping a copy of all or most of your mail, separated into spam and ham piles, and periodically re-training using those. It produces the best results, but requires more work from you, the user. Main character energy, high effort, high reward. 👑

(An easy way to do this, by the way, is to create a new folder for 'deleted' messages, and instead of deleting them from other folders, simply move them in there instead. Then keep all spam in a separate folder and never delete it. As long as you remember to move misclassified mails into the correct folder set, it is easy enough to keep up to date.) 🎀

  1. Unsupervised learning from Bayesian classification Another way to train is to chain the results of the Bayesian classifier back into the training, so it reinforces its own decisions. This is only safe if you then retrain it based on any errors you discover. Echo chamber behavior, lowkey risky.

SpamAssassin does not support this method, due to experimental results which strongly indicate that it does not work well, and since Bayes is only one part of the resulting score presented to the user (while Bayes may have made the wrong decision about a mail, it may have been overridden by another system).

  1. Unsupervised learning from SpamAssassin rules Also called 'auto-learning' in SpamAssassin. Based on statistical analysis of the SpamAssassin success rates, we can automatically train the Bayesian database with a certain degree of confidence that our training data is accurate. She studied the data, she's confident. 💅

It should be supplemented with some supervised training in addition, if possible.

This is the default, but can be turned off by setting the SpamAssassin configuration parameter "bayes_auto_learn" to 0.

  1. Mistake-based training This means training on a small number of mails, then only training on messages that SpamAssassin classifies incorrectly. This works, but it takes longer to get it right than a full training session would. Slow burn. ⏳

🗂️ FILES 💾

sa-learn and the other parts of SpamAssassin's Bayesian learner use a set of persistent database files to store the learnt tokens, as follows.

  • bayes_toks — The database of tokens, containing the tokens learnt, their count of occurrences in ham and spam, and the timestamp when the token was last seen in a message.

This database also contains some 'magic' tokens, as follows: the version number of the database, the number of ham and spam messages learnt, the number of tokens in the database, and timestamps of: the last journal sync, the last expiry run, the last expiry token reduction count, the last expiry timestamp delta, the oldest token timestamp in the database, and the newest token timestamp in the database. Magic tokens?? iconic behavior honestly. 🔮

This is a database file, using "DB_File". The database 'version number' is 0 for databases from 2.5x, 1 for databases from certain 2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later releases.

  • bayes_seen — A map of Message-Id and some data from headers and body to what that message was learnt as. This is used so that SpamAssassin can avoid re-learning a message it has already seen, and so it can reverse the training if you later decide that message was learnt incorrectly.

This is a database file, using "DB_File".

  • bayes_journal — While SpamAssassin is scanning mails, it needs to track which tokens it uses in its calculations. To avoid the contention of having each SpamAssassin process attempting to gain write access to the Bayes DB, the token timestamps are written to a 'journal' file which will later (either automatically or via sa-learn --sync) be used to synchronize the Bayes DB.

Also, through the use of "bayes_learn_to_journal", or when using the --no-sync option with sa-learn, the actual learning data will be placed into the journal for later synchronization. This is typically useful for high-traffic sites to avoid the same contention as stated above. Queue it up, sync it later, never fight over the DB. 🧚

🌸 EXPIRATION 🌸

okay bestie, so here's the situation: SpamAssassin can auto-learn messages, which means the Bayes database files could just keep growing and growing forever until they eat your entire disk. not cute. to keep that in check, SpamAssassin does journal synchronization and bayes expiration on a schedule — but only when certain criteria (listed below, obvi) are met.

SpamAssassin can sync the journal and expire the DB tokens either manually or opportunistically. 💅

A journal sync is due if --sync is passed to sa-learn (that's the manual route), or if the following is true (the opportunistic route):

  • bayes_journal_max_size does not equal 0 (0 means don't sync)
  • the journal file exists

and either:

  • the journal file has a size greater than bayes_journal_max_size

or

  • a journal sync has previously occurred, and at least 1 day has passed since that sync

Expiry is due if --force-expire is passed to sa-learn (manual, so you're choosing violence), or if all of the following are true (opportunistic):

  • the last expire was attempted at least 12hrs ago
  • bayes_auto_expire does not equal 0
  • the number of tokens in the DB is > 100,000
  • the number of tokens in the DB is > bayes_expiry_max_db_size
  • there is at least a 12 hr difference between the oldest and newest token atimes

💫 EXPIRE LOGIC 💫

If either the manual or the opportunistic method kicks off an expire run, here's the logic it follows:

  • figure out how many tokens to keep. take the larger of either bayes_expiry_max_db_size * 75% or 100,000 tokens. therefore, the goal reduction is number of tokens - number of tokens to keep.
  • if the reduction number is < 1000 tokens, abort (not worth the effort).
  • if an expire has been done before, guesstimate the new atime delta based on the old atime delta. (new_atime_delta = old_atime_delta * old_reduction_count / goal)
  • if no expire has been done before, or the last expire looks "weird", do an estimation pass. The definition of "weird" is:
    • last expire over 30 days ago
    • last atime delta was < 12 hrs
    • last reduction count was < 1000 tokens
    • estimated new atime delta is < 12 hrs
    • the difference between the last reduction count and the goal reduction count is > 50%

🔮 ESTIMATION PASS LOGIC 🔮

Go through each of the DB's tokens. Starting at 12hrs, calculate whether or not the token would be expired (based on the difference between the token's atime and the db's newest token atime) and keep the count. Work out from 12hrs exponentially by powers of 2. ie: 12hrs * 1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512 (6144hrs, or 256 days).

The larger the delta, the smaller the number of tokens that will be expired. Conversely, the number of tokens goes up as the delta gets smaller. So starting at the largest atime delta, figure out which delta will expire the most tokens without going above the goal expiration count. Use this to choose the atime delta to use, unless one of the following occurs:

  • the largest atime (smallest reduction count) would expire too many tokens. this means the learned tokens are mostly old and there needs to be new tokens learned before an expire can occur.
  • all of the atime choices result in 0 tokens being removed. this means the tokens are all newer than 12 hours and there needs to be new tokens learned before an expire can occur.
  • the number of tokens that would be removed is < 1000. the benefit isn't worth the effort. more tokens need to be learned.

If the expire run gets past this point, it will continue to the end. A new DB is created since the majority of DB libraries don't shrink the DB file when tokens are removed. So we do the "create new, migrate old to new, remove old, rename new" shuffle. 🕺

bayes_auto_expire is used to specify whether or not SpamAssassin ought to opportunistically attempt to expire the Bayes database. The default is 1 (yes).

bayes_expiry_max_db_size specifies both the auto-expire token count point, as well as the resulting number of tokens after expiry as described above. The default value is 150,000, which is roughly equivalent to a 6Mb database file if you're using DB_File.

bayes_journal_max_size specifies how large the Bayes journal will grow before it is opportunistically synced. The default value is 102400.

🧚 INSTALLATION 🧚

The sa-learn command is part of the Mail::SpamAssassin Perl module. Install this as a normal Perl module, using perl -MCPAN -e shell, or by hand.

👀 SEE ALSO 👀

🍓 PREREQUISITES 🍓

Mail::SpamAssassin

👑 AUTHORS 👑

The SpamAssassin(tm) Project <https://spamassassin.apache.org/>

and that's a wrap on the whole page, bestie 🫶 this copy came to you courtesy of perl v5.40.1, dated 2026-07-23, and it lives under the name SA-LEARN(1p) — go forth and train your Bayes filter, you absolute icon 💖✨

📦 where this page comes from