MAILTOE(1)

📦
version 1.14 · 2 versions
, in nixos 26.05

okay bestie, listen up — this is mailtoe from section 1 of the manual (yes, MAILTOE(1), cite it correctly at the function 💅), and it's genuinely such a nerdy little icon. It's long, so this is part 1 of 2. Stay with me.

🎀 NAME 🎀

mailtoe — a train-on-error simulator for use with dbacl.

✨ SYNOPSIS ✨

mailtoe command [ command_arguments ]

That's it. One command, then whatever arguments that command wants. Iconic minimalism, no notes.

💖 DESCRIPTION 💖

So mailtoe automates the whole task of testing email filtering and classification programs like dbacl(1). You feed it a set of already-categorized documents, it kicks off test runs to estimate the classification errors, and then you get to fine tune the parameters of your classifier. Basically it runs the experiment so you can tweak the settings. Slay.

Train-on-error (TOE, and yes the acronym is a whole vibe) is a learning method that gets advocated sometimes for email classifiers. Here's the deal: you've got an incoming email stream, and the method reuses a fixed set of category databases until the very first misclassification happens. At that point, the offending email gets used to relearn the relevant category — and then it just chills until the next misclassification. So categories only get updated when errors actually occur. It's a direct model of how some email classifiers are used in the real world. Fr fr, that's the whole method.

⚠️ Big caveat, don't skip this one. TOE's error rates depend directly on the order in which the emails are seen. A tiny change in ordering — the kind of thing that happens because of networking delays — can have a large impact on the number of misclassifications. So mailtoe does not give meaningful results unless the sample emails are chosen carefully. That said, since this method is commonly used by spam filters, it's still worth computing so you can make comparisons. Other methods (peep mailcross(1) and mailfoot(1)) try to capture the behaviour of classification errors in other ways.

To improve and stabilize the error rate calculation, mailtoe runs the TOE simulations several times on slightly reordered email streams and averages the results. The reorderings come from multiplexing the emails from each category mailbox in random order. So if there are three categories, the first email classified is chosen randomly from the front of the sample email streams of each type. The second email is also chosen randomly among the three types, from the front of the streams after the first email was removed. Simulation stops when all sample streams are exhausted. Clean, fair, no cap.

mailtoe uses the environment variable MAILTOE_FILTER when executing, which lets you simulate arbitrary filters — as long as they satisfy the compatibility conditions in the ENVIRONMENT section below. 👀 (we'll get there, bestie)

For convenience, mailtoe also implements a testsuite framework with predefined wrappers for several open source classifiers. That means you can directly compare dbacl(1) with competing classifiers on the same set of email samples. See the USAGE section below for the whole routine.

During preparation, mailtoe builds a subdirectory named mailtoe.d in the current working directory. All the needed calculations happen inside that subdirectory — so everything's contained, nothing's spilling out. 🫶

🌈 EXIT STATUS 🌈

  • 0 — success, we ate 💅
  • 1 — a problem occurred 😭

👑 COMMANDS 👑

Each command is a whole little era. Here they all are, in order:

🌸 prepare size

Prepares a subdirectory named mailtoe.d in the current working directory, and populates it with empty subdirectories for exactly size subsets. This is like setting up the venue before the party.

🌸 add category [ FILE ]...

Takes a set of emails from either FILE if specified, or STDIN, and associates them with category. The ordering of emails within FILE is preserved, and subsequent FILEs are appended to the first in each category. This command can be repeated several times, but should be executed at least once. At least once, bestie. I'm serious.

🌸 clean

Deletes the directory mailtoe.d and all its contents. Poof. Gone. Closure, honestly.

🌸 run

Multiplexes randomly from the email streams added earlier, and relearns categories only when a misclassification occurs. The simulation is repeated size times. That's the TOE loop, that's the main character energy right there.

🌸 summarize

Prints average error rates for the simulations.

🌸 plot [ ps | logscale ]...

Plots the number of errors over simulation time. The ps option, if present, writes the plot to a postscript file in the directory mailtoe/plots, instead of being shown on-screen. The logscale option, if present, causes the plot to be on the log scale for both ordinates. Fancy axes, we love to see it.

🌸 review truecat predcat

Scans the last run statistics and extracts all the messages which belong to category truecat but have been classified into category predcat. The extracted messages are copied to the directory mailtoe.d/review for perusal. It's giving "show me the receipts" and honestly? Valid.

🌸 testsuite list

Shows a list of available filters/wrapper scripts which can be selected. Window shopping, basically. 🛍️

🌸 testsuite select [ FILTER ]...

Prepares the filter(s) named FILTER to be used for simulation. The filter name is the name of a wrapper script located in the directory /nix/store/66jnbrh6haa7grpxg320a4fvkal7irv3-dbacl-1.14/share/dbacl/testsuite. Each filter has a rigid interface documented below, and the act of selecting it copies it to the mailtoe.d/filters directory. Only filters located there are used in the simulations. So like, if it's not in the folder, it's not invited to the party. 🎟️

🌸 testsuite deselect [ FILTER ]...

Removes the named filter(s) from the directory mailtoe.d/filters so that they are not used in the simulation. Uninvited. Blocked. 💀

🌸 testsuite run [ plots ]

Invokes every selected filter on the datasets added previously, and calculates misclassification rates. If the plots option is present, each filter simulation is plotted as a postscript file in the directory mailtoe.d/plots. (Note that this one really is mailtoe.d/plots — check the plot command above, it says mailtoe/plots. I don't make the rules, I just report them. 🙈)

🌸 testsuite status

Describes the scheduled simulations. Like a little itinerary for your classifiers.

🌸 testsuite summarize

Shows the cross validation results for all filters. Only makes sense after the run command — so don't come at it sideways.

🧚 USAGE 🧚

The normal usage pattern goes a little something like this. First, you should separate your email collection into several categories (manually or otherwise). Each category should be associated with one or more folders, but each folder should not contain more than one category. One folder, one category. That's the rule. Next, decide how many runs to use, say 10. The more runs you use, the better the predicted error rates — but more runs take more time. Classic tradeoff, no way around it.

Now you can type:

% mailtoe prepare 10

Next, for every category, you must add every folder associated with that category. Suppose you have three categories named spam, work, and play, which are associated with the mbox files spam.mbox, work.mbox, and play.mbox respectively. You would type:

% mailtoe add spam spam.mbox
% mailtoe add work work.mbox
% mailtoe add play play.mbox

You should aim for a similar number of emails in each category, because the random multiplexing will be unbalanced otherwise. (Unbalanced categories = unbalanced vibes.) The ordering of the email messages in each *.mbox file is important, and is preserved during each simulation. If you repeatedly add to the same category, the later mailboxes will be appended to the first, preserving the implied ordering.

You can now perform as many TOE simulations as desired. The multiplexed emails are classified and learned one at a time, by executing the command given in the environment variable MAILTOE_FILTER. If not set, a default value is used.

% mailtoe run
% mailtoe summarize

The testsuite commands are designed to simplify all of the above steps and allow comparison of a wide range of email classifiers, including but not limited to dbacl. Classifiers are supported through wrapper scripts, which are located in the /nix/store/66jnbrh6haa7grpxg320a4fvkal7irv3-dbacl-1.14/share/dbacl/testsuite directory.

The first stage when using the testsuite is deciding which classifiers to compare. You can view a list of available wrappers by typing:

% mailtoe testsuite list
⚠️ Note that the wrapper scripts are NOT the actual email classifiers — those must be installed separately by your system administrator or otherwise. Do not come for the wrappers, they are not the real thing, they are the liaison. 😭

Once that's done, you can select one or more wrappers for the simulation by typing, for example:

% mailtoe testsuite select dbaclA ifile

If some of the selected classifiers cannot be found on the system, they are not selected. Note also that some wrappers can have hard-coded category names, e.g. if the classifier only supports binary classification. Heed the warning messages. Heed them. They're there for a reason, bestie. 🫶

It remains only to run the simulation. Beware, this can take a long time (several hours depending on the classifier). Like, go make a snack, go touch grass, it'll be here when you get back.

% mailtoe testsuite run
% mailtoe testsuite summarize

Once you are all done, you can delete the working files, log files etc. by typing:

% mailtoe clean

🦋 SCRIPT INTERFACE 🦋

mailtoe testsuite takes care of learning and classifying your prepared email corpora for each selected classifier. Since classifiers have widely varying interfaces, this is only possible by wrapping those interfaces individually into a standard form which can be used by mailtoe testsuite. Basically: every classifier gets a little translator so they can all speak the same language. 🌐

Each wrapper script is a command line tool which accepts a single command followed by zero or more optional arguments, in the standard form:

wrapper command [argument]...

Each wrapper script also makes use of STDIN and STDOUT in a well defined way. If no behaviour is described, then no output or input should be used. (Respect the silence, it's a boundary thing.) The possible commands are described below:

  • filter — In this case, a single email is expected on STDIN, and a list of category filenames is expected in $2, $3, etc. The script writes the category name corresponding to the input email on STDOUT. No trailing newline is required or expected. (No trailing newline! Don't add one! This is a whole thing!)
  • learn — In this case, a standard mbox stream is expected on STDIN, while a suitable category file name is expected in $2. No output is written to STDOUT. Learn it and shut up about it, respectfully.
  • clean — In this case, a directory is expected in $2, which is examined for old database information. If any old databases are found, they are purged or reset. No output is written to STDOUT. Fresh start, no baggage. 🧼
  • describe — In this case, a single line of text is written to STDOUT, describing the filter's functionality. The line should be kept short to prevent line wrapping on a terminal. Respect the terminal width, it's fragile.
  • bootstrap — In this case, a directory is expected in $2. The wrapper script first checks for the existence of its associated classifier, and other prerequisites. If the check is successful, then the wrapper is cloned into the supplied directory. A courtesy notification should be given on STDOUT to express success or failure. It is also permissible to give longer descriptions caveats.
  • toe — In this case, a list of categories is expected in $3, $4, etc. Every possible category must be listed. Preceding this list, the true category is given in $2. Get the order wrong and it's over for you, ngl.
  • foot — Used by mailfoot(1).

...and there's still more where that came from 👀

🌸 ENVIRONMENT 🌸

ok so literally right after loading, mailtoe reads the hidden file .mailtoerc in your $HOME directory — but only if it exists, bestie, don't worry — so this is the spot to define your custom values for environment variables. like, prime real estate. 💅

  • MAILTOE_FILTER — this variable holds a shell command that gets executed repeatedly during the running stage. the command should accept an email message on STDIN and output a resulting category name. also, on the command line, it should accept first the true category name, then a list of all possible category file names. and here's the tea 🫖: if the output category does not match the true category, then the relevant categories are assumed to have been silently updated/relearned. if MAILTOE_FILTER is undefined, mailtoe just uses a default value. no biggie.
  • TEMPDIR — this directory is exported for the benefit of wrapper scripts. so like, scripts that need to create temporary files should place them at the location given in TEMPDIR. that's it, that's the whole vibe. 🎀

✨ NOTES ✨

ok so heads up: the subdirectory mailtoe.d can grow quite large 😭. like, it contains a full copy of the training corpora, plus learning files for size times all the added categories, plus various log files. she's thicc. it's giving storage hog ngl.

so while TOE simulations for dbacl(1) can be used to compare with other classifiers — cute, love that for us — TOE should not be used for real world classifications. 🙈 why? because unlike many other filters, dbacl(1) learns evidence weights in a nonlinear way, and does not preserve relative weights between tokens, even if those tokens aren't seen in new emails. so yeah, simulations only, bestie. keep her in the lab. 🧚

💀 WARNING 💀

bestie, listen. because the ordering of emails within the added mailboxes matters, the estimated error rates are not well defined — or even meaningful in an objective sense. like, at all. 💀

however!! if the sample emails represent an actual snapshot of a user's incoming email, then the error rates are somewhat meaningful. in that case the simulations can be interpreted as alternate realities where a given classifier would have intercepted the incoming mail. multiverse classifier arc, honestly kind of iconic. 🌈

fr fr don't take those numbers as gospel unless your sample is a real snapshot of someone's actual inbox. perspective check, periodt.

🔥 SOURCE 🔥

the source code for the latest version of this program is available at the following locations:

http://www.lbreyer.com/gpl.html
http://dbacl.sourceforge.net

go get her 👑 (these are literal URLs, copy them exactly, don't let autocorrect touch them)

🫶 AUTHOR 🫶

Laird A. Breyer <laird@lbreyer.com>

(the mastermind behind all this, show some respect 💜)

🌟 SEE ALSO 🌟

bayesol(1) dbacl(1), mailinspect(1), mailcross(1), mailfoot(1), regex(7)

(yes that's exactly how it's written in the original, i'm not touching it — character for character, that's the rule 🎀)

aaaand that's a wrap on the MAILTOE(1) page, bestie!! 🦋 this is Version 1.14 of the page, part of the Bayesian Text Classification Tools crew — go forth, classify responsibly, and remember: TOE is for simulations, not the real world. mwah 💋✨🌸

📦 where this page comes from