MIMEDEFANG-MULTIPLEXOR(8)

📦
version 3.7.1 · 4 versions
, in epel 10, epel 9, fedora 43, fedora 44

🌸 mimedefang-multiplexor — part of the System Manager's Manual, section 8 💅

okay bestie, sit down, hydrate, and let me tell you about the little process-pool babysitter that keeps your mail filters from melting 💖

✨ NAME ✨

mimedefang-multiplexor — the process pool controller for mail filters, aka the girl who manages the whole squad 🫶

🌈 SYNOPSIS 🌈

mimedefang-multiplexor [options]

💜 DESCRIPTION 💜

mimedefang-multiplexor runs a whole pool of Perl processes that scan e-mail. It is literally designed to work hand-in-hand with mimedefang(8) and mimedefang.pl(8) — they're a trio, it's giving main character energy 👑

So here's the vibe: mimedefang-multiplexor opens a UNIX-domain socket and just... listens. Waits for mimedefang to come asking for work. As requests roll in, mimedefang-multiplexor spins up Perl processes as needed to scan the mail. And here's the iconic part: those Perl processes do not get killed when the scan is done. Nope. They keep running in a loop 🔁. They get re-used for the next e-mail message, and the next, and the next. Which means you skip the absolutely unhinged overhead of starting a brand new Perl process for every single incoming message. slay.

But — and this is the self-care part — to avoid memory leaks, the Perl processes do get killed after they've handled some number of scans. Gotta rotate the roster bestie 💅

🎀 OPTIONS 🎀

  • -U user — runs the multiplexor as user instead of root. This one is mandatory, no cap. And it must match the -U option you handed to mimedefang. They have to be twins or it's not gonna work 🙈
  • -m minWorkers — the minimum number of Perl processes to keep running at all times. Default is 0.
  • -x maxWorkers — the maximum number of Perl processes running at the same time. If a request comes in and every single process is busy, a temporary failure gets signalled to the SMTP peer. Default is 2.
  • -r maxRequests — the max number of requests one given process handles before it gets killed and a replacement is started. Default is 500.
  • -i idleTime — the idle time in seconds after which we kill off the excess Perl processes. Like, if a process is idle for longer than this, and there are more than minWorkers running, that process gets the axe. Real talk though: this is implemented as a timer that ticks every idleTime seconds, so processes can actually sit idle for up to twice this time before they're killed. The default for idleTime is 300 seconds.
  • -V maxLifetime — the maximum lifetime in seconds of a worker before it's killed and a replacement started. Default is -1, which means no maximum lifetime, live your truth. Note that the lifetime check only happens when a worker becomes idle after a request, or every time the idle-timeout check is made. So on a very quiet system, workers may live longer than maxLifetime by as much as idleTime. Also note: the lifetime is measured not from when the worker started running, but from when it was assigned its very first request. A completely-idle worker that has never processed any requests will not be terminated by the maxLifetime setting. periodt.
  • -b busyTime — the longest a Perl process is allowed to spend scanning an e-mail before it's declared hung up and killed. Default is 120 seconds.
  • -Z — says the multiplexor should accept and process "status updates" from busy workers. Heads up: this consumes one extra file descriptor per worker, plus a small amount of CPU time per status update. 💀
  • -c cmdTime — the timeout for communication between mimedefang-multiplexor and mimedefang, or between mimedefang-multiplexor and a Perl scanning process. Default is 10 seconds. Keep this one quite short, fr fr.
  • -w waitTime — when mimedefang-multiplexor starts the initial workers, or needs to bring the number of running workers up to the number defined by the -m option, it does not start them all at once, because that could overload your server (not cute). Instead it starts one worker every waitTime seconds. Default for waitTime is 3.
  • -W waitTime — if you use this option, mimedefang-multiplexor will never activate a worker until waitTime seconds have elapsed since the last worker activation. This could result in mail being tempfailed if worker activations don't keep pace with incoming mail 😭. However, it may be preferable to tempfail mail rather than let the load on your server spike up too quickly. Default value for this option is 0, meaning mimedefang-multiplexor will start workers as quickly as necessary to keep up with incoming mail.
  • -z spooldir — set the spool directory to spooldir. If you omit this option, the spool directory defaults to /var/spool/MIMEDefang.
  • -s pathName — the UNIX-domain socket on which mimedefang-multiplexor listens for requests. This should be specified as an absolute pathname. If you don't supply this option, it defaults to mimedefang-multiplexor.sock under the spool directory.
  • -a socket — a socket for listening for requests. Similar to the -s socket, except that only a restricted set of requests get processed here. On this socket, the multiplexor will only process requests asking for status; it will not accept any commands to do scanning or anything that would consume a worker. See the SOCKET SPECIFICATION section for the format of socket.
  • -p fileName — causes mimedefang-multiplexor to write its process-ID (after becoming a daemon) to the specified file. The file will be owned by root.
  • -o fileName — causes mimedefang-multiplexor to use fileName as a lock file to avoid multiple instances from running. If you supply -p but not -o, then mimedefang-multiplexor constructs a lock file by appending ".lock" to the pid file. However — and listen — this is less secure than having a root-owned pid file in a root-owned directory and a lock file writable by the user named by the -U option. (The lock file must be writable by the -U user.) 🚨
  • -f filter_path — normally, mimedefang-multiplexor executes a Perl filter script called mimedefang.pl to scan the e-mail. But you can have it execute any program you like by specifying the full path to the program with the -f option. This program must obey the protocol documented in mimedefang-protocol(7); go read that manual page for the deets.

Note that the -f option does not specify the "filter" to use with mimedefang.pl; instead, it specifies the program for mimedefang-multiplexor to execute. You almost certainly should not use this option unless you wish to replace mimedefang.pl with your own program. Like, genuinely, don't 🙈

  • -F rules_path — specifies the path to the filter rules. By default, /etc/mail/mimedefang-filter is used. If you use the -F option, its value is passed to the underlying Perl filter program using -f.
  • -l — log certain events, including the output of the Perl workers' standard-error, using syslog. Normally, the multiplexor does not log much information. (A quiet queen 🌸)
  • -d — write debugging information about event-handling code in /var/log/mimedefang-event-debug.log. This is only of use to people debugging mimedefang-multiplexor.
  • -A — log details about worker scheduling decisions using syslog. Each time a worker is selected to handle a request, a line is logged at LOG_INFO level showing the worker state, worker number, command number, and the worker's activation and last-command timestamps. This option is only useful when debugging worker-scheduling behaviour.
  • -R kbytes — limits the resident-set size of the worker filter processes to kbytes kilobytes. This limit is not supported on all operating systems; it is known to work on Linux 🐧
  • -M kbytes — limits the total memory space of worker filter processes to kbytes kilobytes. This limit is supported on all operating systems which support the setrlimit(2) system call. This should include most modern UNIX systems.

We recommend that you monitor your worker filter processes and get a feel for how much memory they use. You should then limit the memory to two or three times the worst-case that you have observed. This can help mitigate denial-of-service attacks which use complicated MIME messages to force mimedefang.pl to consume lots of memory. Protect the bag 💪

  • -h — print usage information and exit.
  • -t filename — log statistical information to filename. See the section STATISTICS for more information.
  • -T — log statistical information using syslog(2). You may use -t and -T together, in which case statistical information is logged in a file and using syslog. Best of both worlds 💖
  • -u — flush the statistics file after every write. Normally, mimedefang-multiplexor does not flush the file; this is the best choice for minimizing disk I/O on a busy mail server. However, if you wish to watch statistics entries in real-time, you should enable flushing.
  • -D — do not fork into the background and become a daemon. Instead, stay in the foreground. Useful mainly for debugging or if you have a supervisory process managing mimedefang-multiplexor.
  • -q queue_size — normally, if all workers are busy and mimedefang-multiplexor receives another request, it fails it with the error "No free workers." However, if you use the -q option, then up to queue_size requests will be queued. As soon as a worker becomes free, the queued requests get handed off in FIFO order. If the queue is full and another request comes in, then the request is failed with "No free workers".
  • -Q queue_timeout — queued requests should not stay on the queue indefinitely, that's rude. If a queued request cannot be processed within queue_timeout (default 30) seconds of being placed on the queue, it is failed with a "Queued request timed out" message. See the section "QUEUEING REQUESTS" for more discussion.
  • -O sock — listen on a notification socket for connections from listeners. mimedefang-multiplexor can inform external programs of state changes by sending messages over a notification socket. The external programs connect to this socket and then listen for notifications. See the section SOCKET SPECIFICATION for the format of sock.

See the mimedefang-notify(7) man page for details of the notification protocol. 🦋

(...and bestie, we are SO not done yet — more parts are coming, don't go anywhere) 🌈

  • -N map_sock — bestie, this one makes the multiplexor listen on a map socket so Sendmail's SOCKETMAP connections can slide in 💌. Ever since Sendmail 8.13 you can define a Sendmail map type that talks to a daemon over a socket, and mimedefang-multiplexor implements that protocol — go peek at the mimedefang-filter(5) man page for the deets (check the SOCKET MAPS section, iykyk).

For the format of map_sock, scroll down to the SOCKET SPECIFICATION section, it's all laid out there ✨.

  • -I backlog — okay so when mimedefang-multiplexor creates a listening socket, it calculates the "backlog" argument to listen(2) based on the max number of workers. But you can override that manually with -I. Setting the backlog high (like around 30-50) can genuinely help on a very busy server 🫶. And if you're seeing mail log messages like MXCommand: socket: Connection refused during busy periods? That's the universe telling you to raise your listen backlog, no cap 💅.
  • -L interval — logs the worker status every interval seconds. It writes a line through syslog that looks exactly like this:
Worker status: Stopped=s Idle=i Busy=b Killed=k Queued=q Msgs=m Activations=a

Let's decode, bestie 🎀: "Stopped" is the number of non-running workers, "Idle" is the number of idle workers, "Busy" is the number of busy workers, "Killed" is the number of killed workers yet to be reaped, "Queued" is the number of queued requests, "Msgs" is the total number of messages processed since the multiplexor began running, and "Activations" is the number of times a Perl process has been started since the multiplexor began running.

If you hand it an interval of 0 (which is the default, btw), no periodic status logging happens at all. And if you supply an interval of less than 5 seconds, it gets silently reset to 5 seconds 🙈.

  • -S facility — sets the syslog facility for log messages. Default is mail. Check openlog(3) for the list of valid facilities. You can use either the short name ("mail") or the long name ("LOG_MAIL") for the facility name, whatever vibes ⭐.
  • -E — tells the multiplexor to create an embedded Perl interpreter. This can improve performance dramatically 🔥. But fr fr, read the EMBEDDING PERL section for more info before you go wild.
  • -X n — tells the multiplexor to initiate a "tick" request every n seconds. This makes your filter_tick function get called (if you defined it, that is). Heads up though: you have zero control over which worker executes filter_tick 😭. And if all workers are busy when a tick happens, that tick request is skipped and a warning message gets logged.
  • -P n — tells the multiplexor to run n tick requests in parallel. Each tick runs as often as specified with the -X argument. (If you skip the -P option, then the multiplexor just behaves as if -P 1 had been specified.)

If you run parallel ticks, each tick gets assigned an integer identifying its "type". The type ranges from 0 to n-1. And while there may be as many as n tick requests running at a time, only one tick of each type will be active at any given moment 👑.

  • -Y label — sets the tag used in the multiplexor's syslog messages to label instead of mimedefang-multiplexor. Cute customization, obsessed 💖.
  • -k — enables adaptive autoscaling of the worker pool ✨. When you supply this option, mimedefang-multiplexor periodically computes an exponential moving average (EMA) of the busy-worker ratio and automatically starts or stops workers to match demand:
    • If the EMA busy ratio exceeds 80%, an additional worker is activated (up to maxWorkers, as set by -x).
    • If the EMA busy ratio falls below 30%, an excess worker is killed (down to minWorkers, as set by -m).

The check is performed every 15 seconds. And to avoid thrashing, a scale-out event cannot occur within 5 seconds of the previous scale-out, and a scale-in event cannot occur within 30 seconds of the previous scale-out or scale-in.

The current autoscaling state (enabled flag, EMA value, and all the thresholds) can be inspected at any time with the autoscale command via md-mx-ctrl(8) 🦋.

  • -G — normally, mimedefang-multiplexor uses a umask of 027 when creating listening sockets. But if you want the sockets to be readable and writeable by the group as well as the owner, supply the -G option. This makes the umask 007 whenever UNIX-domain sockets are created 🎀.
  • -y n — limits the maximum number of concurrent recipok checks to n on a per-domain basis. The value of n can range from 0 (in which case no limit is applied) to maxWorkers, where maxWorkers is the argument to the -x option. If n is outside that range, it gets ignored (and, yep, no limit is applied).

The recipok command ultimately invokes the filter_recipient function in your filter. So if you're doing recipient verification against servers that may be slow or unreliable, you can use -y to cap the number of concurrent recipient verifications per domain 🙌. That way, if one domain's server goes super slow, it won't consume all your available workers for recipient verification. Instead, its RCPT commands will be tempfailed and there will be workers available to handle RCPT commands for other domains. Slay 💅.

🌸 SOCKET SPECIFICATION 🌸

The -a, -N and -O options take a socket as an argument. The format of the socket parameter is similar to that of the Sendmail Milter library, and it's one of the following:

  • /path/to/socket — a UNIX-domain socket 🫶
  • inet:portnum — a TCP socket bound to port portnum, but which accepts connections only from the IPv4 loopback address (127.0.0.1).
  • inet_any:portnum — a TCP socket bound to port portnum which will accept connections from any address. Use inet_any with caution! ⚠️
  • inet6:portnum — a TCP socket bound to port portnum listening on the IPv6 loopback address.
  • inet6_any:portnum — a TCP socket bound to port portnum listening on the IPv6 wildcard address.

🎀 QUEUEING REQUESTS 🎀

Normally, if all workers are busy, any additional requests get failed immediately. But the -q and -Q options let you queue requests for a short amount of time ⏳. This facility is meant to gracefully handle a temporary overload — most of the time, your queue should be empty, bestie.

Because mimedefang checks the number of free workers when a connection is opened and fails the connection if there are no free workers, the whole intent of the queue is to let SMTP transactions that are already underway keep going if there's a slight overload. Any new connections will be failed if all workers are busy, but existing connections are allowed to continue. Queuing requests may improve throughput on extremely busy servers 🔥.

Note that if you supply the -q option to mimedefang, then even new connections are allowed to queue. This may improve throughput by keeping the worker utilization higher ✨.

The -R option to mimedefang can be used to reserve a specified number of workers for connections from the loopback address. Using -R has the side-effect of permitting new connections from the loopback address to queue 👀.

🧚 EMBEDDING PERL 🧚

Normally, when mimedefang-multiplexor activates a worker, it forks and execs mimedefang.pl. However, if the multiplexor was compiled with embedded Perl support, and you supply the -E command-line option, the multiplexor works like this:

  1. It creates an embedded Perl interpreter, and sources mimedefang.pl with a special command-line argument telling it to read the filter, but not to enter the main loop.
  2. Each time a worker is activated, the multiplexor calls fork() and runs the mimedefang.pl main loop. This invokes filter_initialize and then runs the main loop.

On some platforms (for example, Red Hat Linux 7.3 with Perl 5.6.1), it is not safe to destroy and recreate a Perl interpreter without causing a memory leak 💀. On those platforms, if you attempt to reread the filter file (by sending the multiplexor a HUP signal or reread command), the filter will not be re-read, and a message will be logged to syslog. On those platforms, you must kill and restart MIMEDefang if you change the filter file. Yeah, annoying, we know 😭.

On most platforms, however, a filter reread is accomplished by destroying and re-creating the embedded interpreter, re-sourcing mimedefang.pl and killing workers as soon as they are idle. Iconic behaviour, honestly 👑.

✨ STATISTICS ✨

ok so when you run with the -t option, mimedefang-multiplexor logs certain events to a file. that file is literally a glow-up waiting to happen — you post-process it and suddenly you have stats about your multiplexor 💅. use it to tune how many workers you run, adjust your timeouts, whatever. it's giving data-driven queen.

every single line of that file looks like this:

YYYY/MM/DD:HH:MM:SS timestamp event key=val key=val...

here's the tea on each piece:

  • YYYY/MM/DD:HH:MM:SS is the local time of day, bestie.
  • Timestamp is the number of seconds since January 1, 1970.
  • Event is the name of an event.

and the valid events are:

  • StartWorker — a worker process has been started. baby's first shift 🎀
  • KillWorker — a worker process has been killed.
  • ReapWorker — a dead worker process has been reaped. heads up: it is possible to get a ReapWorker event without a previous KillWorker event, if the worker process terminated abnormally. it just does that sometimes, no cap 😭
  • StartFilter — a worker process has begun filtering an e-mail message.
  • EndFilter — a worker process has finished filtering an e-mail message. ate and left no crumbs 🔥

now the possible keys in those key=value pairs:

  • worker=n — the worker involved in the event. every worker is identified by a small integer, like a little ID badge 🫶
  • nworkers=n — the total number of running workers immediately after the event happened.
  • nbusy=n — the number of busy workers (aka the ones actually processing an e-mail message) immediately after the event happened.
  • reason="string" — the reason for a StartWorker or KillWorker event. (present only for these events, so don't go looking for it elsewhere 💀)
  • numRequests=n — the number of e-mails processed by the worker. present only for an EndFilter event.

ok now the signal section, because this is where it gets spicy 🌶️

if you send the mimedefang-multiplexor process a SIGHUP signal (kill -1 pid), it closes and reopens the statistics file. this is useful during log file rotation. classic move, iconic.

if you send the mimedefang-multiplexor process a SIGINT signal (kill -INT pid), it terminates all active-but-idle workers. also, any active-and-busy workers terminate as soon as they finish filtering the current message. this is useful to force a reread of the filter rules file without stopping and restarting Sendmail. chef's kiss, genuinely so handy ✨

if you send the mimedefang-multiplexor process a SIGTERM signal (kill pid), it terminates all workers and exits immediately. abrupt? yes. effective? also yes 💀

🌸 AUTHOR 🌸

mimedefang-mulitplexor was written by Dianne Skoll <dfs@roaringpenguin.com>. absolute legend behaviour, we stan 👑

the mimedefang home page is https://www.mimedefang.org/ — go say hi 🦋

💖 SEE ALSO 💖

ok bestie that's the whole page, you made it to the end and you understood all of it, main character energy fr fr 🎀

this lil masterpiece is from the 4th Berkeley Distribution, dated 8 February 2005, and the page itself is MIMEDEFANG-MULTIPLEXOR(8) — go forth and multiplex responsibly, mwah 💅✨

📦 where this page comes from