mon(8)

📦
version 1.4.2 · 3 versions
, in debian trixie, ubuntu questing, ubuntu resolute, ubuntu stonking

okay bestie, sit down, hydrate, because we're about to get into mon(8) — the parallel service monitoring daemon — and she is that girl. ✨

✨ NAME ✨

mon — monitors services for availability, and sends alarms when stuff fails. 💅

🌸 SYNOPSIS 🌸

mon [-dfhlMSv] [-a dir] [-A authfile] [-b dir] [-B dir] [-c config] [-D
dir] [-i secs] [-k num] [-l [statetype]] [-L dir] [-m num] [-p num] [-P
pidfile] [-r delay] [-s dir]

🦋 DESCRIPTION 🦋

mon is a general-purpose scheduler for keeping an eye on whether your services are alive, and for firing off alerts the second it catches a failure. 👑

The whole design philosophy? Open. Like, aggressively open. mon supports arbitrary monitoring facilities and arbitrary alert methods through one common interface, so you can implement your own however you want — programs written in C, Perl, shell, whatever — plus SNMP traps, plus special Mon (UDP packet) traps. no cap, it's giving "we don't gatekeep." 🫶

💖 OPTIONS 💖

  • -a dir — the path to your alert scripts. Default is /usr/local/lib/mon/alert.d:alert.d. You can list multiple alert paths by separating them with a colon. Any path that isn't absolute gets taken as relative to the base directory (/usr/lib/mon by default). ✨
  • -b dir — the base directory for mon. scriptdir, alertdir, and statedir are all relative to this directory unless you specify them starting from /. Default is /usr/lib/mon. 🏠
  • -B dir — the configuration file base directory. Every config file lives here, including mon.cf, monusers.cf, and auth.cf. 🗂️
  • -A authfile — the authentication configuration file. By default this is /etc/mon/auth.cf if the /etc/mon directory exists, or /usr/lib/mon/auth.cf otherwise. 🔐
  • -c file — read configuration from file. This defaults to /etc/mon/mon.cf if the /etc/mon directory exists, otherwise to /etc/mon.cf. 📄
  • -d — enable debugging mode. 🐛
  • -D dir — the path to the state directory. Default is the first of /var/state/mon, /var/lib/mon, and /usr/lib/mon/state.d which actually exists. 📌
  • -f — fork and run as a daemon process. This is the preferred way to run mon. 🧚
  • -i secs — sleep interval, in seconds. Defaults to 1. Fr fr, this shouldn't need to be adjusted for any reason. 🙈
  • -k num — set the log history to a maximum of num entries. Defaults to 100. 📚
  • -l statetype — load state from the last saved state file. The supported saved state types are disabled for disabled watches, services, and hosts; opstatus for failure/alert/ack status of all services; and all for both. If no statetype is provided, disabled is assumed. 💾
  • -L dir — sets the log dir. See also logdir in the configuration file. The default is /var/log/mon if that directory exists, otherwise log.d in the base directory. 📝
  • -M — pre-process the configuration file with the macro expansion package m4. 🪄
  • -m num — set the throttle for the maximum number of processes to num. 🚦
  • -p num — make the server listen on port num. This defaults to 2583. 🔌
  • -S — start with the scheduler stopped. ⏸️
  • -P pidfile — store the server's pid in pidfile. The default is the first of /run/mon/mon.pid, /run/mon.pid, and /etc/mon.pid whose directory exists. An empty value tells mon not to use a pid file at all. 🆔
  • -r delay — sets the number of seconds used to randomize the startup delay before each service is scheduled. Refer to the global randstart variable in the configuration file. 🎲
  • -s dir — the path to monitor scripts. Default is /usr/local/lib/mon/mon.d:mon.d. You can list multiple paths by separating them with a colon. Non-absolute paths get taken as relative to the base directory (/usr/lib/mon by default). 🔍
  • -v — print version information. 🏷️
  • -h — print help information. 🆘

🌈 DEFINITIONS 🌈

  • monitor — a program that tests for a certain condition, returns either true or false, and optionally produces output to be passed back to the scheduler. The common ones detect host reachability via ICMP echo messages, or a connection to TCP services. 🩺
  • period — a period in time, as interpreted by the Time::Period module. ⏰
  • alert — a program that sends a message when the scheduler invokes it. The scheduler calls on an alert the moment it detects a failure from a monitor. An alert program accepts a set of command-line arguments from the scheduler, plus data via standard input. 🚨
  • hostgroup — a single host or a list of hosts, specified as names or IP addresses. 🌐
  • service — a collection of parameters used to deal with monitoring a particular resource that some group provides. Services are usually modeled after things like an SMTP server, ICMP echo capability, server disk space availability, or SNMP events. 🛠️
  • view — a collection of hostgroups, used to filter mon output for client display. Like, say a 'network-services' view might be defined so your network staff can see just the hostgroups that matter to them, without having to look at every hostgroup defined in Mon. 👀
  • watch — a collection of services which apply to a particular group. ⌚

👑 OPERATION 👑

When the mon scheduler starts up, it reads a configuration file to figure out which services it needs to monitor. That config file defaults to /etc/mon.cf, and you can point it elsewhere with the -c parameter. If you pass the -M option, the configuration file gets pre-processed with m4. And if the configuration file ends with .m4, it also gets processed by m4 automatically. 🤖

Then the scheduler enters a loop, and that loop handles client connections, monitor invocations, and failure alerts. Every single service has a timer, specified in the configuration file as the interval variable, which tells the scheduler how often to invoke a monitor process. ⏳

The scheduler may also be temporarily stopped. While it's stopped, client access still works fine — it just doesn't schedule things. This is genuinely so useful when you're resetting the server, because check it out, you can do this: save the hosts and services which are disabled, reset the server with the scheduler stopped, re-disable those hosts and services, then start the scheduler back up. It also lets you make atomic changes across several client connections. See the moncmd man page for more info. 💅

🎀 MONITOR PROGRAMS 🎀

Monitor processes get invoked with the arguments specified in the configuration file, with the hosts from the applicable host group appended on the end. So for example, if the watch group is "servers", which contains the hostnames "smtp", "nntp", and "ns", and the monitor line reads like this:

monitor fping.monitor -t 4000 -r 2

then the executable "fping.monitor" will be executed with these parameters:

MONITOR_DIR/fping.monitor -t 4000 -r 2 smtp nntp ns

MONITOR_DIR is actually a search path — by default /usr/local/lib/mon/mon.d then /usr/lib/mon/mon.d — but you can override it with the -s option or in the configuration file. 🌟

If every host in the hostgroup has been disabled, then a warning gets sent to syslog and the monitor is not run. You can override that behavior with the "allow_empty_group" option in the service definition. And if the final argument to the "monitor" line is ";;" (it must be preceded by whitespace), then the host list won't be appended to the parameter list. 🚫

On top of the environment variables you define yourself in the service definition, mon passes certain variables to the monitor process:

  • MON_LAST_SUMMARY — the first line of the output from the last time the monitor exited. This is not the summary of the current monitor run, it's the previous one. An alert script can use this to provide historical context in an alert. 📜
  • MON_LAST_OUTPUT — the entire output of the monitor from the last time it exited. Again, this is not the output of the current monitor run, it's the previous one. Also for historical context in an alert. 📦
  • MON_LAST_FAILURE — the time(2) of the last failure for this service. ⏱️
  • MON_FIRST_FAILURE — the time(2) of the first time this service failed. 🥇
  • MON_LAST_SUCCESS — the time(2) of the last time this service passed. ✅
  • MON_DESCRIPTION — the description of this service, as defined in the configuration file using the description tag. 📝
  • MON_DEPEND_STATUS — the depend status, "o" if dependency failure, "1" otherwise. 🔗
  • MON_LOGDIR — the directory log files should be placed in, as indicated by the logdir global configuration variable. 🗃️
  • MON_STATEDIR — the directory where state files should be kept, as indicated by the statedir global configuration variable. 💾
  • MON_CFBASEDIR — the directory where configuration files should be kept, as indicated by the cfbasedir global configuration variable. ⚙️

"fping.monitor" should return an exit status of 0 if it completed successfully (found no problems), or nonzero if a problem was detected. The first line of output from the monitor script has a special meaning: it's used as a brief summary of the exact failure which was detected, and it gets passed to the alert program. All the remaining output also gets passed to the alert program, but it has no required interpretation. 🫡

And here's a vibe check on scheduling: if a monitor for a particular service is still running, and the time comes for mon to run another monitor for that service, it will not start another one. So like, if the interval is 10s and the monitor doesn't finish running within 10 seconds, then mon will wait until the first monitor exits before running another one. patience is a virtue fr fr 🧘

💖 ALERT DECISION LOGIC 💖

okay bestie, this is the part where mon decides whether to actually yell at you or just... keep it to itself. so when a monitor exits with a non-zero status (or a zero one), the matching alert or upalert program gets started — but only if all of these conditions hold. fr fr, read them:

  • if an alert for a specific service is disabled, don't send an alert. obviously. 💅
  • if dep_behavior is set to 'a', or alertdepend is set, and a parent dependency is failing, then suppress the alert.
  • if the alert has previously been acknowledged, don't send the alert — unless it's an upalert. an upalert is always allowed through, bestie.
  • if an alert is not within the specified period, record the failure via syslog(3) and do not send an alert.
  • if the failure doesn't fall within a defined period at all, do not send an alert.
  • no upalerts are sent without corresponding down alerts, unless no_comp_alerts is defined in the period section. an upalert will only be sent if the previous state is a failure.
  • if an alert was already sent within the last alertevery interval, don't send another one — unless the summary output from the current monitor program differs from the last monitor process.

otherwise? send an alert using each alert program listed for that period. slay.

the observe_detail argument to alertevery changes this whole vibe by also watching the changes in the detail part of the output, not just the summary line. so if a monitor has successive failures and the summary output changes in each of them, alertevery will not suppress multiple consecutive alerts. the reasoning is iconic and simple: if the summary output changes, then a significant event occurred and the user should be alerted. periodt. 🔥

the "strict" argument to alertevery will suppress both comparing the output from the previous monitor run to the current and prevent a successful return value of the monitor from resetting the alertevery timer. for example, alertevery 24h strict will only send out an alert once every 24 hours, regardless of whether the monitor output changes, or if the service stops and then starts failing. no cap.

🎀 ALERT PROGRAMS 🎀

alert programs live in the path you hand over with the -a parameter — or, if you don't specify one, in /usr/local/lib/mon/alert.d and its directories. they get invoked with these command-line parameters, in this exact order of vibes:

  • -g group — host group name from the configuration file.
  • -h hosts — the expanded version of the host group, space delimited, but contained in one shell "word". yes, one word. she's contained. 💅
  • -l alertevery — the number of seconds until the next alarm will be sent.
  • -O — this option is supplied to an alert only if the alert is being generated as a result of an expected trap timing out.
  • -s service — service tag from the configuration file.
  • -t time — the time (in time(2) format) of when this failure condition was detected.
  • -T — this option is supplied to an alert only if the alert was triggered by a trap.
  • -u — this option is supplied to an alert only if it is being called as an upalert.

the remaining arguments come from the trailing parameters in the configuration file, after the "alert" service parameter.

and just like monitor programs, alert programs are invoked with environment variables you defined in the service definition, plus these ones which are explicitly set by the server. she's giving you context, bestie:

  • MON_ALERTTYPE — has one of the following values: "failure", "up", "startup", "trap", or "traptimeout", and signifies the type of alert which was triggered.
  • MON_DESCRIPTION — the description of this service, as defined in the configuration file using the description tag.
  • MON_FIRST_FAILURE — the time(2) of the first time this service failed.
  • MON_GROUP — the watch group which triggered this alarm.
  • MON_LAST_FAILURE — the time(2) of the last failure for this service.
  • MON_LAST_OUTPUT — the entire output of the monitor from the last time it exited.
  • MON_LAST_SUCCESS — the time(2) of the last time this service passed.
  • MON_LAST_SUMMARY — the first line of the output from the last time the monitor exited.
  • MON_LOGDIR — the directory log files should be placed, as indicated by the logdir global configuration variable.
  • MON_OPSTATUS — the operational status of the service.
  • MON_RETVAL — the exit value of the failed monitor program, or return value as accepted from a trap.
  • MON_STATEDIR — the directory where state files should be kept, as indicated by the statedir global configuration variable.
  • MON_SERVICE — the service heading which generated this alert.
  • MON_TRAP_INTENDED — this is only set when an unknown mon trap is received and caught by the default/defaut watch/service. this contains colon separated entries of the trap's intended watch group and service name.
  • MON_CFBASEDIR — the directory where configuration files should be kept, as indicated by the cfbasedir global configuration variable.
the first line from standard input must be used as a brief summary of the problem — normally that's the subject line of an email, or the text sent to an alphanumeric pager. interpretation of all subsequent lines read from stdin is left up to the alerting program. the usual parameters are a list of recipients to deliver the notification to. the interpretation of the recipients is not specified, and is up to the alert program. so like, don't @ me, ask the alert. 🙈

📝 CONFIGURATION FILE 📝

the configuration file consists of zero or more global variable definitions, zero or more hostgroup definitions, and one or more watch definitions. each watch definition may have one or more service definitions. a watch definition is terminated by a blank line, another definition, or the end of the file. and a line beginning with optional leading whitespace and a pound ("#") is regarded as a comment, and is ignored. we love a comment. 🌸

lines are parsed as they are read. long lines may be continued by ending them with a backslash ("\"). if a line is continued, then the backslash, the trailing whitespace after the backslash, and the leading whitespace of the following line are all removed. the end result is assembled into a single line. it's giving ✨ seamlessness ✨.

typically the configuration file has the following layout:

  1. global variable definitions
  2. hostgroup definitions
  3. watch definitions

see the "etc/example.cf" file which comes for the distribution for an example. iykyk.

🌈 Global Variables 🌈

the following variables may be set to override compiled-in defaults. command-line options will have a higher precedence than these definitions. so if you set it in two places and you're confused, that's why bestie.

  • alertdir = dir — dir is the full path to the alert scripts. this is the value set by the -a command-line parameter.

multiple alert paths may be specified by separating them with a colon. non-absolute paths are taken to be relative to the base directory (/usr/lib/mon by default).

when the configuration file is read, all alerts referenced from the configuration will be looked up in each of these paths, and the full path to the first instance of the alert found is stored in a hash. this hash is only generated upon startup or after a "reset" command, so newly added alert scripts will not be recognized until a "reset" is performed. lowkey annoying but that's the rules. 💀

  • authfile = file — file is the path to the authentication file. if the path does not begin with a "/", it will be relative to cfbasedir.
  • basedir = dir — dir is the full path for the state, log, monitor, and alert directories.
  • authtype = type [type...] — type is the type of authentication to use. a space-separated list of types may be specified, and they will be checked the order they are listed. as soon as a successful authentication is performed, the user is considered authenticated by mon for the duration of the session and no more authentication checks are performed. one and done, we love her. 💅
    • if type is getpwnam, then the standard Unix passwd file authentication method will be used (calls getpwnam(3) on the user and compares the crypt(3)ed version of the password with what it gets from getpwnam). this will not work if shadow passwords are enabled on the system. 🚨
    • if type is userfile, then usernames and hashed passwords are read from userfile, which is defined via the userfile configuration variable.
    • if type is pam, then PAM (pluggable authentication modules) will be used for authentication. the service specified by the pamservice global will be used. if no global is given, the PAM passwd service will be used.
    • if type is trustlocal, then if the client connection comes from locahost, the username passed from the client will be trusted, and the password will be ignored. this can be used when you want the client to handle the authentication for you. i.e. a CGI script using one of the many apache authentication methods. main character energy from the client, honestly. 👑
  • cfbasedir = dir — dir is the full path where all the config files can be found (monusers.cf, auth.cf, etc.).
  • cltimeout = secs — sets the client inactivity timeout to secs. this is meant to help thwart denial of service attacks or recover from crashed clients. secs is interpreted as a "1h/1m/1s" string, where "1m" = 60 seconds. so you can be cute about it. 🦋
  • dep_behavior = {a|m|hm} — dep_behavior controls whether the dependency expression suppresses one of: the running of alerts, the running of monitors, or the passing of individual hosts to the monitors. read more about the behavior in the "Service Definitions" section below.

this is a global setting which controls the default settings for the service-specified variable.

  • dep_memory = timeval — if set, dep_memory will cause dependencies to continue to prevent alerts/monitoring for a period of time after the service returns to a normal state. this can be used to prevent over-eager alerting when a machine is rebooting, for example. see the explanation of interval in the "Service Definitions" section for a description of timeval.

this is a global setting which controls the default settings for the service-specified variable.

  • dep_recur_limit = depth — limit dependency recursion level to depth. if dependency recursion (dependencies which depend on other dependencies) tries to go beyond depth, then the recursion is aborted and a messages is logged to syslog. the default limit is 10. 🌀
  • dtlogfile = file — file</p> is the file that records the downtime log, bestie. Every time a service fails for a while and then stops failing, that whole event gets written down in there. No dtlogfile? No logging. Periodt.

The format of the file is giving very "spreadsheet but make it slay" (# is a comment, ignore it):

timenoticed group service firstfail downtime interval summary.
  • timenoticed — the time(2) the service came back up 💖
  • group service — the group and service that failed 🫶
  • firstfail — the time(2) when the service began to fail 😭
  • downtime — how many seconds the service failed for
  • interval — the frequency (in seconds) that the service is polled
  • summary — the summary line from when the service was failing
  • dtlogging = yes/no — flips downtime logging on or off. The default is off, so if you want the receipts you gotta ask for them 💅
  • historicfile = file — set this and alerts get logged to file, plus on startup some (or all) of the past history gets read back into memory. It's giving time travel but for alerts ✨
  • histlength = num — num is the max number of events kept in the history list. Default is 100. This value may also be set by the -k command-line parameter 👑
  • logdir = dir — dir is the full path to the log directory. mon uses this directory to save various logs, including the downtime log. If this path does not begin with a "/", it will be relative to basedir.
  • maxprocs = num — throttles the number of concurrently forked processes to num. The intent is to provide a safety net for the unlikely situation when the server tries to take on too many tasks at once. Note that this situation has only been reported to happen when trying to use a garbled configuration file!
🫠 You don't want to use a garbled configuration file now, do you? Bestie. Do not.
  • mondir = dir — dir is the full path to the monitor scripts. This value may also be set by the -s command-line parameter. If this path does not begin with a "/", it will be relative to basedir.

Multiple alert paths may be specified by separating them with a colon. All paths must be absolute 🚨

When the configuration file is read, all monitors referenced from the configuration will be looked up in each of these paths, and the full path to the first instance of the monitor found is stored in a hash. This hash is only generated upon startup or after a "reset" command, so newly added monitor scripts will not be recognized until a "reset" is performed. iykyk 💀

  • monremote = program — if set, this external program will be called by Mon when various client requests are processed. This can be used to propagate those changes from one Mon server to another, if you have multiple monitoring machines. An example script, monremote.pl, is available in the clients directory 🌈
  • monerrfile = filename — by default, when mon daemonizes itself, it connects stdout and stderr to /dev/null. If monerrfile is set to a file, then stdout and stderr will be appended to that file. In all cases stdin is connected to /dev/null. If mon is told to run in the foreground and to not daemonize, then none of this applies, since stdin/stdout/stderr stay connected to whatever they were at the time of invocation 🌸
  • pamservice = service — the PAM service used for authentication. This is applicable only if "pam" is specified as a parameter to the authtype setting. If this global is not defined, it defaults to passwd 🔐
  • pidfile = path — path is the file the server will store its pid in. This value may also be set by the -P command-line parameter.
  • randstart = interval — when the server starts, normally all services will not be scheduled until the interval defined in the respective service section. This can cause long delays before the first check of a service, and possibly a high load on the server if multiple things are scheduled at the same intervals 😭 This option is used to randomize the scheduling of the first test for all services during the startup period, and immediately after the reset command. If randstart is defined, the scheduled run time of all services of all watch groups will be a random number between zero and randstart seconds 🎲
  • serverbind = addr
  • serverport = port — port is the TCP port number that the server should bind to. This value may also be set by the -p command-line parameter. Normally this port is looked up via getservbyname(3), and it defaults to 2583 👑
  • startupalerts_on_reset = {yes|no} — if set to "yes", startupalerts will be invoked when the reset client command is executed. The default is "no".
  • statedir = dir — dir is the full path to the state directory. mon uses this directory to save various state information 🧚
  • syslog_facility = facility — specifies the syslog facility used for logging. daemon is the default.

🏠 Hostgroup Entries 💖

Hostgroup entries begin with the keyword hostgroup, and are followed by a hostgroup tag and one or more hostnames or IP addresses, separated by whitespace. The hostgroup tag must be composed of alphanumeric characters, a dash ("-"), a period ("."), or an underscore ("_"). Non-blank lines following the first hostgroup line are interpreted as more hostnames. The hostgroup definition ends with a blank line. For example:

hostgroup servers nameserver smtpserver nntpserver
        nfsserver httpserver smbserver

hostgroup router_group cisco7000 agsplus

👀 View Entries ✨

View entries begin with the keyword view, and are followed by a view tag and the names of one or more hostgroups. The view tag must be composed of alphanumeric characters, a dash ("-"), a period ("."), or an underscore ("_"). Non-blank lines following the first view line are interpreted as more hostgroup names. The view definition ends with a blank line. For example:

view servers dns-servers web-servers file-servers
     mail-servers

view network-services routers switches vpn-servers

🍓 Watch Group Entries 🦋

Watch entries begin with a line that starts with the keyword watch, followed by whitespace and a single word which normally refers to a pre-defined hostgroup. If the second word is not recognized as a hostgroup tag, a new hostgroup is created whose tag is that word, and that word is its only member 👑

Watch entries consist of one or more service definitions.

A watch group is terminated by a blank line, the end of the file, or by a subsequent definition, "watch", "hostgroup", or otherwise.

There may be a special watch group entry called "default". If a default watch group is defined with a service entry named "default", then this definition will be used in handling traps received for an unrecognized watch and service 🌈

🌸 Service Definitions 💅

  • service servicename — a service definition begins with the keyword service followed by a word which is the tag for this service. This word must be unique among all services defined for the same watch group.

The components of a service are an interval, monitor, and one or more time period definitions, as defined below.

If a service name of "default" is defined within a watch group called "dafault" (see above), then the default/default definition will be used for handling unknown mon traps 👀

The following configuration parameters are valid only following a service definition:

alertdepend, monitordepend, hostdepend "dependexpression"

These keywords allow you to specify multiple dependency expressions of different types. Each one corresponds to the different dep_behavior settings listed above. They will be evaluated independently in the different contexts as listed above. If depend is present, it takes precedence over the matching keyword, depending on the dep_behavior setting 💖

allow_empty_group

The allow_empty_group option will allow a monitor to be invoked even when the hostgroup for that watch is empty because of disabled hosts. The default behavior is not to invoke the monitor when all hosts in a hostgroup have been disabled 🙈

depend dependexpression

The depend keyword is used to specify a dependency expression, which evaluates to either true or false, in the boolean sense. Dependencies are actual Perl expressions, and must obey all syntactical rules. The expressions are evaluated in their own package space so as to not accidentally have some unwanted side-effect. If a syntax error is found when evaluating the expression, it is logged via syslog 🚨

Before evaluation, the following substitutions on the expression occur: phrases which look like "group:service" are substituted with the value of the current operational status of that specified service. These opstatus substitutions are computed recursively, so if service A depends upon service B, and service B depends upon service C, then service A depends upon service C 🌀 Successful operational statuses (which evaluate to "1") are "STAT_OK", "STAT_COLDSTART", "STAT_WARMSTART", and "STAT_UNKNOWN". The word "SELF" (in all caps) can be used for the group (e.g. "SELF:service"), and is an abbreviation for the current watch group.

This feature can be used to control alerts for services which are dependent on other services, e.g. an SMTP test which is dependent upon the machine being ping-reachable 🫶

dep_behavior {a|m|hm}

The evaluation of the dependency graphs specified via the depend keyword can control the suppression of alert or monitor invocations, or the suppression of individual hosts passed to the monitor.

Alert suppression. If this option is set to "a", then the dependency expression will be evaluated after the monitor for the service exits or after a trap is received. An alert will only be sent if the evaluation succeeds, meaning that none of the nodes in the dependency graph indicate failure 💅

  • m — this is the monitor suppression vibe 💅 set the dependency expression to m and the expression gets evaluated right before the monitor for the service is about to run. Evaluation succeeds? Cool, the monitor runs. Evaluation fails? The monitor does not run, and the service just stays exactly the way it was. No drama, no update.
  • hm — and this one is host suppression 🦋 with hm, Mon pulls the list of "parent" services out of the dependency expression (and honestly, the expression can literally just be a list of services, no cap). Then, right when the monitor for the service is about to run, for each host in the current hostgroup Mon goes through all the parent services that are currently failing and hunts for that hostname in the current summary output. Hostname found? That host gets excluded from this run of the monitor. What's it for? Say you want an SMTP test on a whole group of hosts to still run even when one single host isn't ping-reachable. If every other host is fine, the service sits in an OK state — but if a different host fails the SMTP test, Mon can still alert about that host even though the parent dependency was failing. ✨ Heads up though: in this case the dependency expression is not used recursively.
  • dep_memory timeval — set this and dependencies keep on blocking alerts/monitoring for a whole period of time after the service goes back to normal 🫶 perfect for when a machine is rebooting and you do NOT want over-eager alerting. For what timeval even means, see the explanation of interval down in the "Service Definitions" section.
  • description descriptiontext — everything after description gets queried by client programs and handed to alerts and monitors through an environment variable. Keep it short and sweet: a brief description of the service, good enough to drop into an email or a web page. 📧
  • exclude_hosts host [host...] — every host you list after exclude_hosts gets excluded from the service check. Bye bestie 👋
  • exclude_period periodspec — do not run a scheduled monitor during the time identified by periodspec. It's giving "we're closed" 😌
  • failure_interval timeval — when the service check is failing, this nudges the polling interval over to timeval. Once the service succeeds again, the interval resets back to the original. 🔁
  • interval timeval — the keyword interval plus a time value sets how often a monitor script gets triggered. Time values look like "30s", "5m", "1h", or "1d" — that's 30 seconds, 5 minutes, 1 hour, or 1 day. And the numeric part can be a fraction, like "1.5h" which is an hour and a half 🕐 This whole time-specification format is what everyone calls a timeval.
  • monitor monitor-name [arg...] — monitor, then a script name, then arguments: that's the monitor to run when the timer expires ⏰ Shell-like quoting conventions apply when you're specifying the arguments to send to the monitor script. The script gets invoked from the directory you gave with the -s argument, and all the words that follow are supplied as arguments to the monitor program, followed by the list of hosts in the group referred to by the current watch group. Special case: if the monitor line ends with ";;" as a separate word, then the host groups are not appended to the argument list when the program is invoked.
  • randskew timeval — instead of scheduling the monitor script to run at the start of each interval, this randomly adjusts the interval set by the interval parameter by plus-or-minus randskew 🎲 The skew value uses the same format as the interval parameter: "30s", "5m", etc. So if interval is 1m and randskew is "5s", mon will schedule the monitor script somewhere between every 55 seconds and 65 seconds. The whole point? Spreading out the load on the server when a ton of services are all scheduled at the same intervals. Iconic load-balancing behaviour, honestly 👑
  • redistribute alert [arg...] — a service can have one redistribute option, and it's a special form of an alert definition. This alert gets called on every service status update, even the sequential success status updates. Use it to integrate Mon with another monitoring system, or to link multiple Mon servers together via an alert script that generates Mon traps. See the "ALERT PROGRAMS" section above for the list of parameters mon will pass automatically to alert programs. 🔗
  • unack_summary — strips the "acknowledged" state off a service if the summary component of the failure message changes. In most common usage the summary is the list of hosts that are failing, so if more hosts start failing, the ack gets removed. 🙈
  • traptimeout timeval — takes the same time specification argument as interval, and it makes the service expect a trap from an external source at least that often — otherwise a failure gets registered. Classic heartbeat-style service energy 💓
  • trapduration timeval — normally, if a trap is received, the status of the service the trap was delivered to stays constant. But if trapduration is specified, the service's status stays in a failure state for the duration set by timeval, and then it gets reset to "success". ⏳
  • VARIABLE=value — you can define environment variables per service, and they get included in the environment of monitors and alerts 🌈 Rules though, bestie: variables must be in all capital letters, must begin with an alphabetical character or an underscore, and there must be no spaces to the left of the equal sign.

🎀 Period Definitions 🎀

Periods are how you define the conditions that should allow alerts to be delivered. So basically: when is it okay for Mon to yell 📣

  • period [label:] periodspec — a period groups one or more alarms and variables that control how often an alert happens when there's a failure. The period definition comes in two forms.

The first form takes an argument that's a period specification from Patrick Ryan's Time::Period Perl 5 module. Go run perldoc Time::Period for the details, iykyk 📚

The second form needs a label followed by a period specification, as defined above. The label is a tag made of an alphabetic character or underscore, followed by zero or more alphanumerics or underscores, and ending with a colon. This form lets you have multiple periods with the same period definition. One use case: you've got a period definition with no alertafter or alertevery parameters for a particular time period, and another one for that same time period with a different set of alerts that does have those parameters. 👯

Period definitions, in either form, must be unique within each service definition. So if you need two periods both for "wd {Sun-Sat}", then one or both of them has to specify a label, like "period t1: wd {Sun-Sat}" and "period t2: wd {Sun-Sat}".

  • alert alert [arg...] — a period can hold multiple alerts, and they get triggered when the service fails. An alert is specified with the alert keyword, followed by an optional exit parameter, then arguments interpreted the same way as the monitor definition — but without the ";;" exception. The exit parameter looks like exit=x or exit=x-y, and it means the alert is only called if the exit status of the monitor script falls within that range. So if your alert line is
alert exit=10-20 mail.alert mis

then mail-alert only gets invoked with mis as its arguments if the monitor program's exit value is between 10 and 20. This is how you trigger different alerts at different severity levels — like when free disk space slides from 8% to 3% 😭

See the ALERT PROGRAMS section above for the list of parameters mon will pass automatically to alert programs.

  • alertafter num
  • alertafter num timeval
  • alertafter timeval

The alertafter keyword (inside a period section) has three forms: just the num argument, the num timeval arguments, or just the timeval argument. Let's break it down 💅

Form one: an alert will only be invoked after num consecutive failures.

Form two: the arguments are a positive integer followed by an interval, exactly as described by the interval variable above. With these, the alerts for that period are only called after that many failures happen within that interval. For example, if alertafter gets the arguments "3 30m", then the alert is called if 3 failures happen within 30 minutes. ⏱️

Form three: the argument is an interval, again as described by the interval variable above. Alerts for that period are only called if the service has been in a failure state for longer than the length of time described by the interval — regardless of how many failures were noticed within that interval. 🔥

  • alertevery timeval [observe_detail | strict] — the alertevery keyword (inside a period definition) takes the same type of argument as the interval variable, and it limits how many times an alert is sent while the service keeps failing. So if the interval is "1h", then the alerts in the period section only trigger once every hour. Omit alertevery in a period entry and an alert goes out every time a failure is detected. By default, if the summary output of two successive failures changes, the alertevery interval gets overridden and an alert is sent 🚨 If the string "observe_detail" is the last argument, then both the summary and detail output lines get considered when comparing the output of successive failures. If the string "strict" is the last argument, then the output of the monitor, or the state change of the service, has no effect on when alerts are sent — meaning "alertevery 24h strict" sends only one alert every 24 hours, no matter what. Please refer to the ALERT DECISION LOGIC section for a detailed explanation of how alerts are suppressed. 📖
  • no_comp_alerts
  • upalerts — bestie, if you turn this on, upalerts get called every single time a service flips from failure to success, not just when there was a matching "down" alert before. lowkey more chatty, highkey useful.
  • numalerts num — caps it: no more than num alerts during one failure. The alert counter is tracked per-period, and it resets every time a success comes through. ✨
  • startupalert alert [arg...] — a startupalert only fires when the mon server itself starts up, or when someone issues a reset command to the server — and which one depends on how the startupalerts_on_reset global is set. Unlike other alerts, startupalerts are not called after a monitor exits. They're called in their own right, so heads up: the exit= argument does not apply here. periodt.
  • upalert alert [arg...] — the compliment of an alert 💖 (yes, compliment, that's the original's word). An upalert fires when a service makes the state transition from failure to success, if a corresponding "down" alert was sent earlier. The upalert script gets the same parameters as the alert script, plus the -u parameter — which just exists so the script knows it's being called as an upalert. You can specify multiple upalerts for each period definition. And if you set the per-period no_comp_alerts option, the upalert gets sent regardless of whether a "down" alert ever went out. 🫶
  • upalertafter timeval — this one follows the syntax of the interval parameter ("30s", "1m", etc.) and controls when an upalert triggers. If a service comes back up after being down for a time greater than or equal to this value, the upalert gets called. Use it to stop upalerts from firing over "blips" (brief outages). no more 3am pings for a two-second hiccup, thank you 💅

🔐 AUTHENTICATION CONFIGURATION FILE

The file named by the authfile variable in the config file (or handed over via the -A parameter) gets loaded at startup. This file is where you decide which client commands which users are allowed to run. It's a text file made of comments, command definitions, and trap authentication parameters. A comment line starts with optional whitespace followed by a pound sign. Blank lines? Ignored. 💀

The file splits into a command section and a trap section. You declare sections with a single line containing one of these:

command section

or

trap section

Everything after one of those statements belongs to that section — until either the end of the file or another section starts. Simple as. ✨

A command definition is: a command, then a colon, then a comma-separated list of users allowed to execute it. Default behaviour: nobody can execute anything unless they're explicitly allowed in this file. For clarity, you can deny a user by prefixing their name with !. If the word AUTH_ANY is used as the username, then any authenticated user can run that command. If the word all is used as the username, then that command can be run by anybody — authenticated or not. 👑

The trap section configures which users may send traps from which hosts. Syntax is: a source host (name or ip address), whitespace, a username, whitespace, and a plaintext password for that user. If the source host is *, allow traps from any host. If the username is *, accept traps without caring about username or password. And if no hosts or users are specified, then no traps will be accepted.

An example configuration file:

command section
list:           all
reset:          root,admin
loadstate:              root
savestate:              root

trap section
127.0.0.1       root    r@@tp4sswrd

So in plain English: every client can run the list command, root can run reset, loadstate, and savestate, and admin can execute reset. iconic.

🌐 CLIENT-SERVER INTERFACE

The server listens on TCP port 2583, and you can override that with the -p port option. Commands are one single line each, terminated by a newline. The server handles any number of simultaneous client connections — she's scalable, she's thriving. 💅

🎀 CLIENT INTERFACE COMMANDS

See the manual page for moncmd.

🪤 MON TRAPPING

Mon can receive special "mon traps" from any local or remote machine. Right now, the only available way to send mon traps is through the Mon::Client perl interface — but the UDP packet format is defined well enough that you could write traps in other languages too. iykyk 🧚

Traps get handled similarly to monitors: a trap sends an operational status, a summary line, and description text, and mon generates an alert or upalert as needed.

Any watch/service group set up in the mon config file can catch traps, but it's suggested you configure watch/service groups specifically for the traps you expect to receive. When you define a special watch/service group for traps, don't include a "monitor" directive (no monitor needs to be invoked). And since no monitor is being invoked, your watch definition doesn't need a hostgroup containing real host names — just make up a useful name and mon will automatically create the watch group for you. 🌈

Here's a simple config file example:

watch trap-service
     service host1-disks
          description TRAP: for host1 disk status
          period wd {Sun-Sat}
               alert mail.alert someone@your.org
               upalert mail.alert -u someone@your.org

Since mon listens on a UDP port for any trap, there's a default facility for handling traps aimed at unknown groups or services. To turn it on, include a "default" watch group with a "default" service entry containing the specifics of alarms. If a default/default watch group and service aren't configured, unknown traps get logged via syslog and no alarm is sent.

NOTE: The default/default facility is a single entity as far as accounting and alarming go. Alarm programs that aren't aware of this may send confusing information when a failure trap comes from one machine, followed by a success (ok) trap from a different machine. See the alarm environment variable MON_TRAP_INTENDED above for a possible way around this. It is intended that default/default be used as a facility to catch unknown traps, and should not be relied upon to catch all traps in a production environment. If you're lazy and only want to use default/default for catching all traps, it would be best to disable upalerts, and use the MON_TRAP_INTENDED environment variable in alert scripts to make the alerts more meaningful to you.

Here's an example default facility:

watch default
     service default
          description Default trap service
          period wd {Sun-Sat}
               alert mail.alert someone@your.org
               upalert mail.alert -u someone@your.org

🍓 EXAMPLES

The mon distribution comes with an example configuration called example.cf. Go read that file for more info, bestie. ✨

👀 SEE ALSO

moncmd(1), Time::Period(3pm), Mon::Client(3pm)

📜 HISTORY

mon was written because I couldn't find anything out there that did just what I needed, and nothing was worth modifying to add the features I wanted. It doesn't have a cool name, and that bothers me because I couldn't think of one. 😭 (relatable, honestly)

🐛 BUGS

Report bugs to the email address below.

👑 AUTHOR

Jim Trocki <trockij@arctic.org>

And that's a wrap on the whole page, bestie!! 🎀 This one's from the Linux manpage set, mon(8), dated $Date: 2007/06/25 13:10:07 $ — so yeah, it's giving vintage, it's giving classic, it's giving the blueprint. Go forth and monitor everything. 💖✨🦋

📦 where this page comes from