tenshi(8)

📦 version 0.13, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 tenshi — the log-watching bestie 💖

hiii okay so this is tenshi(8), straight out of the System Manager's Manual, section 8 (the sysadmin section, main character energy fr fr 👑). sit down, grab a snack 🍓, we're doing this together.

✨ NAME ✨

tenshi — Log Monitoring and Reporting tool. that's her. that's the whole vibe. 💅

🎀 SYNOPSIS 🎀

tenshi [ -c <conf file> ] [ -C ] [ -d <debug level> ] [ -f ] [ -h ] [ -p ] [ -P <pid file> ]

🦋 DESCRIPTION 🦋

so tenshi is a log monitoring program, and her entire job is watching one or more log files for lines that match regular expressions you define, then reporting on the matches. like a tiny guardian angel for your logs 🧚

those regular expressions get assigned to queues, and every queue has an alert interval plus a list of mail recipients. so organized, so cute.

queues can be set to do either of two things:

  • send a notification the literal second a log line gets assigned to it, or
  • send periodic reports 🌈

also, and this is the part everyone's obsessed with: boring fields inside log lines (think PID numbers) can be masked with the standard regular expression grouping operators ( ). that means your reports come out cleaner and way more readable. all reports are separated by hostname, and all messages get condensed when possible. chef's kiss 😭

the program reads a configuration file (tenshi.conf) and then forks a daemon for monitoring the specified log files.

⭐ OPTIONS ⭐

  • -c <conf file> — read configuration from file. the default file is /etc/tenshi/tenshi.conf.
  • -C — perform a syntax check of the configuration file. the program exits after parsing the configuration with either a return code of 0 or an error.
  • -d <debug level> — enable debug messages. default level is 1 if none is specified, level 2 enables SMTP connection debug messages. in this mode the main process remains in the foreground.
  • -f — enable foreground mode. in this mode the main process operates normally but remains in the foreground; this is needed for some process supervisors.
  • -p — enable profiling mode. in this mode the main process remains in the foreground and expects log lines to be fed to standard in. when it receives an EOF it will stop processing. no alerts will be sent in this mode — it is used solely for measuring tenshi's line processing speed. for example:
time $(cat /var/log/messages|tenshi -p > /dev/null)
  • -P <pid file> — define the file containing the PID of the process. this overrides any pidfile option present in the configuration file.

🍓 CONFIGURATION FILE 🍓

all directives are shown with the standard default value where applicable. if you omit one, the default value gets used. easy 💖

🌸 EXTERNAL CONFIGURATION FILES 🌸

all configuration directives can optionally be split into different configuration files, and then read with the two following statements:

  • include <configuration file> — parse the specified configuration file.
  • includedir <directory> — parse all files inside <directory>. the files will be parsed in alphabetical order — keep in mind that regexps order matters, so includedir should be used carefully, see REGEXP DEFINITIONS for details. 👀

💜 STATIC OPTIONS 💜

these options will be read the first time tenshi reads its config file. they cannot be changed by re-reading the config file. if you change one of these options and HUP tenshi, it will die. you have been warned. 💀

  • set uid tenshi — specify the effective user ID of the process when in daemon mode. the user must be able to read the selected log files, the configuration file, and write the specified pid file. never use privileged users here since it's not usually necessary (log files perms can be set accordingly with most syslog implementations).
  • set gid tenshi — specify the effective group ID of the process when in daemon mode.
  • set pidfile /var/run/tenshi.pid — the file containing the PID of the process, useful for start/stop scripts.
  • set logfile <log file path> — a log file to monitor. this may be specified multiple times to watch more than one log file. depending on your tail implementation you might need to use the tail_multiple setting for multiple files to work. this mode can be used along with fifo and listen settings.
  • set tail /usr/bin/tail -q --follow=name --retry -n 0 — specify the path and arguments for the tail binary used for reading the log files. the invocation must be tuned against your current tail implementation. default values are configured for standard GNU coreutils tail. the --follow=name and --retry flags should deal properly with log rotation — if they're missing on your implementation we suggest that you use something like cp /dev/null logfile as a safe way for clearing the log file upon rotation.
  • set tail_multiple <on|off> — some tail implementations do not handle more than one log file. when this option is enabled multiple tail commands will be forked, instead of a single command with multiple arguments. this option is disabled by default.
  • set fifo <fifo path> — a FIFO file to monitor. this option allows you to use a syslog-ng pipe() destination (or any other syslog implementation that allows FIFO usage). this may be specified multiple times to watch more than one fifo file. this option is meant to be used only when the installed tail binary doesn't behave properly with FIFOs — please test your tail implementation before using this. this mode can be used along with logfile and listen settings.
  • set listen 0.0.0.0:514 — enables syslog server mode. with this option tenshi will bind to the specified address:port pair and read messages acting like a syslog server. we always recommend to filter the port accordingly and possibly use something like stunnel for encrypting the traffic. this mode can be used along with logfile and fifo settings.

🎀 DYNAMIC OPTIONS 🎀

these options are set each time the config file is read. tenshi reads its config file once on start-up and whenever it receives a HUP.

  • set sleep 5 — the loop sleep time for the notification process. the value must be <= 60 seconds.
  • set limit <number of lines> — the maximum number of messages per hostname allowed in a report. any lines after the maximum will be omitted and a warning included. if this option is omitted then no limit is applied.
  • set pager_limit <number of lines> — the maximum number of messages per hostname allowed in pager friendly reports. any lines after the maximum will be omitted. if this option is omitted then no limit is applied.
  • set logprefix <regexp> — all valid syslog messages are parsed by default, while non-syslog ones are discarded unless the special noprefix queue is set. this option allows one to define an additional valid prefix for watching other type of logs. if the regexp is matched then the prefix is removed from the log and the first grouped string is used for the hostname field. this may be specified multiple times to watch many different non-syslog logs.
  • set mask ______ — the mask for strings enclosed by the grouping operators ( ). see the REGEXP DEFINITIONS section. set mask on its own will set the mask to an empty string.
  • set mailserver localhost — the mail server to be contacted for sending out reports.
  • set mailtimeout 10 — the timeout in seconds for mail server reply.
  • set subject tenshi report — the subject of report emails; the queue name is always automatically appended.
  • set hidepid <on|off> — this option turns on automatic stripping of foo[1234]: style PID strings from the start of log lines, i.e. foo[1234]: becomes foo:. this allows you to write regexps without worrying about masking the PID. bear in mind that any time you change this option you will need to re-write your regex rules or they will not work. this option is disabled by default.
  • set filter <queue> <filter path> <arguments> — when this option is enabled all reports matching the specified queue will be passed as STDIN to the specified filter, the resulting output is sent via smtp instead of the original report. the full path of the filter application must be specified.
  • set csv <cron_spec> <filter path> <arguments> — this feature allows periodic reporting, using a five-field cron-style specification like the set queue option, to the specified filter. the output is pre-formatted as CSV (Comma Separated Values) with hostname,log,hits format. this feature was coded for using AfterGlow (http://afterglow.sf.net) as a filter and graphing tenshi output. check the FAQ for sample usage.
  • set sort_order <descending|ascending> — the sorting order for reports. it can be either descending or ascending, the number of messages is used as a key for sorting the log messages. the default order is ascending.
  • set resolve <on|off> — this option turns on resolution of the fully qualified domain name for the hostname passed along with log messages and, if found, reports it along with the original one. this only affects reports and not pager messages. the name resolution is cached in order to avoid re-resolving addresses that have been seen already — you have to restart or HUP tenshi in order to flush the cache. this option is disabled by default.
  • set threshold <queue> <count> <regex> — this option can be used to discard lines from a report that have a count below the given threshold. if a line matches the regex in the given queue but has fewer hits than count, it is discarded and omitted from the report. note that this matches on the content of the lines that will actually appear in the report, in contrast to queue escalation which uses a count based on the regex that is matched.

👑 QUEUES OPTIONS 👑

all messages are assigned to queues. every queue is processed periodically according to its notification interval. there are four default builtin queues:

  • trash — to which unwanted messages can be assigned (think /dev/null)
  • repeat — which is used for smart repeat messages handling
  • group and group_host — see REGEXP DEFINITIONS for details.

there's also a special noprefix queue — read further for details about it.

all queues are automatically flushed before shutdown when a SIGTERM is received. please see section SIGNALS for additional information.

the syntax is the following:

set queue <queue_name> <mail_from> [pager:]<mail_to> <cron_spec> [<subject>]
  • <queue_name> — the queue name. can be any alphanumeric character string except for the builtin queues name.

okay and that's where this part cuts off bestie, there's literally more coming, don't move 💖

  • <mail_from> — the sender address that reports for this queue get mailed from. 💌
  • <mail_to> — who gets the reports for this queue. You can list multiple addresses, just separate them with commas. And bestie, listen: if you slap a pager: prefix on an address, that report comes out pager-friendly. ✨
  • [<cron_spec>] — a five-field cron-style spec telling tenshi when to email the reports. Ranges and skip values work like the de facto crontab syntax, with a couple of exceptions. Go read the crontab man page for the full syntax explanation, iykyk. The day names it supports are: Mon, Tue, Wed, Thu, Fri, Sat, Sun. Monday is 1, Sunday is 0 or 7. The month names it supports are: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec. Day and month names are not case sensitive. Oh and also — you can literally just write now if you want immediate notifications. 🚨
  • <subject> — the subject line for email reports about this queue. Don't specify one? Then the default subject gets used, no biggie.

The special noprefix queue can be used and defined exactly like any other queue, with one difference: it collects all the messages that don't match any configured prefix. It's giving catch-all energy. 👑

Examples:

set queue report tenshi@localhost sysadmin@localhost [0 9-17 * * *]
set queue report tenshi@localhost sysadmin@localhost [30 18 * * *]
set queue report tenshi@localhost sysadmin@localhost [*/10 * * * *]
set queue critical tenshi@localhost sysadmin@localhost,noc@localhost
[now] CRITICAL WARNING -
set queue pager tenshi@localhost
pager:sysadmin_pager@localhost,pager:noc_pager@localhost [now] ALERT

🌈 REGEXP DEFINITIONS 🌈

Every valid syslog message gets matched against standard perl regexps, and all regexps are defined with this syntax:

<queue_name>[,<queue_name>[:<escalation_number>]..] <regexp>

The regexps are evaluated in order, so once a message matches, it does not get checked against the regexps after it. Keep that in mind when you're assembling the config file, fr fr. It's advisable to catch all messages by placing an all-matching regexp at the very end of the config file. It's also good for performance to put trash rules that aren't logically connected to other matching rules at the beginning of the section. You can define multiple queues with a comma separated list — but heads up, builtin queues cannot be used when you're using this syntax. 🙅

If you give a queue an escalation number, the matched message only lands in that queue once <escalation_number> messages have matched the regexp. The queue receives the message that matched the regexp at the time of escalation, with a count equal to the escalation number. The count of messages matching the regexp resets when the left most queue mentioned in the queue list is mailed. The left most queue can't have an escalation number unless it's the only queue listed. When the number of messages matching the regexp hits the greatest escalation number mentioned, escalation starts again into the escalation queues, modulus the greatest escalation number. For example, using the queues a,b:10,c:50, when 10 messages match the regexp a message goes into b; when 50 match, one goes into c. At 60, another goes into b; at 100, another into c; 110 to b, 150 to c, and so on. 🔁 Escalation numbers must be positive integers greater than zero, and must be listed in increasing order from left to right. All queues without escalation numbers must be listed more left than the queues with escalation numbers.

The standard grouping operators ( ) can be used for string masking, and literal ( and ) can be protected with the standard quotation operator \. There's a lot of documentation about regular expressions out there — a good start could be the perl perlre and perlretut manual pages. 📚 You can also use the (?: ) operators to use groups without masking. That lets you match, for example, output from several programs in a similar format. There's an example of this below (the sudo/su line).

The builtin queue repeat can be used for special handling of "last message repeated x times" style log lines. When the assigned regexps are matched, the line count for the last line received from the same host is incremented by the first grouped string. Keep in mind that it's possible for syslog lines to be received from remote hosts out of order. If that happens you should not use this feature, because tenshi will mis-report line counts. 😬

The builtin queue group can be used to group sets of regex together to speed up line matching. If a line fails to match a regex assigned to the group queue, then tenshi skips all the regex up until the next group_end statement. Nested groups are allowed. There's an example of this included below. 🫶

The builtin group_host queue can be used for selective hostname matching. Like the group queue it's also terminated with the group_end statement. All regex definitions within that group will only apply if the hostname associated to the log entries matches the regex passed to the group_host definition.

The regexps below assume hidepid is turned on. If you have it turned off then you'll need to add in \[(.+)\] to the regex following the program name to get them to work.

For example: mail ^sendmail: (.+): to=(.+),(.+)delay=(.+) becomes: mail ^sendmail\[(.+)\]: (.+): to=(.+),(.+)delay=(.+)

Examples:

trash ^xinetd

repeat ^(?:last message repeated|above message repeats) (\d+) time

group ^sendmail:
mail ^sendmail: (.+): to=(.+),(.+)delay=(.+)
mail ^sendmail: (.+): to=(.+),(.+)relay=(.+),(.+)stat=Sent
group_end

group_host mailserver1
mail1 ^sendmail
mail1 ^sendmail:.+
critical,mail1 ^sendmail:.+SYSERR.+
group_end

mail ^ipop3d: Login user=(.+)

critical,report ^sshd: Illegal user

general,urgent:200,critical:1000 ^sshd: Illegal user

root ^sshd\(pam_unix\): session opened for user root by root\(uid=0\)

report ^sshd: Accepted rsa for (.+) from (.+) port (.+)

trash ^sshd

critical ^(?:sudo|su):

critical,pager ^Oops

misc .*

🚦 SIGNALS 🚦

tenshi can handle different signals sent to the process. Here's the list of supported ones, bestie:

  • TERM — flush all queues and then exit. 🚪
  • INT — flush all queues and then exit.
  • USR1 — flush any queues which have reached their notification interval. ⏰
  • USR2 — force all queues to be flushed, even if they haven't reached their notification interval. 💥
  • HUP — force all queues to be flushed, even if they haven't reached their notification interval, re-read the config file, and continue as normal. 🔄
WARNING: If you change a STATIC OPTION in the config file and send tenshi a HUP it will die. You will need to restart tenshi for changes to STATIC OPTIONs to take effect. 💀

✨ EXAMPLES ✨

See the included tenshi.conf. That's the whole vibe.

🧚 REQUIREMENTS 🧚

tenshi needs a working 'tail' implementation when not using FIFO mode.

It also requires the Net::SMTP module for mailing reports, which should be included in your perl installation, and IO::BufferedSelect. If you're missing any of them you can grab them at CPAN (<http://www.cpan.org>) — wait, no links, so: at CPAN, http://www.cpan.org — or using the CPAN shell (`perl -e shell -MCPAN`).

🐛 BUGS 🐛

Double quotation characters present in your logs might break csv output (depending on how you pipe/process it in the filter), since there's no escape code (yet). 🙈

Please report any bugs you find at <tenshi@inversepath.com>.

💜 LICENSE 💜

tenshi is distributed under the terms of the following ISC-style license:

Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

🌸 DISTRIBUTION 🌸

The tenshi project page is http://www.inversepath.com/tenshi.html

📝 NOTES 📝

tenshi was once known as wasabi, but the name was changed as we were informed that wasabi is a registered trademark relating to another piece of software. Glow-up era. 💅

👀 SEE ALSO 👀

It should be noted that tenshi was initially a perl rewrite of oak (http://www.ktools.org).

Friedl, Jeffrey E. F. Mastering Regular Expressions, 2nd Edition. O'Reilly

👑 AUTHORS 👑

Copyright 2004-2011 Andrea Barisani <andrea@inversepath.com>

And that's the whole page, bestie! 🫶 This is version 0.13, dated 13 Jul 2011, and it's the tenshi(8) manual — go forth and wrangle those logs, you absolute icon. ✨💖

📦 where this page comes from