Messag...okbook(3pm)

💖 Message::Passing::Manual::Cookbook — the perl cookbook, but make it a group chat 💖

okay bestie, listen. this is Message::Passing::Manual::Cookbook from the Perl world — the User Contributed Perl Documentation corner of the man pages, section 3pm, iykyk. and the subtitle fr fr is just "Common recipies" (yes, spelled like that, we don't fix the author's spelling, we respect the author's spelling 💅).

it's a recipe book. it's giving ✨ cooking show but for logs ✨. let's get into it.

---

🌈 Aggregating logs 🌈

🍓 Logging from an application

so you've got an app and you want its logs somewhere else. easy. you can use Log::Dispatch, or literally any log system that will output into Log::Dispatch. that's the vibe. it's giving plug-and-play.

use Log::Dispatch;
use Log::Dispatch::Message::Passing;
use Message::Passing::Filter::Encoder::JSON;
use Message::Passing::Output::ZeroMQ;

my $log = Log::Dispatch->new;

$log->add(Log::Dispatch::Message::Passing->new(
    name      => 'myapp_aggregate_log',
    min_level => 'debug',
    output    => Message::Passing::Filter::Encoder::JSON->new(
        output_to => Message::Passing::Output::ZeroMQ->new(
            connect => 'tcp://192.168.0.1:5558',
        ),
    ),
));

$log->warn($_) for qw/ foo bar baz /;

look at that. chef's kiss. no cap.

🌸 Aggregating this log

now you gotta actually catch those logs on the other side. it is literally as simple as using the command line interface:

message-pass --input ZeroMQ --input_options '{"socket_bind":"tcp://192.168.0.1:5558"}' \
    --output File --output_options '{"filename":"/tmp/mylog"}'

and boom 💥 — you've now got a multi-host log aggregation system for your application. main character infrastructure energy honestly.

🦋 Doing it manually

you do NOT have to do any of the above if you don't want to. nobody is forcing you, bestie. you can easily reuse the ZeroMQ output yourself:

my $log = Message::Passing::Output::ZeroMQ->new(
    connect => 'tcp://192.168.0.1:5558',
    linger  => 1,  # make sure message is sent (flushed) before thread dies
);
$log->consume("A log message");

do it yourself era. we love a DIY queen 👑.

💜 A note about outputs

ZeroMQ is the recommended output for sending messages from within your application. this is because ZeroMQ uses a different (POSIX) thread to send messages — meaning that it transports messages independently to whatever your perl code is doing.

that's the whole reason it slays, okay? it's off doing its own thing in its own thread while your perl code is busy being perl code.

this is not the case for other message outputs, and therefore they are unlikely to work well, or at all, unless your application is already asynchronous and using an AnyEvent supported event library. so like… read the room. ⚠️

🌟 A note about ZeroMQ

by default Message::Passing::ZeroMQ will use PUB/SUB sockets for logging, with a finite 'high water mark'. 💀

this means that if your application logs significantly more data than you can fit down the network, you will drop logs. not "maybe", not "sometimes" — you will drop them. bye bye logs.

if your application needs to do this, you can either:

• increase this high water mark, or
• disable it (so ZeroMQ will buffer an infinite number of messages at the sending client — potentially using infinite RAM 😭)

the default setting is for the output to buffer up to 10000 messages on the output side, which should be enough to manage short term peaks, but is low enough to be reasonably safe in terms of memory consumption for buffering.

ten thousand. remember the number. it's the number. 🔥

---

🧚 Aggregating syslog 🧚

okay so. assuming that you've got a regular syslogd setup and working, then you probably want to keep that. having some of the log files on individual hosts can be very useful. also, we'd like to avoid the script being a privileged user (which would be needed to get the standard port). and honestly? fair. privilege is a lot of responsibility and we're just trying to aggregate logs.

therefore, we'll run a syslog listener on a high port (5140), and get the regular syslogd to ship messages to it. the listener will then forward from each host to a central aggregate logger (which is setup as above). ✨

🎀 On host collector

message-pass --input Syslog --output ZeroMQ --output_options '{"connect":"tcp://192.168.0.1:5558"}'

that's it. that's the whole thing. she's beauty, she's grace. 💅

⭐ Configuring your syslogd

this should be easy. here's an example of what to add to rsyslogd.conf to get the syslog resent:

*.* =192.168.0.1:5140

copy, paste, ascend. 🫶

💖 Aggregating everything

if you have hosts with both applications and syslog that you want to aggregate, then you can easily do both at once. this also means that your apps ship logs to a local buffer process rather than directly across the network — which is more resilient to short network outages.

lowkey the best setup. the local buffer is the bestie who holds your stuff while you're offline. 🥹

---

👑 AUTHOR, COPYRIGHT & LICENSE

See Message::Passing.

---

and that's the whole page, bestie!! this bad girl came from perl v5.32.1, dated 2021-11-09 — so like, vintage, but make it iconic. 💖 go forth and aggregate your logs. periodt. 🎀✨