PMDAWEBLOG(1)

✨ NAME ✨

hey bestie, welcome to the pmdaweblog page — it lives in the General Commands Manual and it's literally so good 🫶

• pmdaweblog — performance metrics domain agent (PMDA) for Web server logs 📜🌐

---

🦋 SYNOPSIS 🦋

$PCP_PMDAS_DIR/weblog/pmdaweblog [-Cp] [-D debug] [-d domain] [-h
helpfile] [-i port] [-l logfile] [-n idlesec] [-S num] [-t delay] [-u
socket] [-U username] configfile

---

💖 DESCRIPTION 💖

okay so pmdaweblog is a Performance Metrics Domain Agent (PMDA(3)) that goes through Web server logs and pulls out metrics that characterize what your Web server is actually doing. then those performance metrics get served up through the whole Performance Co-Pilot (PCP) infrastructure. it's giving observability, no cap 💅

the configfile tells it which Web servers are being monitored, which access logs and error logs belong to them, and a regular-expression based scheme for extracting detailed information about every single Web access. this file is maintained as part of the PMDA installation and/or de-installation by the scripts Install and Remove in the directory $PCP_PMDAS_DIR/weblog. want the details? the installation section below has you covered, bestie 👇

once it's started, pmdaweblog monitors a set of log files, and whenever someone asks it for information it will process any new stuff that's been appended to the log files — basically a tail(1) situation. there's also a periodic "catch up" that processes new information from all the log files, plus a scheme to detect when a log file has been rotated. icon behavior, honestly ✨

and like every other PMDA, pmdaweblog is launched by pmcd(1) using the command line options specified in $PCP_PMCDCONF_PATH — the Install script will prompt you for appropriate values for those command line options, and then update $PCP_PMCDCONF_PATH. teamwork 🫶

here's the quick rundown of the pmdaweblog command line options:

---

🎀 OPTIONS 🎀

• -C — check the configuration and exit. vibe check, then dip 🚪
• -d domain — specify the domain number. and bestie, it is absolutely crucial that the performance metrics domain number you put here is unique and consistent. like: domain should be different for every PMDA on the one host, and the same domain number should be used for the pmdaweblog PMDA on all hosts. no mixing, no matching, periodt.

for most installations, the default domain as encapsulated in the file $PCP_PMDAS_DIR/weblog/domain.h will suffice. for alternate values, check $PCP_PMCDCONF_PATH for the domain values already in use on this host, and the file $PCP_VAR_DIR/pmns/stdpmid contains a repository of "well known" domain assignments that probably should be avoided 👀

• -h helpfile — get the help text from the supplied helpfile rather than from the default location.
• -i port — communicate with pmcd(1) on the specified Internet port (which may be a number or a name).
• -l logfile — location of the log file. by default, a log file named weblog.log is written in the current directory of pmcd(1) when pmdaweblog is started, i.e. $PCP_LOG_DIR/pmcd. if the log file cannot be created or is not writable, output is written to the standard error instead. plan B is still a plan 💁‍♀️
• -n idlesec — if a Web server log file has not been modified for idlesec seconds, then the file will be closed and re-opened. this is the only way pmdaweblog can detect any asynchronous rotation of the logs by Web server administrative scripts. the default period is 20 seconds. this value may be changed dynamically using pmstore(1) to modify the value of the performance metric web.config.check. live tweaking, we love to see it 🔥
• -p — communicate with pmcd(1) via a pipe.
• -S num — specify the maximum number of Web servers per sproc. it may be desirable (from a latency and load balancing perspective) or necessary (due to file descriptor limits) to delegate responsibility for scanning the Web server log files to several sprocs. pmdaweblog will ensure that each sproc handles the log files for at most num Web servers. the default value is 80 Web servers per sproc. dividing the labor, main character energy 👑
• -t delay — to avoid the need to scan a lot of information from the Web server logs in response to a single request for performance metrics, all log files will be checked at least once every delay seconds. the default is 15 seconds. this value may by changed dynamically using pmstore(1) to modify the value of the performance metric web.config.catchup.
• -u socket — communicate with pmcd(1) via the given Unix domain socket.
• -U — user account under which to run the agent. the default is the unprivileged "pcp" account in current versions of PCP, but in older versions the superuser account ("root") was used by default. glow-up confirmed 🌸

---

🌟 INSTALLATION 🌟

the PCP framework allows metrics to be collected on one host and monitored from another. these hosts are referred to as collector and monitor hosts, respectively. and yes, a host may be both a collector and a monitor. multitasking queen 💅

collector hosts require the installation of the agent, while monitoring hosts require no agent installation at all. zero, nada, nothing 🎉

for collector hosts, do the following as root:

# cd $PCP_PMDAS_DIR/weblog
# ./Install

the installation procedure prompts for a default or non-default installation. a default installation will search for known server configurations and automatically configure the PMDA for any server log files that are found. a non-default installation will step through each server, prompting the user for other server configurations and arguments to pmdaweblog. either way, the end result of a collector installation is to build a configuration file that is passed to pmdaweblog via the configfile argument. one file to rule them all 📝

if you want to undo the installation, do the following as root:

# cd $PCP_PMDAS_DIR/weblog
# ./Remove

heads up bestie: pmdaweblog is launched by pmcd(1) and should never be executed directly. the Install and Remove scripts notify pmcd(1) when the agent is installed or removed. let the framework handle it, fr fr 🙈

✨ CONFIGURATION ✨

okay bestie, so the config file for the weblog PMDA is literally just an ASCII file you can edit super easily. 💖 Empty lines and lines starting with #? Ignored, periodt. Everything else has to be either a regular expression or a server spec. No cap.

Regular expressions, which get used on BOTH the access and error log files, have to look like this:

regex regexName regexp

or

regex_posix regexName ordering regexp_posix

The regexName is a word that uniquely identifies the regex. It's the reference you use in the server spec. The regexp for access logs is in the format described for regcmp(3). The regexp_posix for access logs is in the format described for regcomp(3). The ordering argument is explained below. The Posix form should be available on all platforms. 🌈

The regex needs up to four arguments extracted from each line of a Web server access log, depending on the server type. In the most common case there are two arguments: method and size.

For the non-Posix version, argument $0 should contain the method: GET, HEAD , POST or PUT. (Note: PUT is treated as a synonym for POST, and anything else is categorized as OTHER.)

The second argument, $1, should contain the size of the request. A size of - or (space) is treated as unknown. 🙈

Argument $3 should contain the status code returned to the client browser and argument $4 should contain the status code returned to the server from a remote host. These latter two arguments are used for caching servers and must be specified as a pair (or $3 will be ignored). For further info on status codes, check out the web site http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

Some legal non-Posix regex expression specs for monitoring an access log are:

# pattern for CERN, NCSA, Netscape etc Access Logs
regex CERN ] "([A-Za-z][-A-Za-z]+)$0 .*" [-0-9]+ ([-0-9]+)$1

# pattern for FTP Server access logs (normally in SYSLOG)
regex SYSLOG_FTP ftpd[.*]: ([gp][-A-Za-z]+)$0( )$1

There is 1 special types of access logs with the RegexName SQUID. This formats extract 4 parameters but since the Squid log file uses text-based status codes, it is handled as a special case.

In the examples below, NS_PROXY parses the Netscape/W3C Common Extended Log Format and SQUID parses the default Squid Object Cache format log file.

# pattern for Netscape Proxy Server Extended Logs
regex NS_PROXY ] "([A-Za-z][-A-Za-z]+)$0 .*" ([-0-9]+)$2 \
     ([-0-9]+)$1 ([-0-9]+)$3

# pattern for Squid Cache logs
regex SQUID [0-9]+.[0-9]+[ ]+[0-9]+ [a-zA-Z0-9.]+ \
     ([_A-Z]+)$3([0-9]+)$2 ([0-9]+)$1 ([A-Z]+)$0

The regexp for the error logs does not require any arguments, only a match. Some legal expressions are:

# pattern for CERN, NCSA, Netscape etc Error Logs
regex CERN_err .

# pattern for FTP Server error logs (normally in SYSLOG)
regex SYSLOG_FTP_err FTP LOGIN FAILED

If POSIX compliant regular expressions are used, additional info is required since the order of parameters cannot be specified in the regex. For backwards compatibility, the common case of two parameters the order may be specified as method,size or size,method. In the general case, the ordering is specified by one of the following methods:

• n1,n2,n3,n4 — where nX is a digit between 1 and 4. Each comma-separated field represents (in order) the argument number for method,size,client_status,server_status
• - — Used for cases like the error logs where the content is ignored.

As for the non-Posix format, the SQUID RegexName is treated as a special case to match the non-numerical status codes.

Some legal Posix regex expression specs for monitoring an access log are:

# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN method,size ][ \]+"([A-Za-z][-A-Za-z]+) \
     [^"]*" [-0-9]+ ([-0-9]+)

# pattern for CERN, NCSA, Netscape, Apache etc Access Logs
regex_posix CERN 1,2 ][ \]+"([A-Za-z][-A-Za-z]+) \
     [^"]*" [-0-9]+ ([-0-9]+)

# pattern for FTP Server access logs (normally in SYSLOG)
regex_posix SYSLOG_FTP method,size ftpd[.*]: \
     ([gp][-A-Za-z]+)( )

# pattern for Netscape Proxy Server Extended Logs
regex_posix NS_PROXY 1,3,2,4 ][ ]+"([A-Za-z][-A-Za-z]+) \
     [^"]*" ([-0-9]+) ([-0-9]+) ([-0-9]+)

# pattern for Squid Cache logs
regex_posix SQUID 4,3,2,1 [0-9]+.[0-9]+[ ]+[0-9]+ \
     [a-zA-Z0-9.]+ ([_A-Z]+)([0-9]+) ([0-9]+) ([A-Z]+)

# pattern for CERN, NCSA, Netscape etc Error Logs
regex_posix CERN_err - .

# pattern for FTP Server error logs (normally in SYSLOG)
regex_posix SYSLOG_FTP_err - FTP LOGIN FAILED

A Web server can be specified using this syntax:

server serverName on|off accessRegex accessFile errorRegex errorFile

The serverName must be unique for each server, and is the name given to the instance for the associated performance metrics. See PMAPI(3) for a discussion of PCP instance domains. The on or off flag indicates whether the server is to be monitored when the PMDA is installed. This can be altered dynamically using pmstore(1) for the metric web.perserver.watched, which has one instance for each Web server named in configfile.

Two files are monitored for each Web server: the access and the error log. Each file requires the name of a previously declared regular expression, and a file name. The log files specified for each server do not have to exist when the weblog PMDA is installed. The PMDA will continue to check for non-existent log files and open them when possible. Some legal server specs are:

# Netscape Server on Port 80 at IP address 127.55.555.555
server 127.55.555.555:80 on CERN /logs/access CERN_err /logs/errors

# FTP Server.
server ftpd on SYSLOG_FTP /var/log/messages SYSLOG_FTP_err /var/log/messages

---

⚠️ CAVEATS ⚠️

Specifying regular expressions with an incorrect number of arguments — anything other than 2 for access logs, and none for error logs — may cause the PMDA to behave incorrectly and even crash. This is due to limitations in the interface of regex(3). 💀 So don't do that, bestie.

---

📁 FILES 📁

• $PCP_PMDAS_DIR/weblog — installation directory for the weblog PMDA
• $PCP_PMDAS_DIR/weblog/Install — installation script for the weblog PMDA
• $PCP_PMDAS_DIR/weblog/Remove — de-installation script for the weblog PMDA
• $PCP_LOG_DIR/pmcd/weblog.log — default log file for error reporting
• $PCP_PMCDCONF_PATH — pmcd configuration file that specifies the command line options to be used when pmdaweblog is launched
• $PCP_LOG_DIR/NOTICES — log of PMDA installations and removals
• $PCP_VAR_DIR/config/web/weblog.conf — likely location of the weblog PMDA configuration file
• $PCP_DOC_DIR/pcpweb/index.html — the online HTML documentation for PCPWEB

---

🌍 PCP ENVIRONMENT 🌍

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

---

🐛 DEBUGGING OPTIONS 🐛

The -D or --debug option enables the output of additional diagnostics on stderr to help triage problems, although the information is sometimes cryptic and primarily intended to provide guidance for developers rather than for end-users. debug is a comma separated list of debugging options; use pmdbg(1) with the -l option to obtain a list of the available debugging options and their meaning.

Debugging options specific to pmdaweblog are as follows:

• appl0 — initialization, configfile parsing and log file processing
• appl1 — timer operations, log file probing
• appl2 — IPC with worker processes, log lines processed

---

👀 SEE ALSO 👀

pmcd(1), pmchart(1), pmdawebping(1), pminfo(1), pmstore(1), pmview(1), tail(1), weblogvis(1), webvis(1), PMAPI(3), PMDA(3) and regcmp(3).

---

💖 That's it! This page is from Performance Co-Pilot, PCP, and the manpage version/date is in the footer: PMDAWEBLOG(1). Stay slay, bestie. ✨