PMLOGGER_DAILY(1)

📦
version 7.2.2 · 7 versions
, in arch rolling, centos-stream 10, fedora 43, fedora 44, ubuntu resolute, ubuntu stonking

💖 pmlogger_daily — from the General Commands Manual 💖

okay bestie, sit down, hydrate, because we're about to talk about the most iconic little admin script in the PCP cinematic universe: pmlogger_daily. yes, it's the PMLOGGER_DAILY(1) page. yes, it's long. no, we are NOT skipping anything. let's go 🎀

✨ NAME ✨

pmlogger_daily — administration of Performance Co-Pilot archive files

the way that this little script just... handles your archives?? main character energy fr 💅

🌸 SYNOPSIS 🌸

$PCP_BINADM_DIR/pmlogger_daily [-DEfKMNnoPpQrRVzZ?]  [-c control] [-d
fssize] [-k time] [-l logfile] [-m addresses] [-s size] [-t want] [-x
time] [-X program] [-Y regex]

look at all those flags. she's got range. 🦋

👑 DESCRIPTION 👑

okay so here's the tea, bestie. pmlogger_daily and its little sibling pmlogger_check(1), plus the associated control files (peep pmlogger.control(5)), are how you build a customized regime of administration and management for historical archives of performance data inside the Performance Co-Pilot infrastructure (that's PCPIntro(1), iykyk).

The vibe: pmlogger_daily is meant to be run once per day, preferably in the early morning, as soon after midnight as is actually practical. Its whole job? Aggregate, rotate, and do general housekeeping on one or more sets of PCP archives. she's the responsible one. she does the chores. 🧹✨

Now — because PMDAs evolve and production logging environments change (ugh, growth), pmlogger_daily is integrated with pmlogrewrite(1) so your archives can get optionally and automatically rewritten before merging. here's how you set that up:

  • If you've got global rewriting rules that should apply across all archives mentioned in the control file(s), make the directory $PCP_SYSCONF_DIR/pmlogrewrite and drop your pmlogrewrite(1) rewriting rules in there. global queen. 👑
  • If your rewriting rules are specific to just one family of archives, use the directory name from the control file(s) — that's the fourth field, bestie — and create a file, or a directory, or a symbolic link named pmlogrewrite inside that directory, then put the required rewriting rule(s) in the pmlogrewrite file or in files inside the pmlogrewrite subdirectory.

and then pmlogger_daily picks like this, in order:

  1. rewriting rules from the archive directory, if they exist
  2. else rewriting rules from $PCP_SYSCONF_DIR/pmlogrewrite, if that directory exists
  3. else no rewriting is attempted. (it's giving "we tried, we move on" 💀)

Alternate mechanism alert 🚨 if the file $PCP_LOG_DIR/pmlogger/.NeedRewrite exists when pmlogger_daily starts, that's treated the exact same as passing -R on the command line. and $PCP_LOG_DIR/pmlogger/.NeedRewrite gets removed once all the rewriting is done. tidy. immaculate. periodt.

💅 OPTIONS 💅

okay this is the long part. don't panic. we're doing every single one. grab a snack 🍓
  • -c control, --control=control — so both pmlogger_daily and pmlogger_check(1) are controlled by PCP logger control file(s), and those files specify the pmlogger(1) instances you're managing. the default control file is $PCP_PMLOGGERCONTROL_PATH, but you can point at an alternate one with -c. and plot twist: if the directory $PCP_PMLOGGERCONTROL_PATH.d exists (or control.d from whatever you gave to -c), then the contents of any extra control files in there get appended to the main control file (and the main one must exist, no cap).
  • -D, --noreport — do NOT do the conditional pmlogger_daily_report(1) processing that's described further down. skipping it. not today. 🙈
  • -d fssize, --disk=fssize — this one enforces a maximum total file system space usage per host directory for PCP archive files stored under each $PCP_ARCHIVE_DIR/<hostname> and $PCP_REMOTE_ARCHIVE_DIR/<hostname> location. the fssize has to be an integer with a suffix: K or k for kilobytes, M or m for megabytes, G or g for gigabytes (so like 500m or 10G, you get it). as a special little treat, fssize can be the keyword unlimited to prevent any file system space limit enforcement at all — and heads up, unlimited is the default behaviour.

Alternatively, the file system space limit can be set by exporting the $PCP_SPACELIMIT environment variable, which gets handled analogously to $PCP_CULLAFTER. and if you provide both $PCP_SPACELIMIT and -d and they disagree? the value from the environment variable wins and a warning is issued. sorry to -d. 🫶

So after the normal daily log processing finishes (compression, merging, culling by age, etc.), if the total file system space consumed by any pmlogger(1) instance exceeds fssize, then extra archive files get purged — like, actually removed — oldest first, until file system usage is reduced to no more than fssize. and this purging never deletes archive files with today's date, so it always preserves the most recent archives that active pmlogger(1) processes need. she's ruthless but she's careful. 💀

⚠️ real talk though: enforcement only happens when pmlogger_daily runs (typically once a day), not continuously. so the total file system usage for any pmlogger(1) instance can totally exceed your specified maximum between pmlogger_daily executions — for example as new archives get created, or when archives get uncompressed for merging, or before asynchronous archive compression. so like, it's a vibe check, not a bodyguard. 😭
  • -E, --expunge — this makes pmlogger_daily pass the -E flag to pmlogger_merge(1) so it expunges metrics with metadata inconsistencies and just... continues, instead of failing. this is intended for automated daily archive rotation, where you really want unattended daily archive merging, rewriting and compression to succeed. for further details, see pmlogger_merge(1) and the description for the -x flag in pmlogextract(1). resilience queen ✨
  • -f, --force — this forces pmlogger_daily to attempt compression actions. and the manpage is very clear: using this option in production is not recommended. so like, do it in the lab, bestie, not in prod. 🙈
  • -k time, --discard=time — after some period, old PCP archives are discarded. time is a time specification in the syntax of find-filter(1), so DD[:HH[:MM]]. the optional HH (hours) and MM (minutes) parts are 0 if you don't specify them. by default the time is 14:0:0, i.e. 14 days, but you can change that with this option.

Some special values are recognized for time, namely 0 to keep no archives beyond the ones currently being written by pmlogger(1), and forever or never to prevent any archives being discarded. (iconic options tbh)

time can also be set using the $PCP_CULLAFTER variable, set in either the environment or in a control file. and if $PCP_CULLAFTER and -k specify different values for time, then the environment variable value is used and a warning is issued — meaning if $PCP_CULLAFTER is set in the control file, it overrides -k given on the command line. control file stays winning. 💅

🚨 note that the semantics of time are that it's measured from the time of last modification of each archive, and not from the original archive creation date. this has subtle implications for compression (see below) — because the compression process creates new archive files with new modification times. in that case, the time period (re)starts from the time of compression. so the clock resets. no cap.
  • -K — when this is specified for pmlogger_daily, then only the compression tasks are attempted, so: no pmlogger rotation, no culling, no rewriting, etc. just compression, nothing else.

When -K is used and a period of 0 is in effect (from -x on the command line, or $PCP_COMPRESSAFTER in the environment, or via the control file), this is intended for environments where you want archives compressed before the scheduled daily processing happens. to achieve this: once pmlogger_check(1) has completed regular processing, it calls pmlogger_daily with just the -K option. and provided $PCP_COMPRESSAFTER is set to 0 along with any other required compression options to match the scheduled invocation of pmlogger_daily, then this will compress all volumes except the ones being currently written by pmlogger(1).

If $PCP_COMPRESSAFTER is set to a value greater than zero, then manually running pmlogger_daily with the -x option may be used to compress volumes that are younger than the $PCP_COMPRESSAFTER time. this can be used to reclaim filesystem space by compressing volumes earlier than they'd otherwise have been compressed. 🎀

⚠️ note that since the default value of $PCP_COMPRESSAFTER is 0 days, the -x option has no effect unless the control file has been edited and $PCP_COMPRESSAFTER has been set to a value greater than 0.

The default value of $PCP_COMPRESSAFTER is 0 if zstd(1) is installed, or if xz(1) is installed and the lzma library is available (as reported for the transparent_decompress option by pmconfig(1)). Otherwise the default value of $PCP_COMPRESSAFTER is never.

  • -l file, --logfile=file — okay so in order to make sure mail is not unintentionally sent when these scripts are run from cron(8), or when systemd(1) diagnostics happen, diagnostics are always sent to log files. by default this file is $PCP_LOG_DIR/pmlogger/pmlogger_daily.log, but you can change it with -l. and if this log file already exists when the script starts, it gets renamed with a .prev suffix (which overwrites any log file saved earlier — yes, it does) before diagnostics get generated to the log file. 📝
💀 also: the -l and -t options cannot be used together. pick a lane, bestie.
  • -m addresses, --mail=addresses — using this option makes pmlogger_daily construct a summary of the notices'' file entries generated in the last 24 hours, and e-mail that summary to the set of space-separated *addresses*. this daily summary is stored in the file `$PCP_LOG_DIR/NOTICES.daily`, which will be empty when no new notices'' entries were made in the previous 24 hour period. inbox staying clean, we love to see it 🫶
  • -M — this option disables archive merging (or renaming) and rewriting. heads up: -M implies -r. this is most useful in cases where archives are being incrementally copied to a remote repository, e.g. using rsync(1). merging, renaming and rewriting all risk an increase in synchronization load, especially immediately after pmlogger_daily has run, so -M may be useful in these cases. protecting the sync, one flag at a time 🦋
  • -N, --showme — this enables a ``show me'' mode, where the program's actions are echoed but not executed, in the style of make -n. and using -N together with -V maximizes the diagnostic capabilities for debugging. bestie debug combo fr fr 🔥
  • -n, --noerror — forces the exit status to be 0, other than for catastrophic errors. useful when called from a systemd(1) service or timer unit, so the associated unit doesn't get marked as failed in a way that would prevent subsequent execution from systemd(1). no failed unit, no drama 💅
  • -o — by default all possible archives will be merged. this option reinstates the old behaviour, in which only yesterday's archives will be considered as merge candidates. and in the special case where only a single input archive needs to be merged, pmlogmv(1) is used to rename the archive; otherwise pmlogger_merge(1) is used to merge all of the archives for a single host and a single day into a new PCP archive, and the individual archives are removed. out with the old, in with the merged ✨
  • -P, --noproxy — okay so by default, on top of all the archives that local pmlogger(1) instances made down in the $PCP_ARCHIVE_DIR/<hostname> directories, pmlogger_daily will also handle the archives that got "pushed" over from a remote pmlogger(1) through the local pmproxy(1) and stashed under the $PCP_LOG_DIR/pmproxy/<hostname> directories. The -P option is the buzzkill that stops all that pmproxy archive processing. 💅
  • -p — bestie, this one is the "did you forget??" option. If you pass it to pmlogger_daily, the status of the daily processing gets polled, and if the whole daily pmlogger(1) situation (rotation, culling, rewriting, compressing, the works) hasn't happened in the last 24 hours, then it gets done right now. The whole point is to have pmlogger_daily called regularly with -p — in the default cron(8) setup that's 30 mins past the hour, every hour — so daily processing happens ASAP if it got missed at its regularly scheduled time (which is 00:10 by default), like if the system was down or suspended at that moment. With this option, pmlogger_daily just exits if the previous day's processing already got done. ✨
Heads up: this option is not used on platforms that support systemd(1), because the pmlogger_daily.timer service unit specifies a timer setting with Persistent=true. Also the -K and -p options to pmlogger_daily are mutually exclusive.
  • -Q, --proxyonly — the flip side of -P up there. Only process the archives that got "pushed" from a remote pmlogger(1) via the local pmproxy(1), and ignore every single archive made by local pmlogger(1) instances below the $PCP_ARCHIVE_DIR/<hostname> directories. 🙈
  • -r, --norewrite — this command line option is an override, and it blocks all archive rewriting with pmlogrewrite(1), no matter what rewriting rule files or directories happen to be hanging around. 🚫
  • -R, --rewriteall — sometimes PMDA changes mean every archive needs rewriting, not just the ones caught up in whatever current merging is going on. Like after a PCP upgrade where a new version of an existing PMDA has revised metadata. The -R command line option forces this universal-style rewriting. 🔥
The -R option to pmlogger_daily is mutually exclusive with both the -r and -M options.
  • *-s *size, --rotate=size — if the PCP "notices" file ($PCP_LOG_DIR/NOTICES) is bigger than 20480 bytes, pmlogger_daily renames it with a .old suffix and starts a fresh "notices" file. You can change that rotate threshold from 20480 to size bytes with the -s option. 🎀
  • -t period — to help with debugging or diagnosing intermittent failures, you can use the -t option. It turns on very verbose tracing (-VV) and captures the trace output in a file named $PCP_LOG_DIR/pmlogger/daily.datestamp.trace, where datestamp is the time pmlogger_daily was run in the format YYYYMMDD.HH.MM. And the period argument makes sure trace files created with -t are kept for period days and then discarded. 🧚
  • -V, --verbose — the output from the cron(8) execution of the scripts can be extended with the -V option, which enables verbose tracing of their activity. By default the scripts generate no output unless some error or warning condition comes up. A second -V increases the verbosity. Using -N together with -V maximizes the diagnostic capabilities for debugging. 🔍
  • *-x *time, --compress-after=time — archive data files can optionally be compressed after some period to save file system space. This is especially useful when you've got large numbers of pmlogger(1) processes under the control of pmlogger_daily. 📦

time is a time specification in the syntax of find-filter(1), so DD[:HH[:MM]]. The optional HH (hours) and MM (minutes) parts are 0 if not specified.

Some special values are recognized for time: 0 to apply compression as soon as possible, and forever or never to prevent any compression being done.

If transparent_decompress is enabled when libpcp was built (you can check with the pmconfig(1) -L option), then the default behaviour is compression "as soon as possible". Otherwise the default behaviour is to not compress files (which matches the historical default behaviour in earlier PCP releases).

The time can also be set using the $PCP_COMPRESSAFTER variable, set in either the environment or in a control file. If both $PCP_COMPRESSAFTER and -x specify different values for time, then the environment variable value is used and a warning is issued.

For important other detailed notes concerning volume compression, see the -K and -k options (above).
  • *-X *program, --compressor=program — this option specifies the program to use for compression — by default this is pmlogcompress(1). The environment variable $PCP_COMPRESS can be used as an alternative mechanism to define program. If both $PCP_COMPRESS and -X specify different compression programs, then the environment variable value is used and a warning is issued. 🗜️
  • *-Y *regex, --regex=regex — this option lets you specify a regular expression that causes files in the set of files matched for compression to be omitted. That way only the data file gets compressed, and it also stops the program from trying to compress it more than once. The default regex is
"\.(index|Z|gz|bz2|zip|xz|lzma|lzo|lz4|zst)$"

— such files get filtered using the -v option to egrep(1). The environment variable $PCP_COMPRESSREGEX can be used as an alternative mechanism to define regex. If both $PCP_COMPRESSREGEX and -Y specify different values for regex, then the environment variable value is used and a warning is issued. 🦋

  • -z — this option makes pmlogger_daily not "re-exec", see pmlogger(1), when it would otherwise choose to do so. It's intended only for QA testing. 💀
  • -Z — this option makes pmlogger_daily "re-exec", see pmlogger(1), whenever that is possible. Also intended only for QA testing. 💀
  • -?, --help — display the usage message and exit. 🫶

🫶 CALLBACKS 🫶

okay so bestie, on top of everything else, pmlogger_daily also supports these little "hooks" so you can run your own extra stuff at the key moments in the daily archive processing. These callbacks are controlled by variables you can set in the environment or in the control file. very choose-your-own-adventure of it 💅

⚠️ heads up: the merge callbacks and the autosaving thing are not enabled when only compression tasks are being attempted, i.e. when the -K command line option is used.

Also — and this part matters — all of the callback script execution and the autosave file moving runs as the non-privileged user pcp and group pcp, so you might need to set up the right permissions in advance. no cap, plan ahead 🎀

  • $PCP_MERGE_CALLBACK — so as each day's archive gets created by merging, and before any compression happens, if $PCP_MERGE_CALLBACK is defined, it's assumed to be a script that gets called with one argument: the name of the archive (stripped of any suffixes), so something like /some/directory/path/YYYYMMDD. The script has to be either a full pathname or something findable on the shell's $PATH. It runs in the foreground, so pmlogger_daily will literally just wait for it to finish.

If the control file has more than one $PCP_MERGE_CALLBACK spec, they run serially in the order they appear in the control file. And if $PCP_MERGE_CALLBACK is defined in the environment when pmlogger_daily runs, that's treated like it was the first one in the control file, i.e. it runs before any merge callbacks mentioned in the control file.

If the pcp-zeroconf package is installed, a special merge callback gets added to call pmlogger_daily_report(1) first, before any other merge callback options — but only for the primary pmlogger(1) instance. Go read pmlogger_daily_report(1) for the whole explanation of the pcp-zeroconf requirements. 📖

If pmlogger_daily is in "catch up" mode (aka more than one day's worth of archives need to be combined), then each callback gets executed once for every day's archive that's generated.

A typical use? Producing daily reports from the PCP archive — that needs to wait until the archive exists, but it's way more efficient if it happens before any potential compression of the archive. ✨

  • $PCP_COMPRESS_CALLBACK — if pmlogger_daily is run with -x 0 or $PCP_COMPRESSAFTER=0, then compression happens immediately after merging. As each day's archive gets compressed, if $PCP_COMPRESS_CALLBACK is defined, it's assumed to be a script called with one argument: the name of the archive (stripped of any suffixes), so something like /some/directory/path/YYYYMMDD. Again, the script must be a full pathname or findable on the shell's $PATH. Runs in the foreground, so pmlogger_daily waits for it to complete. 🦋

More than one $PCP_COMPRESS_CALLBACK in the control file? They run serially in order. Defined in the environment instead? Treated as the first one in the control file, i.e. it runs first.

In "catch up" mode (more than one day's worth of archives need compressing), each callback runs once for each day's archive that's compressed.

A typical use: keeping recent archives uncompressed so querying stays fast, but moving the older ones off to some other storage location once compression is done. iconic behavior honestly 💜

  • $PCP_AUTOSAVE_DIR — once the merging and possible compression is done by pmlogger_daily, if $PCP_AUTOSAVE_DIR is defined, then all the physical files that make up one day's archive get moved (autosaved) to the directory named by $PCP_AUTOSAVE_DIR. 🎀

The basename of the archive sets the reserved words DATEYYYY (year), DATEMM (month) and DATEDD (day), and these (along with LOCALHOSTNAME) can appear literally in $PCP_AUTOSAVE_DIR — they get substituted at execution time to build the destination directory name. Like so:

$PCP_AUTOSAVE_DIR=/gpfs/LOCALHOSTNAME/DATEYYYY/DATEMM-DATEDD

Note that those "date" reserved words match the date the archive data was collected, not the date pmlogger_daily was run. important distinction, don't mix them up 🙈

If $PCP_AUTOSAVE_DIR (after LOCALHOSTNAME and "date" substitution) doesn't exist, pmlogger_daily will try to create it, along with any parent directories that don't exist. Just know that this directory creation runs under the uid of the user pcp, so directories along the path to $PCP_AUTOSAVE_DIR might need to be writeable by that non-root user.

And when we say "move" the archives, we mean a paranoid checksum-copy-checksum-remove (using the -c option for pmlogmv(1)) that will bail if the copy fails or the checksums don't match. Because these archives are important and we cannot risk something like a full filesystem or a permissions issue messing with the copy process. 💀

In "catch up" mode (more than one day's worth of archives need combining), more than one day's archives could get copied in this step.

A typical use: make PCP archives on a local filesystem first, then once all the data for a single day is collected and merged, migrate that day's archive to a shared filesystem or a remote filesystem. This can allow automatic backup to off-site storage and/or reduce the number of I/O operations and filesystem metadata operations on the (potentially slower) non-local filesystem. 🔥

🌈 CONFIGURATION 🌈

Go check out pmlogger.control(5) for the description of the control file(s) that decide which pmlogger(1) instances and which archives are managed by pmlogger_check(1) and pmlogger_daily. 📋

And additionally, when pmproxy(1) archives are being processed, environment variables like $PCP_CULLAFTER or $PCP_COMPRESSAFTER can be set either for all remote hosts via the file $PCP_LOG_DIR/pmproxy/control and/or per-host via files named $PCP_LOG_DIR/pmproxy/<hostname>/control. slay, granular control 💅

🎀 FILES 🎀

  • $PCP_VAR_DIR/config/pmlogger/config.default — the default pmlogger(1) configuration file location for the local primary logger, typically generated automatically by pmlogconf(1). ✨
  • $PCP_ARCHIVE_DIR/<hostname> — the default location for archives of performance information collected from the host hostname.
  • $PCP_ARCHIVE_DIR/<hostname>/lock — a transient lock file to guarantee mutual exclusion during pmlogger(1) administration for the host hostname. If it's present, it can be safely removed if neither pmlogger_daily nor pmlogger_check(1) are running. 🫶
  • $PCP_ARCHIVE_DIR/<hostname>/Latest — the PCP archive folio created by mkaf(1) for the most recently launched archive containing performance metrics from the host hostname.
  • $PCP_LOG_DIR/NOTICES — the PCP "notices" file used by pmie(1) and friends. 👑
  • $PCP_LOG_DIR/pmlogger/pmlogger_daily.log — if the previous execution of pmlogger_daily produced any output, it's saved here. The normal case is no output at all, in which case the file doesn't exist. (no thoughts, no output, just vibes 🦋)
  • $PCP_ARCHIVE_DIR/SaveLogs — if this directory exists, then the log file from the -l argument for pmlogger_daily gets saved in this directory with the name format <date>-pmlogger_daily.log.<pid> or <date>-pmlogger_daily-K.log.<pid>. This lets you inspect the log file later, even if several pmlogger_daily executions have been launched in the meantime. And because the PCP archive management tools run under the $PCP_USER account pcp, $PCP_ARCHIVE_DIR/SaveLogs typically needs to be owned by the user pcp. 💜
  • $PCP_ARCHIVE_DIR/<hostname>/SaveLogs — if this directory exists, then the log file from the -l argument of a newly launched pmlogger(1) for hostname gets saved in this directory with the name archive.log where archive is the basename of the associated pmlogger(1) PCP archive files. Same deal — lets you inspect the log file later, even if several pmlogger(1) instances for hostname have been launched in the interim. And because the PCP archive management tools run under the uid of the user pcp, $PCP_ARCHIVE_DIR/<hostname>/SaveLogs typically needs to be owned by the user pcp.
  • $PCP_LOG_DIR/pmlogger/.NeedRewrite — if this file exists, it's treated as equivalent to using -R on the command line, and the file gets removed once all rewriting has been done. 📝

💖 PCP ENVIRONMENT 💖

  • $PCP_SPACELIMIT — if set, this specifies a maximum allowed total file system space (in kilobytes, or with optional K/M/G suffix as for the -d option) for each per-host archive directory under $PCP_ARCHIVE_DIR or $PCP_REMOTE_ARCHIVE_DIR. pmlogger_daily uses this value when enforcing archive retention limits for file system space — unless the -d option is provided, in which case the command line flag overrides the environment variable and a warning is issued. Enforcement happens after each run of pmlogger_daily and only applies at that time — not continuously. ⏰

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable can be used to specify an alternative configuration file, as described in pcp.conf(5). 🌸

😭 COMPATIBILITY ISSUES 😭

Earlier versions of pmlogger_daily used find(1) to locate files for compressing or culling, and the -k and -x options took only integer values meaning "days". The semantics of that were pretty loose, ngl, given that find(1) offers different precision and semantics across platforms. 🫠

The current implementation of pmlogger_daily uses find-filter(1), which provides high precision intervals and semantics that are relative to the time of execution and are consistent across platforms. glow up fr fr ✨

🌈 SEE ALSO 🌈

okay so if you wanna go deeper (and bestie, you know you do), here's the whole extended universe. every single one of these is your next hyperfixation:

like. it's a whole friend group. read them all, they're literally right there 💅

okay that's genuinely the whole page, bestie — you made it to the end and i'm so proud of you 🥹🫶

for the record: this is the Performance Co-Pilot manual, tagged PCP, and the page itself is PMLOGGER_DAILY(1) — dated right there in the footer like every good little manpage does. go log something and slay ✨💖

📦 where this page comes from