PMLOGEXTRACT(1)

✨ NAME ✨

pmlogextract — the girlboss tool that reduces, extracts, concatenates and merges Performance Co-Pilot archives 💖

---

🌸 SYNOPSIS 🌸

pmlogextract [-dfmwxz?] [-c configfile] [-D debug] [-S starttime] [-s samples] [-T endtime] [-V version] [-v volsize] [-Z timezone] input [...] output

---

💅 DESCRIPTION 💅

okay bestie, so pmlogextract reads one or more Performance Co-Pilot (PCP) archives (those are your input args) and creates a merged and/or reduced PCP archive called output. slay. 🎀

Each input argument is either a name or a comma-separated list of names, and each name is either:

• the name of one file from an archive, or
• the base name of an archive, or
• the name of a directory containing one or more archives.

The way merging works is controlled by how many input archives you give it, and the way data reduction works is controlled by the command line arguments. no cap. The input arguments must be archives created by pmlogger(1) with performance data collected from the same host — but usually over different time periods, and possibly (although not usually) with different performance metrics being logged. iykyk.

If you only specify one input, then the default behavior literally just copies the input PCP archive into the output PCP archive (with a possible conversion to a newer version of the archive format, see -V below, bestie). But when two or more PCP archives are specified as input, the archives get merged (or concatenated, cute) and written to output. 👑

In the output archive, a <mark> record may be inserted at a time just past the end of each of the input archives, to flag a possible temporal discontinuity between the end of one input archive and the start of the next one. See the MARK RECORDS section below for more. There is no <mark> record after the end of the last (in temporal order) of the records from the input archive(s). periodt. ✨

---

🦋 OPTIONS 🦋

The available command line options are, and yes we're listing every single one because we're thorough like that 💖:

• -c config, --config=config — extract only the metrics specified in config from the input PCP archive(s). The config syntax that pmlogextract accepts is explained in more detail in the CONFIGURATION FILE SYNTAX section. obsessed with a good filter ngl.
• -d, --desperate — desperate mode 😭. Normally, if a fatal error occurs, all trace of the partially written PCP archive output is removed. With the -d option, the output archive is not removed. it's giving "we're keeping the mess".
• -f, --first — for most common uses, all of the input archives will have been collected in the same timezone. But if that's not the case, then pmlogextract must choose one of the timezones from the input archives to be used as the timezone for the output archive. The default is to use the timezone from the last input archive. The -f option forces the timezone from the first input archive to be used instead. ✨
• -m, --mark — as described in the MARK RECORDS section below, sometimes it's possible to safely omit <mark> records from the output archive. If the -m option is specified, then the epilogue and prologue test is skipped and a <mark> record will always be inserted at the end of each input archive (except the last). This is the original behaviour for pmlogextract. vintage, iconic.
• -S starttime, --start=starttime — define the start of a time window to restrict the records processed; refer to PCPIntro(1). See also the -w option. 🌸
• -s samples, --samples=samples — the argument samples defines the number of samples (or records) to be written to output. If samples is 0 or -s is not specified, pmlogextract will continue until the end of all the input archives, or until the end of the time window as specified by -T, whichever comes first. The -s option will override the -T option if it occurs sooner. fr fr, first one wins.
• -T endtime, --finish=endtime — define the end of a time window to restrict the records processed; refer to PCPIntro(1). See also the -w option. 💜
• -V version, --outputversion=version — each PCP archive has a version for the physical record format, currently 2 or 3. By default, the output archive is created with a version equal to the maximum of the version of the input archives. The -V option may be used to explicitly force the version for output, provided version is no smaller than the archive version that would have been chosen by the default rule.

For example, specifying -V 3 may be used to produce a version 3 output archive from input archives that could be a mixture of version 2 and/or version 3. 💅

• -v volsize — the output archive is potentially a multi-volume data set, and the -v option causes pmlogextract to start a new volume after reaching an archive volume size of volsize. If volsize is an integer then at most volsize records will be written to each volume. If volsize is an integer suffixed by b or bytes then at most volsize bytes will be written out to each volume. Other viable file size units include: K, Kb, KiB, Kbyte, Kilobyte for kilobytes, and M, Mb, MiB, Mbyte, Megabyte for megabytes, and G, Gb, GiB, Gbyte, Gigabyte for gigabytes. These units may be optionally suffixed by an s and may be of mixed case. Alternatively volsize may be an integer or a floating point number suffixed using a time unit as described in PCPIntro(1) for the interval argument (to the standard PCP -t command line option).

Some examples of different formats:

-s 100
-s 100bytes
-s 100K
-s 100Mb
-s 10Gbyte
-s 10mins
-s 1.5hours

The default is for pmlogextract to produce as few volumes as possible.

Independent of any -v option, each volume of an archive is limited to no more than 2^31 bytes, so pmlogextract will automatically create a new volume for the archive before this limit is reached. 🫶

• -w — where -S and -T specify a time window within the same day, the -w flag will cause the data within the time window to be extracted, for every day in the archive. For example, the options -w -S @11:00 -T @15:00 specify that pmlogextract should include archive records only for the periods from 11am to 3pm on each day. When -w is used, the output archive will contain <mark> records to indicate the temporal discontinuity between the end of one time window and the start of the next. 🌈
• -x — it is expected that the metadata (name, PMID, type, semantics and units) for each metric will be consistent across all of the input PCP archive(s) in which that metric appears. In rare cases, e.g. in development, in QA and when a PMDA is upgraded, this may not be the case and pmlogextract will report the issue and abort without creating the output archive. This is done so the problem can be fixed with pmlogrewrite(1) before retrying the merge. In unattended or QA environments it may be preferable to force the merge and omit the metrics with the mismatched metadata. The -x option does this. 😤
• -Z timezone, --timezone=timezone — use timezone when displaying the date and time in diagnostics. Timezone is in the format of the environment variable TZ as described in environ(7). The default is to initially use the timezone of the local host. ⭐
• -z, --hostzone — use the local timezone of the host from the input archive(s) when displaying the date and time in diagnostics. The default is to initially use the timezone of the local host. ✨
• -?, --help — display usage message and exit. the classic. 💅

---

🎀 CONFIGURATION FILE SYNTAX 🎀

The configfile contains metrics of interest — only those metrics (or instances) mentioned explicitly or implicitly in the configuration file will be included in the output archive. Each specification must begin on a new line, and may span multiple lines in the configuration file. Instances may also be specified, but they are optional. The format for each specification is

metric

or

metric [ instance ... ]

where metric may be a leaf or a non-leaf name of a metric in the Performance Metrics Name Space (PMNS, see PMNS(5)). If a metric refers to a non-leaf node in the PMNS, pmlogextract will recursively descend the PMNS and include all metrics corresponding to descendent leaf nodes. 🌳

Instances are optional and are specified as a list space (or comma) separated of instance identifiers, with the list enclosed by square brackets. Each instance identifier may be a number or a string (enclosed in single or double quotes). Instance identifiers that are numbers are assumed to be internal instance identifiers, else the string values are assumed to be external instance identifiers; see pmGetInDom(3) for more information. If no instances are given, then all instances of the associated metric(s) will be extracted. no cap.

Any additional white space is ignored and comments may be added with a `#' prefix. 💖

---

🌟 CONFIGURATION FILE EXAMPLE 🌟

This is an example of a valid configfile:

#
# config file for pmlogextract
#

kernel.all.cpu
kernel.percpu.cpu.sys ["cpu0","cpu1"]
disk.dev ["dks0d1"]

ate, honestly. 💅✨

💾 MARK RECORDS

okay bestie, this one is important so pay attention 💖 When more than one input archive is feeding performance data into your output archive, then <mark> records might get inserted to flag a possible temporal discontinuity in the performance data. iykyk.

A <mark> record is literally just a timestamp and no performance data. its whole job is to say "hey, there's a stretch of time in this PCP archive where we genuinely do not know the values of any performance metrics" — because no pmlogger(1) was out there collecting anything during that window. and since those gaps are usually tied to a service restart, a pmcd(1) restart, or a whole system reboot, there can be considerable doubt about whether the performance data is even continuous across that period. 😭

Most current archives get made with a prologue record at the start and an epilogue record at the end. those records tell you the state of pmcd(1) at the time, and pmlogextract can use them to figure out that there's no discontinuity between the end of one archive and the next output record — which means the <mark> record can safely be left out of the output archive. slay. ✨

Wanna see the rationale behind <mark> records in action? here's the vibe check. imagine one input archive that starts at 00:10 and ends at 09:15 on the same day, and another input archive that starts at 09:20 on the same day and ends at 00:10 the following morning. this is super common for archives managed and rotated by pmlogger_check(1) and pmlogger_daily(1).

The output archive created by pmlogextract would contain:

00:10.000    first record from first input archive
...
09:15.000    last record from first input archive
09:15.001    <mark> record
09:20.000    first record from second input archive
...
01:10.000    last record from second input archive

so the time period where performance data is missing starts just after 09:15 and ends just before 09:20. when you process the output archive with any of the PCP reporting tools, that <mark> record is what tells it "missing data here bestie". for example, using that output archive above, imagine you're reporting the average I/O rate at 30 minute intervals aligned on the hour and half-hour. the I/O count metric is a counter, so the average I/O rate needs two valid values from consecutive sample times. so you'd get values for all the intervals ending at 09:00, then no values at 09:30 because of the <mark> record, then no values at 10:00 because the ``prior'' value at 09:30 is not available, then the rate would be reported again at 10:30 and keep going every 30 minutes until the last reported value at 01:00. 🫶

and if you wanna know whether <mark> records are even in a PCP archive, you can check with pmlogdump(1) — a timestamp plus the annotation <mark> is how a <mark> record shows up. iconic.

---

🔍 METADATA CHECKS

When you specify more than one input archive, pmlogextract does a bunch of checks to make sure the metadata is consistent for metrics that show up in more than one of the input archives. these checks include:

• metric data type is the same
• metric semantics are the same
• metric units are the same
• metric is either always singular or always has the same instance domain
• metrics with the same name have the same PMID
• metrics with the same PMID have the same name

If any of these checks fail, pmlogextract reports the details and aborts without creating the output archive. no cap, it just stops. 💀

To fix these semantic issues, use pmlogrewrite(1) to translate the input archives into equivalent archives with consistent metadata before you run pmlogextract. 💅

And hey — refer to the -x and -d command line options above for alternatives to the default handling of errors during metadata checks.

---

⚠️ CAVEATS

The prologue metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host, and pmcd.pmlogger.port), which pmlogger records automatically at the start of the archive, may not be present in the archive output by pmlogextract. these metrics are only relevant while the archive is being created, and have zero significance once recording has finished. periodt.

---

🚨 DIAGNOSTICS

All error conditions detected by pmlogextract get reported on stderr with a textual (if sometimes terse) explanation.

If one of the input archives contains no archive records, then an ``empty archive'' warning is issued and that archive is skipped.

Should one of the input archive(s) be corrupted (this can happen if the pmlogger instance writing the archive suddenly dies 😵), then pmlogextract will detect it and report the position of the corruption in the file, and any subsequent information from that archive will not be processed.

If any error is detected, pmlogextract will exit with a non-zero status.

---

📁 FILES

For each of the input and output archive, several physical files are used.

• archive.meta — metadata (metric descriptions, instance domains, etc.) for the archive
• archive.0 — initial volume of metrics values (subsequent volumes have suffixes 1, 2, ...) - for input these files may have been previously compressed with bzip2(1) or gzip(1) and thus may have an additional .bz2 or .gz suffix.
• archive.index — temporal index to support rapid random access to the other files in the archive.

---

🌍 PCP ENVIRONMENT

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5). 🌸

For environment variables affecting PCP tools, see pmGetOptions(3).

---

🐛 DEBUGGING OPTIONS

The -D or --debug option enables the output of additional diagnostics on stderr to help triage problems — although ngl, the information is sometimes cryptic and primarily intended to provide guidance for developers rather than for end-users. 🙈 debug is a comma separated list of debugging options; use pmdbg(1) with the -l option to obtain a list of the available debugging options and their meaning.

Debugging options specific to pmlogger are as follows:

• appl0 — parser diagnostics for configfile if -c specified
• appl1 — memory allocations and reclist operations
• appl2 — time window and end-of-file tests
• appl3 — input and output archive version decisions
• appl4 — instance domain juggling
• appl5 — output volume switching

---

👀 SEE ALSO

PCPIntro(1), pmlc(1), pmlogdump(1), pmlogger(1), pmlogreduce(1), pmlogrewrite(1), pcp.conf(5), pcp.env(5) and PMNS(5).

---

and that's the whole page bestie, you made it to the end 🎀 this little masterpiece is brought to you by Performance Co-Pilot 💖 under the PCP banner, filed under PMLOGEXTRACT(1) — go forth and merge those archives, you absolute legend 👑 ✨🦋