PMDAOPENMETRICS(1)

💖 NAME 💖

pmdaopenmetrics — the OpenMetrics PMDA. That's it, that's the name, she's iconic and she knows it 🎀

🦋 SYNOPSIS 🦋

$PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics [-D] [-n] [-c config] [-d domain] [-l logfile] [-R refresh_timeout] [-r root] [-t timeout] [-u user]

✨ DESCRIPTION ✨

okay so bestie, pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) — and the way it works is so cool — it dynamically creates PCP metrics from configured OpenMetrics endpoints, which are the ones that provide HTTP based access to application metrics. Basically the PMDA is a whole bridge between Prometheus and PCP, and it lets PCP just. casually. ingest performance data from more than 650 registered end-points plus a whole bunch of other application specific end-points. no cap. 🔥

The default config directory is $PCP_PMDAS_DIR/openmetrics/config.d/ — go peek at ``CONFIGURATION SOURCES'' below for the tea on that one. The default URL fetch timeout is 2 seconds. And the default user, if you don't hand it over with the -u option, is just... the current user. you're the main character, congrats 💅

If the -n option is given, the list of configuration files will not be sorted prior to processing. It is sorted by default, but heads up — that sorting can get expensive if you have a truly unhinged number of configuration files (URLs and/or scripts). 🫶

---

Now for the flags, one at a time, because we do not skip:

• -D — give it this and extra diagnostic messages get written to the PMDA log file, which by default lives at $PCP_LOG_DIR/pmcd/openmetrics.log (see also -lbelow). Also, the metric openmetrics.control.debug controls that same debug flag, and you can set it like this:

pmstore openmetrics.control.debug value

where value is either 1 (to enable verbose log messages) or 0 (to disable verbose log messages). This is especially useful for examining the http headers passed to each fetch request, the filter settings, and other processing details that get logged when the debugging flag is on. lowkey the most useful thing ever when something's acting up 😭

• -d — override the default performance metrics domain number, which defaults to 144. > bestie, it is strongly recommended not to change this. The domain number has to be different for every PMDA on the one host, and the same domain number should be used for pmdaopenmetrics PMDA on all hosts. Also see the -r option, which lets you change the root of the dynamic namespace away from the default openmetrics.
• -l — use logfile as the destination for PMDA messages instead of the default $PCP_LOG_DIR/pmcd/openmetrics.log. Special case incoming: logfile can be "-" to send messages to the stderr stream instead, like -l-. Normally that would be the stderr stream for the parent process, pmcd(1), which may itself have redirected stderr. That redirection is usually most useful in a containerized environment, or when you're using dbpmda(1). 🧚
• -r — lets the root of the dynamic namespace be changed to root, instead of the default openmetrics. Team it up with the other command line options and pmdaopenmetrics can be deployed as a totally different PMDA, with a distinct metrics namespace and metrics domain, on the same host system. ⚠️ Note that all PMDAs require a unique domain number, so the -d option must also be specified. Using -r may also change the defaults for some other command line options, e.g. the default log file name and the default configuration directory.
• -R — lets you configure the timeout, in seconds, between cluster refreshes. The default value is 10 seconds. ✨

🌸 CONFIGURATION SOURCES 🌸

As it runs, pmdaopenmetrics periodically, recursively scans the $PCP_PMDAS_DIR/openmetrics/config.d directory (or whichever directory you specified with the -c option), looking for source URL files (*.url) and executable scripts or binaries. 👑

Any files that don't have the .url suffix, or that aren't executable, get ignored — and that's on purpose, because it means documentation files like "README" and non-executable "common" script function definitions can just chill there without anyone thinking they're config files. periodt.

A remote server does not have to be up, or stay running — the PMDA tolerates remote URLs that come and go over time. It will relay data and metadata when/if they're available, and it will return errors when/if they're down. PCP metric IDs, internal and external instance domain identifiers are persisted, and they'll be restored when individual metric sources become available and/or when the PMDA is restarted. 🦋

Plus — and this is the part that ate — the PMDA checks directory modification times and will rescan for new or changed configuration files dynamically. So you do not need to restart the PMDA when adding, removing or changing configuration files. we love a queen who keeps up 💜

💅 URL SOURCES 💅

Every file with the .url suffix that you find in the config directory or sub-directory contains one complete HTTP or HTTPS URL, at which pmdaopenmetrics can reach an OpenMetrics endpoint. Local file access is supported too, with a conventional file:///somepath/somefile URL — in that case, somepath/somefile should contain openmetrics formatted metric data. 🍓

The first line of a .url config file should be the URL, as described above. Subsequent lines, if any, are prefixed with a keyword that can be used to alter the http GET request. A keyword must end with ':' (colon), and the text extends to the end of the line. Comment lines that start with # and blank lines are ignored. The only currently supported keywords are HEADER: and FILTER:. 🙈

HEADER: headername: value ... to end of line

That one adds headername and its value to the headers passed in the http GET request for the configured URL. Here's an example configuration file that provides 3 commonly used headers and an authentication token — look at her go:

http://somehost/path/endpoint.html
# this is a comment
HEADER: Accept: text/html
HEADER: Keep-Alive: 300
HEADER: Connection: keep-alive
HEADER: Authorization: token ABCDEF1234567890

As mentioned above, header values extend to the end of the line. They may contain any valid characters, including colons. Multiple spaces will be collapsed down to a single space, and leading and trailing spaces are trimmed. A common use for headers is to configure a proxy agent and the assorted parameters it may require. 😌

🔥 METRIC FILTERING 🔥

Metric filtering is a configuration file feature that lets ingested metrics be included or excluded, i.e. filtered. This is useful because most end-points return multiple metrics, and usually only some of them are interesting for monitoring purposes. The syntax is:

FILTER: INCLUDE METRIC regex

or

FILTER: EXCLUDE METRIC regex

Dynamically created metric names that match regex will be either included or excluded in the name space, as specified. Note that only the PMNS leaf component of the metric name (as ingested from the URL source) is compared with the regex pattern. The simple rule is: the first matching filter regex for a particular metric leaf name is the rule that prevails. 🎯

If no filter regex matches (or there are no filters at all), then the metric is included by default — i.e. the default filter, if none are specified, is FILTER: INCLUDE METRIC .*. This is backward compatible with older versions of the configuration file that didn't support filters. 💖

Multiple FILTER: lines would normally be used. e.g. to include some metrics but exclude all others, use FILTER: EXCLUDE METRIC .* as the last of several filters that include the desired metrics. Conversely, to exclude some metrics but include all others, use FILTER: EXCLUDE METRIC regex. In that case it's not necessary (though it doesn't hurt) to specify the final FILTER: INCLUDE METRIC .*, because — as stated above — any metric that doesn't match any filter regex will be included by default. ✨

👑 LABEL FILTERING 👑

Label filtering uses similar FILTER: syntax and semantics as metric filtering. FILTER: EXCLUDE LABEL regex will delete all labels with a label name matching regex from all metrics defined by the configuration file. The same rules as for metric filters apply for label filters too — an implicit rule FILTER: INCLUDE LABEL .* applies to all labels that do not match any earlier label filter rule. 💜

FILTER: OPTIONAL LABEL regex specifies that matching label names are to be included in the returned metric labelsets (i.e. included), but are not to be used as part of the the external instance names. All included labels that are not optional (i.e. the intrinsic labels) will be concatenated together and used for external instance naming. In addition, non-intrinsic labels (i.e. labels tagged as OPTIONAL) will have the PM_LABEL_OPTIONAL flag set in the labelsets returned by notes callbacks. This flag affects how the labels are used in certain clients. For further details, see pmLookupLabels(3) and related man pages for further details. And note — external instance names begin with the unique numeric internal instance identifier followed by a space, so external instance names are always unique. 🌈

⚠️ Caution is needed with label filtering, bestie, because by default all labels are used to construct the PCP instance name. By excluding some labels (or changing them to optional), the instance names will change. In addition, excluding all labels for a particular metric changes that metric to be singular, i.e. have no instance domain. By excluding some labels, different instances returned by the URL or scripted configuration entry for the same metric may become duplicates. When such duplicates occur, the last duplicate instance returned by the end-point URL or script prevails over any earlier instances.

For these reasons, it is recommended that label filtering rules be configured when the configuration file is first defined, and not changed thereafter. If a label filtering change is required, the configuration file should be renamed — which effectively defines a new metric (or set of peer metrics as returned by the URL or script), with the new (or changed) instance naming. 🎀

Unrecognized keywords in configuration files are reported in the PMDA log file but otherwise ignored. 💀

---

🦋 SCRIPTED SOURCES 🦋

okay so not everything has to be a URL, bestie 💅 — you can also drop executable scripts into the $PCP_PMDAS_DIR/openmetrics/config.d directory (or any sub-directory of it, we love a nesting moment 🪆) and the PMDA will run them. Whatever they spit out on stdout that's in OpenMetrics format gets parsed exactly like it came from a URL or a file. Same energy, different source. And the stderr stream from a script? That goes straight to the PMDA log file, which by default lives at $(PCP_LOG_DIR)/pmcd/openmetrics.log 📝.

⚠️ Real talk though: scripted sources do not support label filtering or metric filtering (the stuff described above for URL sources). Nope, not a thing here. Instead your script just does its own filtering with sed(1), awk(1), or literally whatever tool you vibe with ✨.

Here's a lil baby example of a scripted config entry, she's cute:

#! /bin/sh
awk '{
    print("# HELP loadavg local load average")
    print("# TYPE loadavg gauge")
    printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
    printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
    printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
}' /proc/loadavg

And when you run it, this is the OpenMetrics-formatted metric data she produces 🌸:

# HELP loadavg local load average
# TYPE loadavg gauge
loadavg {interval="1-minute"} 0.12
loadavg {interval="5-minute"} 0.27
loadavg {interval="15-minute"} 0.54

So like, if you saved that script, made it executable, and dropped it at $PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh? You'd get a brand new PCP metric named openmetrics.local.system.loadavg 🎀 and she'd have three instances for the current load average values: 1-minute, 5-minute and 15-minute. Iconic.

Scripted config entries can produce more than one PCP leaf metric name, she's multifaceted. For example, that same "system.sh" script could also export other metrics like CPU statistics by reading /proc/stat on the local system. Those extra metrics would show up as peer metrics in the same PCP metric subtree, besties hanging out together 🫶. Oh and heads up — for CPU counters, the metric type definition should be counter, not gauge. Get it right or the vibes are off 😤. For the full tea on the OpenMetrics exposition formats, go read https://github.com/OpenObservability/OpenMetrics/blob/master/specification/OpenMetrics.md.

---

🔐 SELinux CONSIDERATIONS 🔐

Scripted config files get executed by the pmdaopenmetrics PMDA with the same SELinux context and policy as the local pmcd(1). For simple scripts — like the load average example we just did — this is normally totally fine, no stress 💆.

But. AVC errors can absolutely pop up for scripts that make library or system calls that the prevailing SELinux context and policies restrict 🙈. And in those cases, you cannot just unilaterally grant pmcd or it's PMDAs an unconfined execution policy. Not a thing. So for these site specific cases you'll need to create a local SELinux policy module. Here's the recipe: capture the AVC record(s) from the local audit log, generate a local policy module using audit2allow, then load the new module using semodule, like so:

$ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
| audit2allow -M mypolicy
$ sudo semodule -i mypolicy.pp

And if these local policies need to persist across reboots, add a scriptlet similar to the above example to the local pmcd RC file (typically /etc/pcp/pmcd/rc.local). For further details, see audit2allow(1) and semodule(1) 📚.

---

🏷️ METRIC NAMING 🏷️

Every metric from a file named JOB.* gets exported as PCP metrics with the openmetrics.JOB metric name prefix. So the JOB name must be a valid non-leaf name for PCP PMNS metric names, no cap 🧢.

And if the JOB name has multiple dot-separated components, the resulting PMNS names will include those components — and you gotta be careful to make sure there are no overlapping definitions. Like, metrics returned by JOB.response may overlap or conflict with metrics returned by JOB.response.time 💀. Watch out for that.

Config file entries (URLs or scripts) found in subdirectories of the config directory also result in hierarchical metric names, because of course they do, we love a hierarchy. For example, a config file named $PCP_PMDAS_DIR/openmetrics/config.d/mysource/latency/get.url will result in metrics being created (by fetching that source URL) below openmetrics.mysource.latency.get in the PCP namespace ✨. Scripts found in subdirectories of the config directory similarly result in hierarchical PCP metric names. Same vibe, script edition.

---

🔄 DYNAMIC METRIC NAMES 🔄

So as we said up top, you can make changes and add new stuff to files in the configuration directory without restarting the PMDA. No restart, no downtime, no drama 💅. These changes are detected automatically and the PCP metric names below openmetrics in the PMNS get updated accordingly — meaning new metrics get dynamically added and/or existing metrics get removed. Live edits, main character energy 👑.

On top of that, pmdaopenmetrics honors the PMCD_NAMES_CHANGE pmFetch(3) protocol that was introduced in PCP version 4.0 🎉. Specifically: if openmetrics metrics are being logged by a PCP version 4.0 or later pmlogger(1), then new metrics that appear as a result of changes in the PMDA configuration directory will automatically start to be logged — provided the root of the openmetrics PMDA namespace is configured for logging in the pmlogger configuration file. See pmlogger(1) for details. Here's an example of such a pmlogger configuration file:

log mandatory on 2 second {
     # log all metrics below the root of the openmetrics namespace
     openmetrics
}

---

🧬 METADATA 🧬

Metric data returned by URL or scripted configuration files may contain metadata that the openmetrics PMDA can use to specify the semantics, data type, scaling and units of dynamically created metrics. Fancy, right? 💖 This metadata is prefixed with # PCP5 or # PCP in the ingested metric data. For additional info about PCP metadata, see pmLookupDesc(3) and pmParseUnitsStr(3) and the examples in the shipped configuration files 📖.

In-line "PCP5" metadata must be supplied by the metrics source end-point (URL or script). The alternative is to specify it in the URL configuration file directly — which has the advantage of not having to modify the source/end-point if the metadata is incorrect or missing 🫶. And important: metadata specified in the URL configuration file over-rides any in-line metadata. Periodt.

The configuration file syntax for specifying metadata is:

METADATA: regex type indom semantics units ... to EOL

Where:

• METADATA: is literal
• regex is an extended regular expression to match one or more metric names returned by the URL
• type is one of the PCP numeric types (32, u32, 64, u64, float or double)
• indom is an arbitrary instance domain name, or PM_INDOM_NULL
• semantics is either counter, instant or discrete
• units is either none or a string extending to end of line that is parsable by pmParseUnitsStr(3) — i.e. the units, dimensions and scaling to be used for matching metrics

Here's an example config file that ingests metrics from the Grafana /metrics end-point on localhost, filters out all metrics returned by that URL except for grafana_api_response_status_total, and then specifies the metric type is an unsigned 32 bit integer with a non-singular instance domain named response and counter semantics with units of count 🎀:

http://localhost:3000/metrics
FILTER: INCLUDE METRIC grafana_api_response_status_total
FILTER: EXCLUDE METRIC .*
METADATA: grafana_api_response_status_total u32 response counter count

📌 Note that the name in the indom field is presently ignored unless it is PM_INDOM_NULL — in which case the metric has no instance domain (i.e. it's singular), even if it has labels which would otherwise be used for instance naming. Yeah. That's just how it is, bestie 🙈.

---

🎛️ CONTROL METRICS 🎛️

The PMDA maintains special control metrics, as described below 👇. Apart from openmetrics.control.debug, each of these metrics has one instance for each configured metric source. All of these metrics have integer values with counter semantics, except openmetrics.control.status, which has a string value (she's different, she's special 💅).

⚠️ Super important: fetching any of the openmetrics.control metrics will only update the counters and status values if the corresponding URL is actually fetched. If the source URL is not fetched, the control metric values do not trigger a refresh, and the control values reported represent the most recent fetch of each corresponding source. iykyk 🔥.

The instance domain for the openmetrics.control metrics is adjusted dynamically as new sources are discovered. And if there are no sources configured, the metric names are still defined but the instance domain will be empty and a fetch will return no values. Nothing. Nada. 🫥

• openmetrics.control.status — a string representing the status of the last fetch of the corresponding source. This will generally be success for an http response code of 200. This metric can be used for service availability monitoring — provided, as stated above, that the corresponding source URL is fetched too 👀.
• openmetrics.control.status_code — this one is similar to openmetrics.control.status except that it's the integer response code of the last fetch. A value of 200 usually signifies success and any other value means failure. This metric can also be used for service availability monitoring, with the same caveats as openmetrics.control.status 📊.
• openmetrics.control.calls — total number of times each configured metric source has been fetched (if it's a URL) or executed (if it's a script), since the PMDA started. This metric has counter semantics and would normally be converted to a rate/second by client tools 📈.
• openmetrics.control.fetch_time — total time in milliseconds that each configured metric source has taken to return a document, excluding the time to parse the document. This metric has counter semantics and would normally be rate converted by client tools — but it's also useful in raw form as the accumulated parse time since the PMDA was started ⏱️.
• openmetrics.control.parse_time — total time in milliseconds that each configured metric source has taken to parse each document, excluding the time to fetch the document. This metric has counter semantics and would normally be rate converted by client tools — but again, also useful in raw form as the accumulated parse time since the PMDA was started ⏳.

When converted to a rate, the calls metric represents the average fetch rate of each source over the sampling interval (the time delta between samples). And the fetch_time and parse_time counters, when converted to a rate, represent the average fetch and parsing latency (respectfully), during the sampling interval 🌸.

The openmetrics.control.debug metric has a singular value, defaulting to 0. If a non-zero value is stored into this metric using pmstore(1), additional debug messages will be written to the PMDA log file 🐛✨.

🌸 LIMITATIONS 🌸

okay so bestie, there are some hard limits here and they come from pmdaopenmetrics and libpcp internals, not from anyone being dramatic 💅 so don't come for us:

• the number of sources: 4095
• metrics within each source: 1024
• instances for each metric: 4194304

that's it, that's the ceiling, no cap 🫶

---

✨ INSTALLATION ✨

to install the OpenMetrics PMDA, run the Install script as root:

# cd $PCP_PMDAS_DIR/openmetrics
# ./Install

and to uninstall, also as root, you do this:

# cd $PCP_PMDAS_DIR/openmetrics
# ./Remove

🚨 real talk: pmdaopenmetrics gets launched by pmcd(1) and should never be executed directly. like, ever. it's giving "don't poke the bear" energy 😭

the Install and Remove scripts are the ones that notify pmcd when the agent is installed or removed, so let them do their thing 👑

and here's the cute part — when scripts and .url files get added, removed or changed in the configuration directory, it's usually not necessary to restart the PMDA. the changes will be detected and managed on subsequent requests to the PMDA. lowkey iconic, it just handles itself 🦋

---

🎀 FILES 🎀

• $PCP_PMDAS_DIR/openmetrics/Install — the installation script for the pmdaopenmetrics agent, the one that does the setting-up 🫶
• $PCP_PMDAS_DIR/openmetrics/Remove — the undo installation script for the pmdaopenmetrics agent, aka the "jk jk nevermind" button 🙈
• $PCP_PMDAS_DIR/openmetrics/config.d/ — this one contains URLs and scripts used by the pmdaopenmetrics agent as sources of openmetrics metric data ✨
• $PCP_LOG_DIR/pmcd/openmetrics.log — the default log file for error messages from pmdaopenmetrics, so when it's feeling messy, this is where it spills the tea 💀
• $PCP_VAR_DIR/config/144.* — files containing internal tables for metric and instance ID number persistence (domain 144) 🍓

---

💜 PCP ENVIRONMENT 💜

environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP 🌈

on each installation, the file /etc/pcp.conf contains the local values for these variables 📝

and the $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5) — so if you wanna point it somewhere else, that's your moment, main character 💅

---

🌟 SEE ALSO 🌟

PCPIntro(1), audit2allow(1), pmcd(1), pminfo(1), pmlogger(1), pmstore(1), PMWEBAPI(3), pmFetch(3), pmLookupLabels(3), semodule(1), and https://prometheus.io/docs/instrumenting/exposition_formats 🔥

go read those, iykyk 📚

---

and that's a wrap on the whole page bestie 🎬💖 this was the Performance Co-Pilot PCP version of PMDAOPENMETRICS(1). obsessed with you for reading all the way through, periodt 👑🌸