PMDAOPENTELEMETRY(1)

📦
version 7.2.2 · 5 versions
, in arch rolling, fedora 43, fedora 44, ubuntu stonking

💖 pmdaopentelemetry — the OpenTelemetry PMDA ✨

okay bestie, welcome to pmdaopentelemetry(1), from the General Commands Manual section of the PCP fam 🫶. she's a Performance Metrics Domain Agent and she is literally out here making PCP metrics out of thin air, no cap.

✨ SYNOPSIS ✨

$PCP_PMDAS_DIR/opentelemetry/pmdaopentelemetry [-D debug] [-n] [-c config] [-d domain] [-l logfile] [-R refresh_timeout] [-r root] [-t timeout] [-u user]

🌸 DESCRIPTION 🌸

so like. pmdaopentelemetry is a Performance Metrics Domain Agent (PMDA), and her whole thing is that she dynamically creates PCP metrics from configured OpenTelemetry endpoints — the ones that give you HTTP-based access to application metrics 💅. she's not reading from some fixed table, she's out here building the metrics as she goes. main character energy fr fr.

the default config directory is $PCP_PMDAS_DIR/opentelemetry/config.d/ — peek at the CONFIGURATION SOURCES section below for the whole tea ☕.

the default URL fetch timeout is 2 seconds. clock is ticking bestie ⏱️

the default user, if you don't say otherwise with the -u option, is the "pcp" user. she's got a whole identity already, she doesn't need your input (but she'll take it if you insist).

if you pass the -n option, the list of configuration files will not be sorted before processing 🙈. it is sorted by default, but heads up — that sorting can get expensive if you've got a huge number of configuration files (URLs and/or scripts). so if you're running a big operation and you don't care about order, -n is your girl.

the -d option lets you override the default performance metrics domain number, which defaults to 164. and listen. it is strongly recommended that you do not change this 🚨. like, the domain number should be different for every PMDA on one host, and the same domain number should be used for pmdaopentelemetry PMDA on all hosts. consistency is the vibe. see also the -r option, which lets you change the root of the dynamic namespace from its default of opentelemetry.

the -l option lets you specify logfile as the destination for PMDA messages, instead of the default $PCP_LOG_DIR/pmcd/opentelemetry.log. special case alert 📢 — logfile can be "-" to send messages to the stderr stream instead, like -l-. that would normally be the stderr stream for the parent process, pmcd(1), which might itself have redirected stderr. this redirection is normally most useful in a containerized environment 🐳, or when you're using dbpmda(1).

the -r option lets you change the root of the dynamic namespace to root, from the default opentelemetry. paired with other command line options, this lets pmdaopentelemetry be deployed as a different PMDA — with its own distinct metrics namespace and metrics domain — on the very same host system 👯. but note: all PMDAs require a unique domain number, so the -d option must also be specified. also, using -r may change the defaults for some other command line options, like the default log file name and the default configuration directory. she's a shapeshifter like that.

the -R option lets you configure the timeout, in seconds, between cluster refreshes. the default value is 10 seconds 🌈.

🎀 CONFIGURATION SOURCES 🎀

while she's running, pmdaopentelemetry periodically recursively scans the $PCP_PMDAS_DIR/opentelemetry/config.d directory (or whatever directory you specified with the -c option), looking for source URL files (*.url) and executable scripts or binaries 🔍.

here's the thing though — any files that don't have the .url suffix or aren't executable get ignored. and that's actually a feature bestie 💅 because it means you can keep documentation files like "README" and non-executable "common" script function definitions sitting right there without them being treated as config files. they're just vibing.

a remote server does not have to be up, and doesn't have to stay running. the PMDA tolerates remote URLs that may come and go over time 🦋. she'll relay data and metadata when/if they're available, and will return errors when/if they're down. PCP metric IDs, internal and external instance domain identifiers are persisted and will be restored when individual metric sources become available and/or when the PMDA is restarted. plus she checks directory modification times and will rescan for new or changed configuration files dynamically ✨. so no — it is not necessary to restart the PMDA when adding, removing or changing configuration files. she already knows. she always knows. 👑

🔥 URL SOURCES 🔥

every file with the .url suffix found in the config directory or sub-directory contains one complete HTTP or HTTPS URL at which pmdaopentelemetry can reach an OpenTelemetry endpoint 🌐.

local file access is also supported, with a conventional file:///somepath/somefile URL — in which case somepath/somefile should contain OpenTelemetry formatted metric data 📁.

the first line of a .url config file should be the URL, as described above. subsequent lines, if any, are prefixed with a keyword that can be used to alter the http GET request. a keyword must end with : (colon) and the text extends to the end of the line. comment lines that start with # and blank lines are ignored 🙈. the only currently supported keywords are HEADER: and FILTER:.

HEADER: headername: value ... to end of line

adds headername and its value to the headers passed in the http GET request for the configured URL. here's an example configuration file that provides 3 commonly used headers and an authentication token:

http://somehost/path/endpoint.html
# this is a comment
HEADER: Accept: text/html
HEADER: Keep-Alive: 300
HEADER: Connection: keep-alive
HEADER: Authorization: token ABCDEF1234567890

as mentioned above, header values extend to the end of the line. they may contain any valid characters, including colons 🫶. multiple spaces will be collapsed to a single space, and leading and trailing spaces are trimmed. a common use for headers is to configure a proxy agent and the assorted parameters it may require.

💜 METRIC FILTERING 💜

metric filtering is a configuration file feature that lets ingested metrics be included or excluded, i.e. filtered 🧚. this is useful because most end-points return multiple metrics, and usually only some of them are interesting for monitoring purposes. she doesn't need the whole closet, just the fits she'll actually wear.

the syntax is:

FILTER: INCLUDE METRIC regex

or

FILTER: EXCLUDE METRIC regex

dynamically created metric names that match regex will be either included or excluded in the name space, as specified. note that only the PMNS leaf component of the metric name (as ingested from the URL source) is compared with the regex pattern 👀.

the simple rule is: the first matching filter regex for a particular metric leaf name is the rule that prevails. if no filter regex matches (or there are no filters), then the metric is included by default — i.e. the default filter if none are specified is:

FILTER: INCLUDE METRIC .*

this is backward compatible with older versions of the configuration file that did not support filters 🫶. multiple FILTER: lines would normally be used, e.g. to include some metrics but exclude all others, use FILTER: EXCLUDE METRIC .* as the last of several filters that include the desired metrics. conversely, to exclude some metrics but include all others, use FILTER: EXCLUDE METRIC regex. in that case it's not necessary (though it doesn't hurt) to specify the final FILTER: INCLUDE METRIC .* because, as stated above, any metric that does not match any filter regex will be included by default. slay 💅

🦋 LABEL FILTERING 🦋

label filtering uses similar FILTER: syntax and semantics as metric filtering.

FILTER: EXCLUDE LABEL regex will delete all labels with label name matching regex from all metrics defined by the configuration file ✂️.

the same rules as for metric filters apply for label filters too — an implicit rule FILTER: INCLUDE LABEL .* applies to all labels that do not match any earlier label filter rule.

FILTER: OPTIONAL LABEL regex specifies that matching label names are to be included in the returned metric labelsets (i.e. included), but are not to be used as part of the external instance names. all included labels that are not optional (i.e. the intrinsic labels) will be concatenated together and used for external instance naming 🔗. in addition, non-intrinsic labels (i.e. labels tagged as OPTIONAL) will have the PM_LABEL_OPTIONAL flag set in the labelsets returned by notes callbacks. this flag affects how the labels are used in certain clients. for further details, see pmLookupLabels(3) and related man pages for further details. note that external instance names begin with the unique numeric internal instance identifier followed by a space, so external instance names are always unique ✨.

⚠️ caution is needed with label filtering because by default, all labels are used to construct the PCP instance name. by excluding some labels (or changing them to optional), the instance names will change. by excluding some labels, different instances returned by the URL or scripted configuration entry for the same metric may become duplicates 💀. when such duplicates occur, the last duplicate instance returned by the end-point URL or script prevails over any earlier instances. for these reasons, it is recommended that label filtering rules be configured when the configuration file is first defined, and not changed thereafter. if a label filtering change is required, the configuration file should be renamed — which effectively defines a new metric (or set of peer metrics as returned by the URL or script), with the new (or changed) instance naming.

unrecognized keywords in configuration files are reported in the PMDA log file but otherwise ignored 🤷♀️.

🌈 SCRIPTED SOURCES 🌈

executable scripts present in the $PCP_PMDAS_DIR/opentelemetry/config.d directory or sub-directories will be executed, and the stdout stream containing OpenTelemetry formatted metric data will be parsed as though it had come from a URL or file 🎬. the stderr stream from a script will be sent to the PMDA log file, which by default can be found in $(PCP_LOG_DIR)/pmcd/opentelemetry.log.

📌 note that scripted sources do not support label or metric filtering (as described above for URL sources) — they can simply do their own filtering in the script itself with sed(1), awk(1), or whatever tool is desired 💅.

for full details of the OpenTelemetry exposition formats see:

https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto

👑 SELINUX CONSIDERATIONS 👑

scripted config files are executed by the pmdaopentelemetry PMDA with the same SELinux context and policy as the local pmcd(1) 🔐.

for simple scripts, this is normally fine. however — AVC errors may result for scripts that make library or system calls that are restricted by the prevailing SELinux context and policies 😭. in these cases it is not feasible to unilaterally grant pmcd or its PMDAs an unconfined execution policy. in these site specific cases it will be necessary to create a local SELinux policy module. this can be done by capturing the AVC record(s) from the local audit log, generating a local policy module using audit2allow, and then loading the new module using semodule, e.g. as follows:

$ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
| audit2allow -M mypolicy
$ sudo semodule -i mypolicy.pp

if these local policies need to be persistent across reboots, then a scriptlet similar to the above example may be added to the local pmcd RC file (typically /etc/pcp/pmcd/rc.local) 🔁. for further details, see audit2allow(1) and semodule(1).

🌸 METRIC NAMING 🌸

okay so here's the tea on how names get made, bestie 💖

EVERY metric that comes out of a file named JOB.* gets exported as a PCP metric with the prefix opentelemetry.JOB. so like, the JOB name has to be a valid non-leaf name for PCP PMNS metric names, no exceptions, that's the law ✨

and if your JOB name has multiple dot-separated components? then those components show up in the resulting PMNS names too, and you gotta be careful — like, genuinely careful, not fake careful — to make sure you don't have overlapping definitions. for example, metrics returned by JOB.response could totally overlap or conflict with metrics returned by JOB.response.time. it's giving chaos if you don't watch it 😭

config file entries (URLs or scripts) that live in subdirectories of the config directory also produce hierarchical metric names, and honestly? iconic behaviour 👑

for example, a config file named $PCP_PMDAS_DIR/opentelemetry/config.d/mysource/latency/get.url will result in metrics being created (by fetching that source URL) below opentelemetry.mysource.latency.get in the PCP namespace. scripts found in subdirectories of the config directory do the same thing — hierarchical PCP metric names, periodt 💅

🦋 DYNAMIC METRIC NAMES 🦋

so remember how we said you can change and add stuff to files in the configuration directory without restarting the PMDA? yeah. that's still true, and it's lowkey the best part 🌈

these changes get detected automatically and the PCP metric names below opentelemetry in the PMNS get updated accordingly — meaning new metrics get dynamically added and/or existing metrics get removed. no cap, it just happens ✨

and on top of that, pmdaopentelemetry honors the PMCD_NAMES_CHANGE pmFetch(3) protocol that was introduced in PCP version 4.0. specifically, if opentelemetry metrics are being logged by a PCP version 4.0 or later pmlogger(1), new metrics that show up because of changes in the PMDA configuration directory will automatically start to be logged — provided the root of the opentelemetry PMDA namespace is configured for logging in the pmlogger configuration file. see pmlogger(1) for the details, bestie 🫶

an example of such a pmlogger configuration file is:

log mandatory on 2 second {
     # log all metrics below the root of the opentelemetry namespace
     opentelemetry
}

🔥 METADATA 🔥

metric data returned by URL or scripted configuration files may contain metadata, and the opentelemetry PMDA can use that metadata to specify the semantics, data type, scaling and units of dynamically created metrics. slay 💅

this metadata is found for each metric under the metric list header in the opentelemetry json data structure. want more info about PCP metadata? check out pmLookupDesc(3) and pmParseUnitsStr(3), plus the examples in the shipped configuration files 📚

here's the thing though — the metadata must be supplied in the json data by the opentelemetry source end-point (URL or script). an alternative is to specify it in the URL configuration file directly, which has the advantage of not having to modify the source/end-point if the metadata is incorrect or missing. and any metadata specified in the URL configuration file over-rides any in-line metadata. that's the hierarchy, periodt 👑

the configuration file syntax for specifying metadata is:

METADATA: regex type indom semantics units ... to EOL

where:

  • METADATA: is literal — like, literally just those characters
  • regex is an extended regular expression to match one or more metric names returned by the URL
  • type is one of the PCP numeric types (32, u32, 64, u64, float or double)
  • indom is an arbitrary instance domain name, or PM_INDOM_NULL
  • semantics is either counter, instant or discrete
  • units is either none or a string extending to end of line that is parsable by pmParseUnitsStr(3) — i.e. the units, dimensions and scaling to be used for matching metrics

okay so here's an example configuration file that ingests metrics from the Grafana /metrics end-point on localhost, filters out all metrics returned by that URL except for grafana_api_response_status_total, and then specifies the metric type is an unsigned 32 bit integer with a non-singular instance domain named response and counter semantics with units of count:

http://localhost:3000/metrics
FILTER: INCLUDE METRIC grafana_api_response_status_total
FILTER: EXCLUDE METRIC .*
METADATA: grafana_api_response_status_total u32 response counter count
⚠️ note that the name in the indom field is presently ignored unless it is PM_INDOM_NULL, in which case the metric has no instance domain (i.e. singular), even if it has labels which would otherwise be used for instance naming. so like, don't get attached to your indom name, she's being ignored rn 💀

👑 CONTROL METRICS 👑

the PMDA maintains special control metrics, described below. apart from opentelemetry.control.debug, each of these metrics has one instance for each configured metric source. all of these metrics have integer values with counter semantics, except opentelemetry.control.status, which has a string value 🌸

🚨 it is important to note that fetching any of the opentelemetry.control metrics will only update the counters and status values if the corresponding URL is actually fetched. if the source URL is not fetched, the control metric values do not trigger a refresh, and the control values reported represent the most recent fetch of each corresponding source. iykyk 😭

the instance domain for the opentelemetry.control metrics is adjusted dynamically as new sources are discovered. and if there are no sources configured, the metric names are still defined — but the instance domain will be empty and a fetch will return no values. it's giving ghost town 👻

  • opentelemetry.control.status — a string representing the status of the last fetch of the corresponding source. this will generally be success for an http response code of 200, or an exit code of 0 for scripted sources. this metric can be used for service availability monitoring — provided, as stated above, the corresponding source URL is fetched too 🫶
  • opentelemetry.control.status_code — this metric is similar to opentelemetry.control.status except that it's the integer response code of the last fetch. a value of 200 usually signifies success and any other value failure. for scripted sources, a zero exit code signifies success and any non zero code failure. this metric can also be used for service availability monitoring, with the same caveats as opentelemetry.control.status ✨
  • opentelemetry.control.calls — total number of times each configured metric source has been fetched (if it's a URL) or executed (if it's a script), since the PMDA started. this metric has counter semantics and would normally be converted to a rate/second by client tools 📈
  • opentelemetry.control.fetch_time — total time in milliseconds that each configured metric source has taken to return a document, excluding the time to parse the document. this metric has counter semantics and would normally be rate converted by client tools, but it's also useful in raw form as the accumulated parse time since the PMDA was started ⏱️
  • opentelemetry.control.parse_time — total time in milliseconds that each configured metric source has taken to parse each document, excluding the time to fetch the document. this metric has counter semantics and would normally be rate converted by client tools, but it's also useful in raw form as the accumulated parse time since the PMDA was started 🧮

when converted to a rate, the calls metric represents the average fetch rate of each source over the sampling interval (the time delta between samples). the fetch_time and parse_time counters, when converted to a rate, represent the average fetch and parsing latency (respectfully), during the sampling interval 🔥

the opentelemetry.control.debug metric has a singular value, defaulting to 0. if a non-zero value is stored into this metric using pmstore(1), additional debug messages will be written to the PMDA log file 🐛

💀 LIMITATIONS 💀

pmdaopentelemetry and libpcp internals impose some numerical constraints, bestie, and you cannot escape them:

  • sources: 4095
  • metrics within each source: 1024
  • instances for each metric: 4194304

that's the ceiling. plan accordingly 🙈

🎀 INSTALLATION 🎀

install the OpenTelemetry PMDA by using the Install script as root:

# cd $PCP_PMDAS_DIR/opentelemetry
# ./Install

to uninstall, the following must be done as root:

# cd $PCP_PMDAS_DIR/opentelemetry
# ./Remove

pmdaopentelemetry is launched by pmcd(1) and should never be executed directly — like, don't do it, she's not built for that 🚫

the Install and Remove scripts notify pmcd when the agent is installed or removed ✨

and when scripts and .url files are added, removed or changed in the configuration directory, it's usually not necessary to restart the PMDA — the changes will be detected and managed on subsequent requests to the PMDA. living her best dynamic life 💅

🌈 FILES 🌈

  • $PCP_PMDAS_DIR/opentelemetry/Install — installation script for the pmdaopentelemetry agent
  • $PCP_PMDAS_DIR/opentelemetry/Remove — undo installation script for the pmdaopentelemetry agent
  • $PCP_PMDAS_DIR/opentelemetry/config.d/ — contains URLs and scripts used by the pmdaopentelemetry agent as sources of opentelemetry metric data
  • $PCP_LOG_DIR/pmcd/opentelemetry.log — default log file for error messages from pmdaopentelemetry
  • $PCP_VAR_DIR/config/164.* — files containing internal tables for metric and instance ID number persistence (domain 164)

⭐ PCP ENVIRONMENT ⭐

environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. on each installation, the file /etc/pcp.conf contains the local values for these variables. the $PCP_CONF variable may be used to specify an alternative configuration file, as described in

✨ DEBUGGING OPTIONS ✨

ok so when stuff is acting up and you need the tea on what's actually happening, the -D or --debug option is your girl 💅. She turns on extra diagnostics that get written to the PMDA log file, which by default lives at $PCP_LOG_DIR/pmcd/opentelemetry.log (yep, same one we talked about back with -l 🫶).

ngl bestie, this output is sometimes cryptic 😭 — it's lowkey aimed at developers who are triaging problems, not really at end-users just vibing. debug is a comma separated list of debugging options, and if you wanna know what options exist and what they even mean, run pmdbg(1) with the -l option. iconic.

The debugging options that are specifically pmdaopentelemetry's own thing are:

  • appl0 — detailed debugging that's particularly useful for examining the http headers passed to each fetch request, plus filter settings and other processing details. it's giving magnifying glass 🔍

and ooh, also — the metric opentelemetry.control.debug controls the appl0 debug setting for a running pmdaopentelemetry, so you don't even have to restart anything. change it like this:

$ pmstore opentelemetry.control.debug value

where value is either 1 (to enable appl0 diagnostics) or 0 (to disable appl0 diagnostics). periodt ✨

🌈 SEE ALSO 🌈

go read these when you're done here, they're all part of the family fr fr:

and that's a wrap on the whole page bestie, you ate that 🎀 this was Performance Co-Pilot, version PCP, for PMDAOPENTELEMETRY(1) — go forth and monitor your telemetry like the main character you are 👑💖

📦 where this page comes from