PMDA(3)

📦
version 7.2.2 · 5 versions
, in arch rolling, centos-stream 10, fedora 43, fedora 44, ubuntu resolute, ubuntu stonking

💖 PMDA(3) — the intro to the Performance Metrics Domain Agent support library 💖

okay bestie, welcome to PMDA(3) from the Library Functions Manual. This is the group chat for building your very own Performance Metrics Domain Agent, and honestly? it's giving main character energy. Let's get into it. ✨

🌸 NAME 🌸

PMDA — introduction to the Performance Metrics Domain Agent support library. That's the whole vibe. It's the "hi hello, here's how you build a PMDA" page. 🫶

✨ C SYNOPSIS ✨

#include <pcp/pmapi.h>
#include <pcp/pmda.h>

 ... assorted routines ...

cc ... -lpcp_pmda -lpcp

So you're pulling in both headers and linking against -lpcp_pmda and -lpcp. Note the ... assorted routines ... in the middle — yes, that's literally what it says, this page is the intro, not the whole menu. 🎀

💅 DESCRIPTION 💅

So here's the deal. To help you build Performance Metric Domain Agents (PMDAs) for the Performance Co-Pilot (PCP), there's a procedural interface. It's an extension of the Performance Metrics Application Programming Interface — that's PMAPI(3) to you and me.

These procedures are designed to let a programmer build a PMDA fast, then test it, then refine it. Iconic. But — and this is important — that also means a PMDA has a particular structure, and that structure might not suit every single PMDA implementation. No cap. 🫠

Once you're comfy with the PCP and PMDA frameworks, you can spin up a new PMDA with just a few data structures and functions. If you want the deep lore, it's covered in far greater detail in the Performance Co-Pilot Programmer's Guide.

A PMDA is responsible for a set of performance metrics. Meaning: it must respond to requests from pmcd(1) for info about performance metrics, instance domains, and instantiated values. That's the job description, bestie. 👑

This man page contains sections of the simple PMDA, which lives at $PCP_PMDAS_DIR/simple. Cute little reference implementation, we stan. 💖

🌈 COMMUNICATING WITH PMCD 🌈

There are two approaches for connecting a PMDA to a pmcd(1) process:

  • A Dynamic Shared Object (DSO) can be attached by pmcd(1) using dlopen(3) when the pmcd(1) process is started. A procedural interface, referenced through a shared data structure, handles requests from pmcd(1) to the PMDA.
  • The preferred approach ✨ is a separate process (a daemon) that talks to pmcd(1) using the Performance Data Units (PDU) Inter-Process Communication (IPC) protocol.

All PMDAs are launched and controlled by the pmcd(1) process on the local host. Clients send requests, pmcd(1) receives them, and forwards them to the appropriate PMDAs. Responses, when they're needed, come back through pmcd(1) to the clients. Chain of command, fr fr. 📬

The requests (PDUs) that may be sent to a PMDA from pmcd(1) are:

  • PDU_FETCH
  • PDU_PROFILE
  • PDU_INSTANCE_REQ
  • PDU_DESC_REQ
  • PDU_TEXT_REQ
  • PDU_RESULT

And if the PMDA implements any dynamic metrics, it may also receive:

  • PDU_PMNS_CHILD
  • PDU_PMNS_IDS
  • PDU_PMNS_NAMES
  • PDU_PMNS_TRAVERSE

🦋 DEFAULT CALLBACKS FOR HANDLING PDUs 🦋

To keep a consistent framework, a daemon PMDA can use pmdaMain(3) to handle the communication protocol — using the same callbacks as a DSO PMDA. Symmetry, we love to see it. ✨

The structure pmdaInterface is what conveys the common procedural interface and state information used by pmcd(1) and a PMDA. That state info includes tables describing the supported metrics and instance domains.

Since most of the procedural interface is identical for all PMDAs, it's provided as part of this support library:

But 🚨 those routines need access to the pmdaInterface state information, so it must be correctly initialized using:

Lowkey if you skip the init step nothing works, so like… don't. 💀

🎀 INSTANCES AND INSTANCE DOMAINS 🎀

Three structures are declared in /usr/include/pcp/pmda.h, and they give you a framework for declaring the metrics and instances your PMDA supports.

Every instance needs a unique integer identifier and a unique name, as defined by the structure pmdaInstid:

/*
 * Instance description: index and name
 */

typedef struct {
    int         i_inst;        /* internal instance identifier */
    char        *i_name;       /* external instance identifier */
} pmdaInstid;

An instance domain needs its own unique identification (pmInDom), the number of instances the domain represents, and a pointer to an array of instance descriptions. That's the structure pmdaIndom:

/*
 * Instance domain description: unique instance id,
 * number of instances in this domain, and the list of
 * instances (not null terminated).
 */

typedef struct {
    pmInDom     it_indom;       /* indom, filled in */
    int         it_numinst;     /* number of instances */
    pmdaInstid  *it_set;        /* instance identifiers */
} pmdaIndom;

The simple PMDA has one instance domain for simple.color with three instances (red, green and blue), and a second instance domain for simple.now with instances that can be specified at run-time. Obsessed with the flexibility ngl. 💅

These instance domains are defined as:

static pmdaInstid _color[] = {
    { 0, "red" }, { 1, "green" }, { 2, "blue" }
};
static pmdaInstid *_timenow = NULL;

static pmdaIndom indomtab[] = {
#define COLOR_INDOM 0
    { COLOR_INDOM, 3, _color },
#define NOW_INDOM 1
    { NOW_INDOM, 0, NULL },
};

The preprocessor macros COLOR_INDOM and NOW_INDOM are used in the metric description table to identify the instance domains of individual metrics. They correspond to the serial value in the instance domain pmInDom structure — the domain field is set by pmdaInit(3) at run-time. That serial value must be unique for each instance domain within the PMDA. ⭐

Now here's a little freedom moment: the indom table shown above, which is usually passed to pmdaInit(3), does not need to be created if you want to write your own Fetch and Instance functions. See pmdaInit(3) for more details. You do you, bestie. 🫶

💜 NAMESPACE 💜

Every PMDA has its own unique namespace, using the format defined in PMNS(5). In summary: the namespace matches the names of the metrics to the unique identifier. Simple as that. ✨

The simple PMDA defines five metrics:

  • simple.numfetch
  • simple.color
  • simple.time.user
  • simple.time.sys
  • simple.now

The namespace for these metrics is defined in $PCP_PMDAS_DIR/simple/pmns and is installed as:

simple {
    numfetch    253:0:0
    color       253:0:1
    time
    now         253:2:4
}

simple.time {
    user        253:1:2
    sys         253:1:3
}

The domain number of 253 is obtained from $PCP_VAR_DIR/pmns/stdpmid. New PMDAs should specify a unique domain number in this file, and obtain the number during installation. That way, the domain number can change by modifying only the file $PCP_VAR_DIR/pmns/stdpmid. Iconic single source of truth behavior. 👑

The simple.time and simple.now metrics are defined in separate clusters to the other metrics, which lets a PMDA support more than 1024 metrics, and also lets you group similar metrics together. So the item numbers for a new cluster may be identical to the item numbers in other clusters. The simple PMDA just keeps incrementing the item numbers to permit direct mapping (see pmdaInit(3)).

The namespace file should be installed and removed with the agent using pmnsadd(1) and pmnsdel(1). See the later sections on INSTALLATION and REMOVAL.

A simple ASCII namespace can be constructed by creating a file similar to $PCP_PMDAS_DIR/simple/root:

/*
 * fake "root" for validating the local PMNS subtree
 */

#include "$PCP_VAR_DIR/pmns/stdpmid"

root { simple }

#include "pmns"

and it can be referred to with the -n option in most PCP tools. 🎀

🍓 METRIC DESCRIPTIONS 🍓

Each metric requires a description (pmDesc), which contains its Performance Metric Identifier (PMID), data type specification, instance domain, semantics and units (see pmLookupDesc(3)).

There's also a handle for application specific information in the pmdaMetric structure:

/*
 * Metric description: handle for extending description,
 * and the description.
 */

typedef struct {
    void*       m_user;         /* for users external use */
    pmDesc      m_desc;         /* metric description */
} pmdaMetric;

The simple PMDA defines the metrics as:

static pmdaMetric metrictab[] = {
/* numfetch */
    { (void *)0,
      { PMDA_PMID(0,0), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
        { 0,0,0,0,0,0} }, },
/* color */
    { (void *)0,
      { PMDA_PMID(0,1), PM_TYPE_32, COLOR_INDOM, PM_SEM_INSTANT,
        { 0,0,0,0,0,0} }, },
/* time.user */
    { (void*)0,
      { PMDA_PMID(1,2), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
           { 0, 1, 0, 0, PM_TIME_SEC, 0 } }, },
/* time.sys */
    { (void*)0,
      { PMDA_PMID(1,3), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
           { 0, 1, 0, 0, PM_TIME_SEC, 0 } }, },
/* now */
    { NULL,
      { PMDA_PMID(2,4), PM_TYPE_U32, NOW_INDOM, PM_SEM_INSTANT,
        { 0,0,0,0,0,0 } }, },
};

The macro PMDA_PMID (defined in /usr/include/pcp/pmda.h) is used to specify each metric's cluster and item fields of the associated pmID. Just like with instance domains, the domain field is set by pmdaInit(3) at run-time — however, the default domain is assumed to be defined by the PMDA in the macro MYDOMAIN. 🧚

And again, the metric table shown above, which is usually passed to pmdaInit(3), does not need to be created if you want to write your own Fetch and Descriptor functions. See pmdaInit(3) for more details.

🔥 DSO PMDA 🔥

A PMDA that's run as a DSO is opened by pmcd(1) with dlopen(3). Then pmcd(1) calls the PMDA's initialization function, which is specified in $PCP_PMCDCONF_PATH.

That function gets passed a pointer to a pmdaInterface structure, and you must complete it. Any callbacks that are not the default PMDA support library callbacks must be specified in the pmdaInterface structure. No skipping. 💅

The simple PMDA uses its own store and fetch callback. simple_fetch() calls pmdaFetch(3), which requires a callback to be set with pmdaSetFetchCallBack(3) — you can see this in $PCP_PMDAS_DIR/simple/simple.c.

The flag _isDSO is used to determine if the PMDA is a daemon or a DSO, so that the correct initialization routine — pmdaDaemon(3) or pmdaDSO(3) — gets called. 🌈

and bestie… there's more of this page coming, don't go anywhere 💖✨

👑 DAEMON PMDA 💅

Okay bestie so a PMDA that runs as a daemon gets forked and executed by pmcd(1). Which means — and this is the key difference from a DSO PMDA — the starting point for a daemon PMDA is main(). Like, you're the main character now. ✨

So what does the agent have to do? It should:

  • parse the command line arguments
  • create a log file
  • initialize some data structures that pmcd(1) would have initialized for you if you were a DSO agent

The pmdaInterface structure must be completely defined by the daemon PMDA, no cap. And pmdaDaemon(3) can be called at the very start of main() to set most of those fields for you, which is honestly such a slay. 🫶

Command line parsing can be simplified with pmdaGetOpt(3) — it's basically getopt(2) but it extracts a common set of options straight into the pmdaInterface structure. Chef's kiss. And stderr can be mapped to a log file using pmdaOpenLog(3), which makes debugging and error messages way less of a mess. 🎀

Then you make the connection to pmcd(1) with pmdaConnect(3), and the loop that handles all the incoming PDUs, pmdaMain(3), should be the last function you call. Last. Finale. End of the episode. 🌈

You can see all of this in action in $PCP_PMDAS_DIR/simple/simple.c — go look, it's giving tutorial energy.

The simple_init() routine is common to an agent that can be run as both a Daemon and a DSO PMDA. Shared code, shared vibes. 💖

📖 HELP TEXT 📖

Every single PMDA has to be able to hand pmcd(1) the help text for each metric. Non-negotiable, it's the law.

Most PMDAs use specially created files with indexes to support efficient retrieval of the help text. And PCP ships with tools to create those help text files in the appropriate format. See newhelp(1) for the deets. 🦋

🛠️ INSTALLATION AND REMOVAL 🛠️

There's a whole series of shell procedures defined in $PCP_SHARE_DIR/lib/pmdaproc.sh that greatly simplify the installation and removal of a PMDA. We love a helper library. 💅

The Install scripts 🍓

For most PMDAs, your Install script should only need to do this little routine:

  1. specify the name of the PMDA in iam
  2. call pmdaSetup — which does a lot of heavy lifting, bestie:
    • sets up some default variables
    • checks permissions (heads up: you have to be ``root'' to install or remove a PMDA, no exceptions 😤)
    • checks that you're in the right directory (somewhere that ends with /pmdas/$iam)
    • optionally generates the Performance Metrics Name Space (PMNS) and PMDA domain number files for Perl or Python PMDAs
    • checks the PMDA domain number is valid
    • etc. etc. the list goes on
  3. specify the communication protocols
  4. finally call pmdaInstall to do all the actual work:
    • updating the PMNS
    • updating the pmcd(1) control file
    • notifying or restarting pmcd(1)

Bonus helpers ✨

Beyond pmdaSetup and pmdaInstall, there's another optional helper routine: pmdaChooseConfigFile. It may be used to interactively select or create a PMDA-specific configuration file, e.g. pmdalogger(1). Cute, right? 🎀

The Remove scripts 🌸

These are even simpler, because setting up the communication protocols is not required. So you just:

  • set the name of the PMDA in iam
  • call pmdaSetup
  • followed by a call to pmdaRemove

That's it. Done. Bye. 👋

Further information is contained in the $PCP_SHARE_DIR/lib/pmdaproc.sh file and the following section. Go read it, iykyk. 📚

The Upgrade script ⭐

Optionally, a PMDA may provide an Upgrade script alongside Install and Remove. If it's present, this script will be used by the pmcd startup process to make sure corrections to an installation have been made before starting the PMDA.

Examples of such corrections include:

  • updates to pmcd.conf when a PMDA script or binary has been renamed
  • when the PMDA supports a new format of its configuration file
  • or if there's some latent problem from an earlier install — like, some PMDAs may need to introduce use of the notready keyword in pmcd.conf, as described below 🫠

It's giving "we fixed it in post." 🔥

✨ INSTALLATION REFINEMENTS ✨

okay so, there's a whole pile of variables you can set inside a PMDA's Install script that basically steer how the installation goes down. Most of them you'd set before you call pmdaInstall, but a couple of them — like $iam, and the special snowflakes called out below — have to be set even earlier, before pmdaSetup. 💅

💬 Communication vibes (how the PMDA talks to pmcd)

These variables control the communication options between the PMDA and pmcd(1). At minimum you gotta set one of $daemon_opt, $dso_opt, $perl_opt or $python_opt so the whole thing knows which mode(s) of communication are even supported. And if you set more than one? The user gets prompted to pick when the Install script runs. choices choices bestie 🎀

  • daemon_opt — we're willing to install the PMDA as a daemon. Default: true
  • dso_opt — we're willing to install the PMDA as a DSO, so pmcd(1) will use the dynamic linking loader to attach the PMDA's DSO at run-time, and communication from pmcd(1) to the PMDA and back uses procedure calls, not an IPC channel. Default: false
  • dso_entry — for a DSO PMDA, this is the name of the PMDA's initialization routine. Default: ${iam}_init
  • dso_name — for a DSO PMDA, this is the full pathname of the PMDA's DSO file. Default: $PCP_PMDAS_DIR/$iam/pmda_$iam.$dso_suffix
  • pipe_opt — for a daemon PMDA, is the default IPC channel via pipe(2)? Default: Platform-specific, so true for most, but false for Windows
  • perl_opt — we're willing to install the PMDA as a Perl script and pmcd(1) will use the perl(1) interpreter to run the PMDA. Default: false
  • pmda_dir — full pathname to the directory where the PMDA's installation files (executable, script, PMNS source, help text source, etc) are to be found. Default: output from pwd(1)
heads up: if you set this one, it's gotta happen before the call to pmdaSetup.
  • pmda_name — for a daemon PMDA, this is the name of the PMDA's executable binary relative to the $pmda_dir directory. Default: pmda$iam
  • python_opt — we're willing to install the PMDA as a Python script and pmcd(1) will use the python(1) interpreter to run the PMDA. Default: false
  • ipc_prot — for a daemon PMDA, this can be set to either binary or text. The default is binary and text is rarely used. On top of that, there's an optional IPC parameter notready you can use to signify that the PMDA will start up in the notready state, e.g. ipc_prot="binary notready". Note that the quotes are required. The IPC parameters for a PMDA show up in pmcd.conf in the IPC Params column. For further details, see pmcd(1), but basically pmcd will not issue any requests to a PMDA that has started in the notready state until the PMDA sends a PM_ERR_PMDAREADY PDU. This lets PMDAs with long startup times initialize correctly without timing out. For details, see pmdaSendError(3) and pmcd(1). And while a PMDA is in the notready state, any client requests sent to pmcd for the PMDA domain will come back with the PM_ERR_PMDANOTREADY error.
  • socket_inet_def — for a daemon PMDA using a socket(2) as the IPC channel, the default Internet port number or service name (if known). Default: ""
  • socket_opt — for a daemon PMDA, is the default IPC channel via a socket(2)? Default: Platform-specific, so false for most, but true for Windows

🌳 PMNS options

These variables control the PMNS options. yes we're still going. stay with me bestie 🦋

  • pmns_dupok — most PMDAs do not have duplicate names for the same PMID in their PMNS. But if that's not your situation, pmns_dupok should be set to true. Default: false
  • pmns_name — every PMDA will add one or more non-leaf nodes to the top of the PMNS. The most common case is that all of the metrics for a PMDA get placed below the node named $iam. If that's not what's happening, and especially when the PMDA adds more than one non-leaf node at the top of the PMNS, pmns_name needs to be set to the list of node names (separated by white space), e.g. for pmdaproc(1) pmns_name is set to "proc cgroup hotproc". Default: $iam
lowkey the most important thing here: if pmns_name is set to a non-default value in the Install script, it must also be set to that exact same value in the Remove script. no cap.
  • pmns_source — the name of the PMDA's PMNS source file. By default, the name is interpreted as a relative pathname from the $pmda_dir directory. Default: pmns

🎁 Assorted extra options

These variables provide assorted additional options associated with the installation of a PMDA. grab a snack, we're almost through the list 🍓

  • args — additional command line args for the PMDA. These will be appended to the PMDA's control line in $PCP_PMCDCONF_PATH. Default: ""
  • check_delay — delay (in seconds) after finishing the PMDA installation (or removal) before checking the availability of metrics from the PMDA. May need to be increased if the PMDA has a lengthy startup procedure. Default: 0.3
  • signal_delay — delay (in seconds) after notifying pmcd(1) with a signal. Required to allow pmcd(1) to complete processing before proceeding to the next step of the installation (or removal). Default: 1
  • configdir — determines the directory in which a PMDA's configuration file will be stored. Used by pmdaChooseConfigFile so it should be set before calling that procedure. Default: $PCP_VAR_DIR/config/$iam
  • configfile — preferred configuration file for the PMDA. Used by pmdaChooseConfigFile so it should be set before calling that procedure. Default: ""
  • default_configfile — default configuration file for the PMDA. Used by pmdaChooseConfigFile so it should be set before calling that procedure. Default: ""
  • dso_suffix — standard suffix for a DSO. Should not need to be changed under normal circumstances. Default: Platform-specific, so 'so' for Linux, 'dylib' for macOS, 'dll' for Windows, etc.
if you set this one, it's gotta be before the call to pmdaSetup.
  • help_source — the name of the help text source file that should be used as input to pmnewhelp(1). By default, the name is interpreted as a relative pathname from the $pmda_dir directory. Default: help
  • python_name — full pathname of the Python script for a Python PMDA. Default: $pmda_dir/pmda$iam.python or $pmda_dir/pmda$iam.py

🧹 about $tmp (temp file etiquette)

The shell procedures in $PCP_SHARE_DIR/lib/pmdaproc.sh manipulate a bunch of temporary files using the variable $tmp as the prefix for the name of the temporary files. $tmp is a directory that is created, used and removed internally within the procedures of $PCP_SHARE_DIR/lib/pmdaproc.sh, but you can also use it as the prefix for temporary files needed by a PMDA's Install or Remove scripts. When you use it that way, $tmp should be followed by a / and then a suffix, e.g. $tmp/myfoo.

and listen — the Install and Remove scripts should not use other temporary file name prefixes, and should not use sh(1) trap statements to clean up temporary files. That's all handled inside $PCP_SHARE_DIR/lib/pmdaproc.sh already. don't do extra. it's giving unnecessary 💀

😭 CAVEAT

Failing to complete any of the data structures, or calling any of the library routines out of order, may cause unexpected behavior in the PMDA. (yes that's a real warning, please respect the order bestie 🙈)

Due to changes to the PMAPI(3) and PMDA(3) API in the PCP 2.0 release — as described in the product release notes — PMDAs built using PCP 2.0 must specify PMDA_INTERFACE_2 or later and link with libpcp_pmda.so.2 and libpcp.so.2. Pre-existing Daemon PMDAs specifying PMDA_INTERFACE_1 will continue to function using the backwards compatible libpcp_pmda.so.1 and libpcp.so.1 libraries, and may be recompiled using the headers installed in /usr/include/pcp1.x/ without any modification. These backwards compatible headers and libraries are contained in the pcp.sw.compat subsystem. iconic backwards compatibility honestly 👑

🩺 DIAGNOSTICS

Any PMDA which uses this library can set the PMAPI(3) debugging control option libpmda (with -Dlibpmda on the command line, or via pmSetDebug(3)) to enable the display of debugging information which may be useful during development (see pmdbg(1)). debugging era fr fr 🔥

The status field of the pmdaInterface structure should be zero after pmdaDaemon, pmdaDSO, pmdaGetOpt, pmdaConnect and pmdaInit are called. A value less than zero indicates that initialization has failed. if it's negative, something went wrong, and no amount of positive thinking will fix it 😭

Some error messages that are common to most functions in this library are:

  • PMDA interface version interface not supported Most of the functions require that the comm.version field of the pmdaInterface structure be set to PMDA_INTERFACE_2 or later. PMDA_INTERFACE_2 or PMDA_INTERFACE_3 implies that the version.two fields are correctly initialized, while PMDA_INTERFACE_4 implies that the version.four fields are correctly initialized (see pmdaDaemon(3) and pmdaDSO(3)). set your version fields bestie, it's literally the bare minimum 💅

📁 FILES 💖

Here's the map of where everything lives, bestie — screenshot this:

  • /usr/include/pcp/pmda.h — the header file for the PMDA support library. literally where all the good stuff gets declared ✨
  • /usr/lib/libpcp_pmda.so — the dynamic library holding all the PMDA support library routines. it's giving "the actual brains of the operation" 🧠
  • $PCP_PMDAS_DIR/trivial — the source of the trivial PMDA.
  • $PCP_PMDAS_DIR/simple — the source of the simple PMDA.
  • $PCP_PMDAS_DIR/txmon — the source of the txmon PMDA.
  • $PCP_PMCDCONF_PATH — configuration file for pmcd(1). yes the main character himself 🫶
  • $PCP_VAR_DIR/pmns — where the namespace descriptions for every single PMDA hang out.
  • $PCP_VAR_DIR/pmns/stdpmid — the unique domain identifiers for each PMDA. everybody needs their own lil ID number, periodt 🔥
  • $PCP_SHARE_DIR/lib/pmdaproc.sh — shell procedures for installing and removing a PMDA. the script that does the heavy lifting so you don't have to 😌

🌈 PCP ENVIRONMENT 🌈

okay so environment variables with the prefix PCP_ are how you parameterize the file and directory names that PCP uses. cute, right?

On every installation, the file /etc/pcp.conf contains the local values for these variables. And plot twist — the $PCP_CONF variable can be used to point at a totally different configuration file instead, as described in pcp.conf(5). icon behaviour 💅

Wanna grab those values programmatically instead of eyeballing them? Use the pmGetConfig(3) function. no cap, that's the move ✨

👀 SEE ALSO 👑

And if you want the FULL story — like, the complete description of the pcp_pmda library and the whole PMDA development process — go read the Performance Co-Pilot Programmer's Guide. it's giving required reading, iykyk 📚

fr fr bestie, that's the whole page. go write a PMDA, you're gonna slay 💜

that's a wrap on PMDA(3) from the Performance Co-Pilot collection, dated PCP 🎀 thanks for reading all four parts, ily, go touch some grass and then go instrument something 🌸🦋

📦 where this page comes from