PMFETCHGROUP(3)

📦
version 6.3.8 · 3 versions
, in centos-stream 9, debian trixie, ubuntu questing

💅 PMFETCHGROUP(3) — bestie, we're fetching metrics and we're doing it cutely ✨

okay so this is the manual page for PMFETCHGROUP(3), and no cap, it's a whole vibe. we're talking the fetchgroup crew: pmCreateFetchGroup, pmExtendFetchGroup_item, pmExtendFetchGroup_indom, pmExtendFetchGroup_event, pmExtendFetchGroup_timestamp, pmExtendFetchGroup_timespec, pmExtendFetchGroup_timeval, pmFetchGroup, pmGetFetchGroupContext, pmClearFetchGroup, pmDestroyFetchGroup — aka simplified performance metrics value fetch and conversion. slay.

🌸 NAME 🌸

pmCreateFetchGroup, pmExtendFetchGroup_item, pmExtendFetchGroup_indom, pmExtendFetchGroup_event, pmExtendFetchGroup_timestamp, pmExtendFetchGroup_timespec, pmExtendFetchGroup_timeval, pmFetchGroup, pmGetFetchGroupContext, pmClearFetchGroup, pmDestroyFetchGroup — simplified performance metrics value fetch and conversion 🦋

✨ C SYNOPSIS ✨

#include <pcp/pmapi.h>

int pmCreateFetchGroup(pmFG *ptr, int type, const char *name);
int pmExtendFetchGroup_item(pmFG pmfg, const char *metric, const char
        *instance, const char *scale, pmAtomValue *out_value, int
        out_type, int out_sts);
int pmExtendFetchGroup_indom(pmFG pmfg, const char *metric, const char
        *scale, int out_inst_codes[], char *out_inst_names[],
        pmAtomValue out_values[], int out_type, int out_stss[], unsigned
        int out_maxnum, unsigned int *out_num, int *out_sts);
int pmExtendFetchGroup_event(pmFG pmfg, const char *metric, const char
        *instance, const char *field, const char *scale, struct timespec
        out_times[], pmAtomValue out_values[], int out_type, int
        out_stss[], unsigned int out_maxnum, unsigned int *out_num, int
        *out_sts);
int pmExtendFetchGroup_timestamp(pmFG pmfg, struct timeval *out_value);
int pmExtendFetchGroup_timespec(pmFG pmfg, struct timespec *out_value);
int pmExtendFetchGroup_timeval(pmFG pmfg, struct timeval *out_value);
int pmGetFetchGroupContext(pmFG pmfg);
int pmFetchGroup(pmFG pmfg);
int pmClearFetchGroup(pmFG pmfg);
int pmDestroyFetchGroup(pmFG pmfg);

cc ... -lpcp

🫶 DESCRIPTION 🫶

so the fetchgroup functions are giving registration-based mechanism energy — like, you register what you want up front, and then everything just happens. they fetch groups of performance metrics, and they automate the general unit, rate and type conversions for you, plus they hand you convenient instance and value encodings. honestly iconic. 💅

they're a powerful and compact alternative to the classic Performance Metrics Application Programming Interface (PMAPI(3)) situation where you have to do a whole separate sequence of lookup, check, fetch, iterate, extract and convert functions. like, one by one. exhausting. not anymore bestie.

the general idea is two stages, and this is important so stay with me:

  1. the setup stage — your application names the metrics it cares about, says what conversions it wants, and registers a unique pmAtomValue output location where the fetchgroup system is gonna drop the result later. you can also identify a metric that has an instance domain and register a unique vector of pmAtomValue objects for all of them.
  2. the operation stage — one single simple pmFetchGroup call fetches, decodes, converts, and stores all the metrics into their destinations, and your application just reads them directly. main character behavior honestly.

and the way that you can call pmFetchGroup repeatedly and each time new pmAtomValue values land in the same destinations?? yeah. also you can request rate conversions between consecutive samples. obsessed.

every fetchgroup gets its own private PMAPI context, so it can mess with instance profiles and other such state without disrupting any other contexts. the instance profile gets manipulated to optimize fetches of individual items, even if some of those items are derived metrics. this private PMAPI context belongs to the fetchgroup, it's used for all of its internal operations, and — heads up — it will be destroyed. 💀

you can run multiple fetchgroups concurrently and independently, no drama. an opaque type pmFG identifies a fetchgroup, and you pass that to all the related function calls. iykyk.

🎀 Creating a fetchgroup 🎀

int pmCreateFetchGroup(pmFG *ptr, int type, const char *name);

this one creates a brand new fetchgroup, associated with a brand new PMAPI context. the type and name parameters get relayed straight to pmNewContext(3) for the context creation. the fetchgroup identifier comes back to you on success through the ptr pointer. you'll use this object later as a parameter to literally every other fetchgroup function. and if you need to peek at the private PMAPI context, pmGetFetchGroupContext has your back.

the normal return code is zero, and ptr is set. this function can fail if pmNewContext or memory allocation goes wrong — that shows up as a negative return code and a cleared ptr value. 😭

💜 Getting the private PMAPI context 💜

int pmGetFetchGroupContext(pmFG pmfg);

this returns the private PMAPI context that the given fetchgroup is using. and you can safely use it to adjust some configuration parameters of that context — like via pmSetMode(3) — as long as you do it before fetchgroup extension and fetching begin. timing is everything bestie.

⚠️ but if you mutate this context with PMAPI functions after that point, you may disrupt fetchgroup functionality. for example, a pmSetMode call could invalidate one rate-conversion time-step. not cute. 🙈

the normal return code is the context number.

🌸 Extending a fetchgroup with a metric instance of interest 🌸

int pmExtendFetchGroup_item(pmFG pmfg, const char *metric, const char
        *instance, const char *scale, pmAtomValue *out_value, int
        out_type, int *out_sts);

this registers your interest in a single metric and an optional instance. the metric name goes in the mandatory metric parameter, which gets checked immediately via pmLookupName(3) and other calls.

if and only if the metric has an instance domain, you can name the specific instance you want in the instance parameter, which gets checked immediately via pmNameInDom(3). otherwise pass NULL.

🦋 little edge case: if the fetchgroup context is a set of archives, it's possible that the metric / instance pair isn't defined yet at the current time origin. so this function may temporarily attempt to seek to the end of the current set of archives to retry that metric / instance lookup. it's doing its best, okay?

the optional scale parameter says what unit/scale/rate conversions you want for the metric value. it can take these values:

  • NULL — no unit/scale conversion. but if the metric has PM_SEM_COUNTER semantics, perform rate conversion anyway. sneaky but fair.
  • rate — perform rate conversion regardless of semantics, and no unit/scale conversion.
  • instant — perform no rate conversion regardless of semantics, and no unit/scale conversion.
  • EXPRESSION — perform unit/scale/rate conversion as specified by the EXPRESSION, which is parsed by pmParseUnitsStr(3). this can be useful to assert a canonical scaling for the resulting metric value, independent of PCP version or configuration. dimensionality must match the metric, except if rate conversion is requested — in that case the time dimension must be one smaller than the metric's time dimension. and note: the type of rate conversion performed here matches the rate(x) function in derived metric expressions, in that it's calculated as the naive difference between previous and current values of a metric, divided by elapsed time. so for example, if a counter wraps around, or a non-counter value decreases, a negative output rate may be computed. not a bug, just math. 💀

the optional but usual out_value parameter specifies the pmAtomValue where the converted result should be stored later. if that value is NULL, fetching and conversions still get attempted and possible errors still get reported, but the result gets tossed away. gone. bye.

the mandatory out_type parameter specifies the PM_TYPE_* you want for the output value. it does not need to match the metric's native type, because the fetchgroup facility can cast between all supported types — including to and from strings. versatility queen. 👑

any errors that show up later during fetching, unit/scale/rate conversion, or casting will result in a sentinel value being assigned to the output pmAtomValue — see the ``UNUSUAL SITUATIONS'' section below for the tea. and if the optional out_sts parameter is specified, an appropriate PMAPI error code gets stored there.

quick review, bestie: only pmfg, metric, and out_type are mandatory. everything else can be NULL to signal that your application simply does not care. disinterest is valid. 💅

the normal return code is zero. this function may fail on various lookup, type- and conversion-checking errors, and those come back as a negative return code.

🦋 Extending a fetchgroup with a metric instance domain of interest 🦋

int pmExtendFetchGroup_indom(pmFG pmfg, const char* metric, const char
        *scale, int out_inst_codes[], char *out_inst_names[],
        pmAtomValue out_values[], int out_type, int out_stss[], unsigned
        int out_maxnum, unsigned int *out_num, int *out_sts);

this one generalizes pmExtendFetchGroup_item by registering interest in a whole entire instance domain. so instead of a singleton, you register preallocated vectors for the output variables. instances get stored in sorted order in the elements of those vectors. everything else is the same concept, just scaled up. glow-up fr fr.

the metric name is specified by the mandatory metric parameter. note that it may refer to a metric without an instance domain — in which case the single output value will show up as one unnamed instance. cute.

the optional scale parameter specifies the unit/scale/rate conversions you want for the metric value, same as above.

the optional out_inst_codes parameter specifies a vector of integers, where the raw instance number of the fetched metrics should be stored later.

the optional out_inst_names parameter specifies a vector of strings, where the instance names of the fetched metrics should be stored later. if an instance doesn't have a corresponding name, a NULL pointer is stored instead.

⚠️ the application must not modify or free(3) the strings in that vector. hands off bestie, they're not yours. 🙅‍♀️

the optional out_values parameter specifies a vector of pmAtomValue objects where the converted results should be stored later. the mandatory out_type parameter specifies the PM_TYPE_* requested for all the output values, same as above.

the optional out_stss parameter specifies a vector of integers where per-instance error codes should be stored.

the mandatory out_maxnum parameter specifies the number of elements in the vectors above. in other words, it tells the fetchgroup the maximum number of instances it should expect. the optional out_num parameter specifies an integer where the actual number of instances should be stored later — it'll range between 0 and out_maxnum, and this function initializes it to 0.

and finally, the optional out_sts parameter specifies a single location where an integer status code for the overall fetch for this metric should be stored. normally that's zero. other than a severe fetch error, you might see a PM_ERR_TOOBIG in there if the number of instances actually encountered was larger than out_maxnum. 😭 the audacity of reality.

Any errors that pop up later during fetching, unit/scale/rate conversion, or casting? They’re gonna drop a sentinel value into the right output pmAtomValue (peep the UNUSUAL SITUATIONS section below for the tea ☕). And if you passed that optional out_stss param, a PMAPI error code gets tucked into the matching spot.

Just a lil recap bestie: only pmfg, metric, out_type, and out_maxnum are mandatory. Everything else can be NULL if your app doesn’t care.

Normal return code is zero. But this function can fail with lookup, type-checking, or conversion-checking errors — those come back as a negative return code.

🎀 Extending a fetchgroup with an event field 🎀

int pmExtendFetchGroup_event(pmFG pmfg, const char *metric, const char
        *instance, const char *field, const char *scale, struct timespec
        out_times[], pmAtomValue out_values[], int out_type, int
        out_stss[], unsigned int out_maxnum, unsigned int *out_num, int
        *out_sts);

Okay so this one? It gets you all up in the instances of ONE field across ALL records of an event metric. Event metrics can return multiple records per fetch, and each record can have multiple fields of a given field metric type, so this function registers preallocated vectors for output variables — just like pmExtendFetchGroup_indom does. They get filled in temporal/sequential order, no cap.

The metric name is mandatory metric. It has to be PM_TYPE_EVENT. If the metric has an instance domain, then instance is mandatory to pick which instance you want.

The field to extract from event records is mandatory, and it’s a metric name of normal scalar type. As is typical for event field metrics, it shouldn’t have an instance domain. The optional scale specifies desired unit/scale conversions on this metric value. Rate conversions? Not available, because it’s ambiguous which previous value to compute rates from. Soz.

The optional out_times is a vector of timespec structs. It gets a copy of the timestamp of the event record where each particular field was found.

The optional out_values is a vector of pmAtomValue objects where the converted result gets stored later. The mandatory out_type specifies the PM_TYPE_* you want for the output values.

The optional out_stss is a vector of integers for per-field error codes.

The mandatory out_maxnum tells the fetchgroup how many elements are in those vectors — i.e. the max number of instances you expect. The optional out_num is an integer where the actual number of instances gets stored later. It’ll be between zero and out_maxnum. This function initializes it to zero.

Finally, the optional out_sts is a single location for an integer status code for the overall fetch for this metric. Normally it’s zero, even if no event field values were found (out_num would be zero then). Other than a severe fetch error, you might see a PM_ERR_TOOBIG here if the number of fields actually encountered was larger than out_maxnum.

Any errors later during fetching, unit/scale conversion, or casting will assign a sentinel value to the appropriate output pmAtomValue (see UNUSUAL SITUATIONS below). And if you passed the optional out_stss, a PMAPI error code gets stored in the appropriate position.

Recap: only pmfg, metric, field, out_type, and out_maxnum are mandatory. Others can be NULL if your app doesn’t care.

Normal return code is zero. This function can fail with various lookup, type- and conversion-checking errors. Those come back as a negative return code.

🕰️ Extending a fetchgroup with the fetch timestamp 🕰️

int pmExtendFetchGroup_timestamp(pmFG pmfg, struct timeval *out_value);
int pmExtendFetchGroup_timespec(pmFG pmfg, struct timespec *out_value);
int pmExtendFetchGroup_timeval(pmFG pmfg, struct timeval *out_value);

These functions register interest in the pmHighResResult timestamp. If out_value is non-NULL, then at every future pmFetchGroup call, the corresponding result timestamp gets copied there. Cute.

🔥 Fetching all metrics in a fetchgroup 🔥

int pmFetchGroup(pmFG pmfg);

This function does one pmFetch on its private PMAPI context, including all the metrics you registered via prior pmExtendFetchGroup_* calls. It runs all the data extraction and conversion operations needed to populate all the requested output variables.

Normal return code is zero or positive, just like the underlying pmFetch. This function can fail with severe fetch errors, which come back as a negative return code.

If there are per-metric availability or conversion errors, or severe fetch errors, output variables get reset to sentinel values and individual error codes get set. PM_ERR_AGAIN signals rate-conversion failure due to lack of a previous value.

However, temporarily absent metrics with discrete semantics are exempt from some sentinel/error processing: if a pmFetchGroup fails to collect a result for a discrete metric (pmHighResResult pmValueSet.numval==0), then the last seen valid value (if any) is retained. This is meant to make processing sets of archives with a mix of once- and repeatedly-sampled metrics easier.

🧼 Clearing a fetchgroup 🧼

int pmClearFetchGroup(pmFG pmfg);

When you don’t need the current fetch state of a fetchgroup anymore, you can reset it explicitly with this function. It releases any dynamically stored state but keeps the private PMAPI context intact for later use (i.e. no change to the context at all, and the context stays at the current fetch offset). It frees any pointers such as indom instance names or strings that might have been stored in output variables.

💀 Destroying a fetchgroup 💀

int pmDestroyFetchGroup(pmFG pmfg);

When the fetchgroup is no longer needed, you can free it explicitly with this function. It releases any dynamically stored state, plus the private PMAPI context. It clears frees any pointers such as indom instance names or strings that may have been stored in output variables.

🌈 EXAMPLE 🌈

The following program demonstrates fetchgroup usage. Run it with different $PCP_DISK_UNITS environment variables to see different unit/rate conversion in effect.

#include <pcp/pmapi.h>
#include <stdio.h>

#define pcpassert(sts) \
    while (sts<0) { fprintf(stderr, "%s\n", pmErrStr(sts)); exit(42); }

int main()
{
    pmFG fg;
    pmAtomValue v, v2;
    enum { v3_maxnum = 100 };
    pmAtomValue v3_values[v3_maxnum];
    char *v3_names[v3_maxnum];
    int v3_stss[v3_maxnum];
    unsigned int v3_num;
    int sts, i;
    char *diskunits = getenv("PCP_DISK_UNITS");
    struct timeval t;

    sts = pmCreateFetchGroup(&fg, PM_CONTEXT_HOST, "local:");
    pcpassert(sts);
    sts = pmExtendFetchGroup_item(fg, "kernel.all.load", "1 minute",
                                  NULL, &v, PM_TYPE_FLOAT, NULL);
    pcpassert(sts);
    sts = pmExtendFetchGroup_item(fg, "kernel.all.idletime", NULL,
                                  "hour", &v2, PM_TYPE_DOUBLE, NULL);
    pcpassert(sts);
    sts = pmExtendFetchGroup_indom(fg, "disk.dev.total", diskunits,
                                   NULL, v3_names,
                                   v3_values, PM_TYPE_STRING,
                                   v3_stss, v3_maxnum, &v3_num, NULL);
    pcpassert(sts);
    sts = pmExtendFetchGroup_timestamp(fg, &t);
    pcpassert(sts);

    for (i=0; i < 10; i++) {
        unsigned int j;
        char stamp[28];

        sts = pmFetchGroup(fg);
        pcpassert(sts);
        printf("%s", pmCtime(&t.tv_sec, stamp));
        printf("1-minute load: %f; idletime: %f h\n", v.f, v2.d);
        for (j=0; j < v3_num; j++) {
            if (v3_stss[j] == 0)
                 printf("disk %s i/o operations (%s): %s\n",
                        v3_names[j] ? v3_names[j] : "?",
                        diskunits ? diskunits : "-",
                        v3_values[j].cp);
        }
        sleep(1);
    }

    sts = pmDestroyFetchGroup(fg);
    pcpassert(sts);
    return 0;
}

😱 UNUSUAL SITUATIONS 😱

The fetchgroup API supports only the numeric, string and event metric types. Aggregates are rejected during pmExtendFetchGroup_*.

Any strings supplied by the fetchgroup API to the application are "owned" by the API. The application should consider them read-only, so it should not modify them nor free them.

Error codes are always negative integers, whether returned from fetchgroup functions as return value, or stored in out_sts type variables. Normal result codes are always zero.

Because of the unique ways in which extracted data is shared between the application and a fetchgroup, the functions in this API are not protected by the multi-threading mutexes conventional in other parts of PMAPI. Specifically, for any given pmFG, it is not safe to concurrently call two or more fetchgroup API functions, nor to traverse the registered output variables while calling one of the functions. Instead, the calling application must ensure that only one thread at a time uses these calls and the registered output variables. On the other hand, concurrency between different pmFG instances is unrestricted, because they share no global data.

Any pointers passed to a successful pmFetchGroupExtent_* call must stay valid throughout the lifetime of the fetchgroup, since future pmFetchGroup calls may write into them.

🔍 DIAGNOSTICS 🔍

okay bestie, so when stuff goes wrong (and sometimes it does, no cap), the fetchgroup API has your back with several ways to collect the receipts. 💅

First up: negative integer error codes can come back from each function when things are genuinely serious. Like, not a vibe. 🫠

Then there's the lowkey elegant option — every output pmAtomValue can have a matching integer variable hanging out next to it, where pmFetchGroup can stash the per-metric per-instance error codes for you. So you know exactly which metric on which instance was being messy. 💀

🎀 the sentinel situation 🎀

As an alternative, those per-metric per-instance error conditions also get signalled by setting the corresponding pmAtomValue to a sentinel value. If you don't need unambiguous, precise error detection, honestly this is enough — it's giving "good enough, bestie". ✨

Here's the sentinel tea, per type:

  • all integers (including unsigned integers — i.e. all bits are set) get negative one
  • floating point types get NaN
  • strings get a NULL pointer
  • and the timestamp gets 0.0s
🫶 the fetchgroup API guarantees that once an output pmAtomValue is registered (during a successful pmExtendFetchGroup_* call), it will be cleared to the sentinel value or to a valid converted metric value — from the moment of registration all the way until the pmDestroyFetchGroup call. no gaps, no ghost values, iconic behaviour fr fr 👑

🌈 SEE ALSO 🌈

go read these next, they're the whole friend group:

and that's the whole page, bestie!! 🦋 this one's from the Performance Co-Pilot collection — PCP, page PMFETCHGROUP(3) — so go forth, fetch your metrics, and remember: sentinel values are not your enemy, they're just your data being honest with you. 💖 mwah! 🎀✨

📦 where this page comes from