PMIE(1)

📦
version 7.2.2 · 6 versions
, in arch rolling, fedora 43, fedora 44, ubuntu stonking

💖 PMIE(1) 💖

General Commands Manual ✨ PMIE(1)

✨ NAME ✨

pmie — inference engine for performance metrics 🧠💅

🦋 SYNOPSIS 🦋

pmie [-bCdeFfPqvVWxXz?]  [-a archive] [-A align] [-c config] [-D debug]
[-h host] [-l logfile] [-m note] [-j stompfile] [-n pmnsfile] [-o
format] [-O offset] [-S starttime] [-t interval] [-T endtime] [-U
username] [-Z timezone] [filename ...]

🌸 DESCRIPTION 🌸

okay so bestie, listen up. pmie takes a whole collection of arithmetic, logical, and rule expressions and evaluates them at whatever frequencies you tell it. that's the whole gig. 💁‍♀️

the base data for all those expressions? it's performance metrics values, delivered in real-time from any host running the Performance Metrics Collection Daemon (PMCD), or from historical data pulled out of Performance Co-Pilot (PCP) archives. both vibes are valid.

and it's not just math, no no. as well as computing arithmetic and logical values, pmie can execute actions in response to specified conditions — popup alarms, write system log messages, and launch programs. iconic behavior fr fr. these actions are extremely useful for detecting, monitoring and correcting performance related problems. main character energy for your servers. 👑

the expressions to be evaluated are read from configuration files specified by one or more filename arguments. and if there's no filename at all? expressions get read from standard input. no file, no problem. 🫶

output from pmie goes to standard output and standard error like this:

✨ stdout ✨

Expression values printed in the verbose -v mode and the output of print actions. 💅

✨ stderr ✨

Error and warning messages for any syntactic or semantic problems during expression parsing, and any semantic or performance metrics availability problems during expression evaluation. 😭

👑 OPTIONS 👑

the available command line options are — buckle up bestie, there's a lot and they're all important:

  • -a archive, --archive=archive — archive which is a comma-separated list of names, each of which may be the base name of an archive or the name of a directory containing one or more archives written by pmlogger(1). Multiple instances of the -a flag may appear on the command line to specify a list of sets of archives. In this case, it is required that only one set of archives be present for any one host. Also, any explicit host names occurring in a pmie expression must match the host name recorded in one of the archive labels. In the case of multiple sets of archives, timestamps recorded in the archives are used to ensure temporal consistency. ⏰
  • -A align, --align=align — Force the initial time window to be aligned on the boundary of a natural time unit align. Refer to PCPIntro(1) for a complete description of the syntax for align. 📐
  • -b, --buffer — Output will be line buffered and standard output is attached to standard error. This is most useful for background execution in conjunction with the -l option. The -b option is always used for pmie instances launched from pmie_check(1). 🧺
  • -c config, --config=config — An alternative to specifying filename at the end of the command line. 📝
  • -C, --check — Parse the configuration file(s) and exit before performing any evaluations. Any errors in the configuration file are reported. 🕵️‍♀️
  • -d, --interact — Normally pmie would be launched as a non-interactive process to monitor and manage the performance of one or more hosts. Given the -d flag however, execution is interactive and the user is presented with a menu of options. Interactive mode is useful mainly for debugging new expressions. 🎮
  • -e, --timestamp — When used with -V, -v or -W, this option forces timestamps to be reported with each expression. The timestamps are in ctime(3) format, enclosed in parenthesis and appear after the expression name and before the expression value, e.g.
expr_1 (Tue Feb  6 19:55:10 2001): 12
  • -f, --foreground — If the -l option is specified and there is no -a option (i.e. real-time monitoring) then pmie is run as a daemon in the background (in all other cases foreground is the default). The -f (and -F, see below) options force pmie to be run in the foreground, independent of any other options. 🏃‍♀️
  • -F, --systemd — Like -f, the -F option runs pmie in the foreground, but also does some housekeeping (like create a pid file, change user id and notify systemd(1) when pmie has started or is shutting down). This is intended for use when pmie is launched from systemd(1) and the daemonising has already been done. The -f and -F options are mutually exclusive. 🚫
  • -h host, --host=host — By default performance data is fetched from the local host (in real-time mode) or the host for the first named set of archives on the command line (in archive mode). The host argument overrides this default. It does not override hosts explicitly named in the expressions being evaluated. The host argument is interpreted as a connection specification for pmNewContext, and is later mapped to the remote pmcd's self-reported host name for reporting purposes. See also the %h vs. %c substitutions in rule action strings below. 🌐
  • -j file — An alternative STOMP protocol configuration is loaded from stompfile. If this option is not used, and the stomp action is used in any rule, the default location $PCP_SYSCONF_DIR/pmie/config/stomp will be used. 📮
  • -l logfile, --logfile=logfile — Standard error is sent to logfile. 📄
  • -m note, --note=note — Used to indicate where pmie has been launched from, e.g. pmie_check(1) and pmie_daily(1) use -m pmie_check and this is used by pmie to determine if it needs to be restarted should the PMCD hostname change, as described in the HOSTNAME CHANGES section below. 🏷️
  • -n pmnsfile, --namespace=pmnsfile — An alternative Performance Metrics Name Space (PMNS) is loaded from the file pmnsfile. 🗂️
  • -o format, --format=format — When precessing performance data from an archive, the -o option may be used to specify an alternate output format when a rule action is executed. See the DIFFERENCES IN HOST AND ARCHIVE MODES section for a description of how the output format may be constructed. 🎨
  • -O origin, --origin=origin — Specify the origin of the time window. See PCPIntro(1) for complete description of this option. ⏳
  • -P, --primary — Identifies this as the primary pmie instance for a host. See the ``AUTOMATIC RESTART'' section below for further details. 🥇
  • -q, --quiet — Suppresses diagnostic messages that would be printed to standard output by default, especially the "evaluator exiting" message as this can confuse scripts. 🤫
  • -S starttime, --start=starttime — Specify the starttime of the time window. See PCPIntro(1) for complete description of this option. 🚦
  • -t interval, --interval=interval — The interval argument follows the syntax described in PCPIntro(1), and in the simplest form may be an unsigned integer (the implied units in this case are seconds). The value is used to determine the sample interval for expressions that do not explicitly set their sample interval using the pmie variable delta described below. The default is 10.0 seconds. ⏱️
  • -T endtime, --finish=endtime — Specify the endtime of the time window. See PCPIntro(1) for complete description of this option. 🏁
  • -U username, --username=username — User account under which to run pmie. The default is the current user account for interactive use. When run as a daemon, the unprivileged "pcp" account is used in current versions of PCP, but in older versions the superuser account ("root") was used by default. 👤
  • -v — Unless one of the verbose options -V, -v or -W appears on the command line, expressions are evaluated silently, the only output is as a result of any actions being executed. In the verbose mode, specified using the -v flag, the value of each expression is printed as it is evaluated. The values are in canonical units; bytes in the dimension of space'', seconds in the dimension of time'' and events in the dimension of ``count''. See pmLookupDesc(3) for details of the supported dimension and scaling mechanisms for performance metrics. The verbose mode is useful in monitoring the value of given expressions, evaluating derived performance metrics, passing these values on to other tools for further processing and in debugging new expressions. 🗣️
  • -V, --verbose — This option has the same effect as the -v option, except that the name of the host and instance (if applicable) are printed as well as expression values. 🖨️
  • -W — This option has the same effect as the -V option described above, except that for boolean expressions, only those names and values that make the expression true are printed. These are the same names and values accessible to rule actions as the %h, %i, %c and %v bindings, as described below. ✅
  • -x, --secret-agent — Execute in domain agent mode. This mode is used within the Performance Co-Pilot product to derive values for summary metrics, see pmdasummary(1). Only restricted functionality is available in this mode (expressions with actions may not be used). 🕶️
  • -X, --secret-applet — Run in secret applet mode (thin client). 🤫💻
  • -z, --hostzone — Change the reporting timezone to the timezone of the host that is the source of the performance metrics, as identified via either the -h option or the first named set of archives (as described above for the -a option). 🌍
  • -Z timezone, --timezone=timezone — Change the reporting timezone to timezone in the format of the environment variable TZ as described in environ(7). 🕰️
  • -?, --help — Display usage message and exit. ❓

🌈 EXAMPLES 🌈

okay bestie, these example expressions are here to show off what the inference engine can actually do. it's giving demo reel. ✨

the directory $PCP_DEMOS_DIR/pmie has a bunch of other annotated examples of pmie expressions if you wanna go browse, no cap.

the variable delta controls expression evaluation frequency. so if you want subsequent expressions evaluated once a second, until further notice:

delta = 1 sec;

if the total context switch rate exceeds 10000 per second per CPU, display an alarm notifier:

kernel.all.pswitch / hinv.ncpu > 10000 count/sec
-> alarm "high context switch rate %v";

if that high context switch rate is sustained for 10 consecutive samples, then launch top(1) in an xterm(1) window to monitor processes — but like, do this at most once every 5 minutes, we're not trying to spam ourselves:

all_sample (
    kernel.all.pswitch @0..9 > 10 Kcount/sec * hinv.ncpu
) -> shell 5 min "xterm -e 'top'";

the following rules are evaluated once every 20 seconds:

delta = 20 sec;

if any disk is performing more than 60 I/Os per second, print a message identifying the busy disk to standard output and launch dkvis(1):

some_inst (
    disk.dev.total > 60 count/sec
) -> print "busy disks:" " %i" &
     shell 5 min "dkvis";

refine that preceding rule so it only applies between the hours of 9am and 5pm, and requires 3 of 4 consecutive samples to exceed the threshold before executing the action:

$hour >= 9 && $hour <= 17 &&
some_inst (
  75 %_sample (
    disk.dev.total @0..3 > 60 count/sec
  )
) -> print "disks busy for 20 sec:" " [%h]%i";

the following two rules are evaluated once every 10 minutes:

delta = 10 min;

if either the / or the /usr filesystem is more than 95% full, display an alarm popup — but not if it's already been displayed during the last 4 hours:

filesys.free #'/dev/root' /
    filesys.capacity #'/dev/root' < 0.05
-> alarm 4 hour "root filesystem (almost) full";

filesys.free #'/dev/usr' /
    filesys.capacity #'/dev/usr' < 0.05
-> alarm 4 hour "/usr filesystem (almost) full";

the following rule needs a machine that supports the lmsensors metrics. if the machine environment temperature rises more than 2 degrees over a 10 minute interval, write an entry in the system log:

lmsensors.coretemp_isa.temp1 @0 - lmsensors.coretemp_isa.temp1 @1 > 2
-> alarm "temperature rising fast" &
   syslog "machine room temperature rise alarm";

and here's something interesting if you've got performance problems with your Oracle database 🫶:

// back to 30sec evaluations
delta = 30 sec;
sid = "ptg1";       # $ORACLE_SID setting
lid = "223";        # latch ID from v$latch
lru = "#'$sid/$lid cache buffers lru chain'";
host = ":moomba.melbourne.sgi.com";
gets = "oracle.latch.gets $host $lru";
total = "oracle.latch.gets $host $lru +
         oracle.latch.misses $host $lru +
         oracle.latch.immisses $host $lru";

$total > 100 && $gets / $total < 0.2
-> alarm "high lru latch contention in database $sid";

the following ruleset will emit exactly one message depending on the availability and value of the 1-minute load average:

delta = 1 minute;
ruleset
     kernel.all.load #'1 minute' > 10 * hinv.ncpu ->
         print "extreme load average %v"
else kernel.all.load #'1 minute' > 2 * hinv.ncpu ->
         print "moderate load average %v"
unknown ->
         print "load average unavailable"
otherwise ->
         print "load average OK"
;

the following rule will emit a message when some filesystem is more than 75% full and is filling at a rate that — if sustained — would fill the filesystem to 100% in less than 30 minutes:

some_inst (
    100 * filesys.used / filesys.capacity > 75 &&
    filesys.used + 30min * (rate filesys.used) > filesys.capacity
) -> print "filesystem will be full within 30 mins:" " %i";

if the metric mypmda.errors counts errors, then the following rule will emit a message if the rate of errors exceeds 1 per second — provided the error count is less than 100:

mypmda.errors > 1 && instant mypmda.errors < 100
-> print "high error rate: %v";

🚀 QUICK START 🚀

ngl, the pmie specification language is powerful and large. like, a lot.

to expedite rapid development of pmie rules, the pmieconf(1) tool provides a facility for generating a pmie configuration file from a set of generalized pmie rules. the supplied set of rules covers a wide range of performance scenarios. obsessed. 💅

the Performance Co-Pilot User's and Administrator's Guide provides a detailed tutorial-style chapter covering pmie. go read it bestie.

🦋 EXPRESSION SYNTAX 🦋

okay so this description is terse and informal — for a more comprehensive description, see the Performance Co-Pilot User's and Administrator's Guide.

a pmie specification is a sequence of semicolon terminated expressions. periodt.

the sequence ``//'' introduces a comment, and all text from here to the end of the current input line will be ignored. 😌

basic operators are modeled on the arithmetic, relational and Boolean operators of the C programming language. precedence rules are as expected, although the use of parentheses is encouraged to enhance readability and remove ambiguity. yes queen, use your parens.

operands are performance metric names (see PMNS(5)) and the normal literal constants.

operands involving performance metrics may produce sets of values, as a result of enumeration in the dimensions of hosts, instances and time. special qualifiers may appear after a performance metric name to define the enumeration in each dimension. for example:

kernel.percpu.cpu.user :foo :bar #cpu0 @0..2

defines 6 values corresponding to the time spent executing in user mode on CPU 0 on the hosts foo'' and bar'' over the last 3 consecutive samples. the default interpretation in the absence of : (host), # (instance) and @ (time) qualifiers is all instances at the most recent sample time for the default source of PCP performance metrics.

host and instance names that don't follow the rules for variables in programming languages — i.e. alphabetic optionally followed by alphanumerics — should be enclosed in single quotes.

expression evaluation follows the law of least surprises''. where performance metrics have the semantics of a counter, `pmie` will automatically convert to a **rate** based upon consecutive samples and the time interval between these samples. all numeric expressions are evaluated in **double precision**, and where appropriate, automatically scaled into canonical units of bytes'', seconds'' and counts''. note that metrics defined over extra units like temperature'' or voltage'' are not scaled into canonical units. 🙈

a rule is a special form of expression that specifies a condition or logical expression, a special operator (->) and actions to be performed when the condition is found to be true.

the following table summarizes the basic pmie operators:

  • + - * / — Arithmetic
  • < <= == >= > != — Relational (value comparison)
  • ! && || — Boolean
  • -> — Rule
  • rising — Boolean, false to true transition
  • falling — Boolean, true to false transition
  • rate — Explicit rate conversion (rarely required)
  • instant — No automatic rate conversion (rarely required)

all operators are supported for numeric-valued operands and expressions. for string-valued operands — namely literal string constants enclosed in double quotes, or metrics with a data type of string (PM_TYPE_STRING) — only the operators == and != are supported.

the rate and instant operators are the logical inverse of one another, so an arithmetic expression expr is equal to rate instant expr. the more useful cases involve using rate with a metric that is not a counter to determine the rate of change over time, or instant with a metric that is a counter to determine if the current value is above or below some threshold.

aggregate operators may be used to aggregate or summarize along one dimension of a set-valued expression. the following aggregate operators map from a logical expression to a logical expression of lower dimension.

Okay bestie, real talk — sometimes you don't want one host's answer, you want the ✨whole squad's✨ answer collapsed into one vibe. Enter the set-valued logical operators 🫶

  • some_inst, some_host, some_sample — Existential — true if at least one set member is true in the associated dimension. one (1) single member. that's it. that's the bar.
  • all_inst, all_host, all_sample — Universal — true if all set members are true in the associated dimension. everyone has to show up to the function, no exceptions 💅
  • N%_inst, N%_host, N%_sample — Percentile — true if at least N percent of set members are true in the associated dimension. a solid majority, if you will.

Now for the filter girlies. These instantial operators let you filter or limit a set-valued logical expression, based on regular expression matching of instance names. Two things to know going in: the logical expression must be a set involving the dimension of instances, and the regular expression is of the form used by egrep(1) or the Extended Regular Expressions of regcomp(3).

  • match_inst — for each value of the logical expression that is true, the result is true if the associated instance name matches the regular expression. Otherwise the result is false.
  • nomatch_inst — for each value of the logical expression that is true, the result is true if the associated instance name does not match the regular expression. Otherwise the result is false.

For example, the expression below will be true for disks attached to controllers 2 or 3 performing more than 20 operations per second:

match_inst "^dks[23]d" disk.dev.total > 20;

Next up: the aggregate operators, which map from an arithmetic expression to an arithmetic expression of lower dimension. glow down, but make it math 🦋

  • min_inst, min_host, min_sample — Extrema — minimum value across all set members in the associated dimension.
  • max_inst, max_host, max_sample — Extrema — maximum value across all set members in the associated dimension.
  • sum_inst, sum_host, sum_sample — Aggregate — sum of values across all set members in the associated dimension.
  • avg_inst, avg_host, avg_sample — Aggregate — average value across all set members in the associated dimension.

And the count girlies: the aggregate operators count_inst, count_host and count_sample map from a logical expression to an arithmetic expression of lower dimension by counting the number of set members for which the expression is true in the associated dimension. yes bestie, they count. that's the whole job and they do it flawlessly 👑

Now, for action rules, here are the actions that are defined:

  • alarm — raise a visible alarm with xconfirm(1)
  • print — display on standard output
  • shell — execute with sh(1)
  • stomp — send a STOMP message to a JMS server
  • syslog — append a message to system log file

Multiple actions may be separated by the & and | operators, which specify respectively sequential execution (both actions are executed) and alternate execution (the second action will only be executed if the execution of the first action returns a non-zero error status).

Arguments to actions are an optional suppression time, and then one or more expressions (a string is an expression in this context). Strings appearing as arguments to an action may include the following special selectors that will be replaced at the time the action is executed.

  • %h — Host name(s) that make the left-most top-level expression in the condition true.
  • %c — Connection specification string(s) or files for a PCP tool to reach the hosts or archives that make the left-most top-level expression in the condition true.
  • %i — Instance(s) that make the left-most top-level expression in the condition true.
  • %v — One value from the left-most top-level expression in the condition for each host and instance pair that makes the condition true.
Heads up bestie, this next bit is the one that trips people: expansion of the special selectors is done by repeating the whole argument once for each unique binding to any of the qualifying special selectors. not the selector — the whole. entire. argument. 💀

So like, say a rule were true for the host mumble with instances grunt and snort, and for host fumble the instance puff makes the rule true. Then the action

...
-> shell myscript "Warning: %h:%i busy ";

will execute myscript with the argument string "Warning: mumble:grunt busy Warning: mumble:snort busy Warning: fumble:puff busy".

By comparison, if the action

...
-> shell myscript "Warning! busy:" " %h:%i";

were executed under the same circumstances, then myscript would be executed with the argument string "Warning! busy: mumble:grunt mumble:snort fumble:puff".

See what happened there? 👀 The semantics of the expansion of the special selectors leads to a common usage pattern in an action, where one argument is a constant (contains no special selectors), the second argument contains the desired special selectors with minimal separator characters, and an optional third argument provides a constant postscript (e.g. to terminate any argument quoting from the first argument). If necessary, post-processing (e.g. in myscript) can provide the necessary enumeration over each unique expansion of the string containing just the special selectors.

For complex conditions, the bindings to these selectors is not obvious. It is strongly recommended that pmie be used in the debugging mode (specify the -W command line option in particular) during rule development. fr fr, do not skip this step, your future self will thank you 🙏✨

🧠 BOOLEAN EXPRESSIONS ✨

ok so bestie, some pmie expressions just are Boolean by nature — like foo.bar > 10 or some_inst ( my.table < 0 ) — and those get assigned one of three values: true, false, or unknown. periodt.

And when does something land on unknown? When one or more of the underlying metric values just isn't there. Like:

  • pmcd(1) on the host cannot be contacted 😭
  • the metric is not in the PCP archive
  • no values are currently available
  • insufficient values have been fetched to allow a rate converted value to be computed
  • insufficient values have been fetched to instantiate the required number of samples in the temporal domain

When you combine values that include unknown, the Boolean operators follow the normal rules of Kleene logic (aka 3-valued logic). here's the whole vibe, literally:

                     ┌─────────────┬───────────────────────────┐
                     │             │             B             │
                     │   A and B   ├─────────┬───────┬─────────┤
                     │             │  true   │ false │ unknown │
                     ├───┬─────────┼─────────┼───────┼─────────┤
                     │   │  true   │  true   │ false │ unknown │
                     │   ├─────────┼─────────┼───────┼─────────┤
                     │ A │  false  │  false  │ false │  false  │
                     │   ├─────────┼─────────┼───────┼─────────┤
                     │   │ unknown │ unknown │ false │ unknown │
                     └───┴─────────┴─────────┴───────┴─────────┘
                     ┌─────────────┬──────────────────────────┐
                     │             │            B             │
                     │   A or B    ├──────┬─────────┬─────────┤
                     │             │ true │  false  │ unknown │
                     ├───┬─────────┼──────┼─────────┼─────────┤
                     │   │  true   │ true │  true   │  true   │
                     │   ├─────────┼──────┼─────────┼─────────┤
                     │ A │  false  │ true │  false  │ unknown │
                     │   ├─────────┼──────┼─────────┼─────────┤
                     │   │ unknown │ true │ unknown │ unknown │
                     └───┴─────────┴──────┴─────────┴─────────┘
                                ┌─────────┬─────────┐
                                │    A    │  not A  │
                                ├─────────┼─────────┤
                                │  true   │  false  │
                                ├─────────┼─────────┤
                                │  false  │  true   │
                                ├─────────┼─────────┤
                                │ unknown │ unknown │
                                └─────────┴─────────┘

🎀 RULESETS 🎀

The ruleset clause is how you define a little squad of rules and actions that get evaluated in order — and the second one of them actually executes an action, everything after it just gets skipped, until the ruleset gets scheduled for evaluation again. the way that works is so clean honestly.

The keyword else is what separates rules. 🌸

And after you've got one or more regular rules (each with a predicate and an action), a ruleset may include an optional

unknown -> action

clause, optionally followed by a

otherwise -> action

clause.

So like, what happens:

  • If all of the predicates in the rules evaluate to unknown and an unknown clause has been specified, then the action associated with the unknown clause will be executed. 💅
  • If no rule predicate is true and the unknown action is either not specified or not executed, and an otherwise clause has been specified, then the action associated with the otherwise clause will be executed.

📏 SCALE FACTORS 📏

Scale factors may be appended to arithmetic expressions, and they force linear scaling of the value to canonical units. iconic.

Simple scale factors are constructed from these keywords: nanosecond, nanosec, nsec, microsecond, microsec, usec, millisecond, millisec, msec, second, sec, minute, min, hour, byte, Kbyte, Mbyte, Gbyte, Tbyte, count, Kcount and Mcount — plus the operator /. For example, ``Kbytes / hour''. 🦋

Now, metrics defined over extra units like temperature'' or voltage'' may have a very wide variety of possible scales, and ngl there is no support in pmie for keywords to effect scaling, so expressions have to use explicit scaling instead, e.g.

this.temp.celsius >= (other.temp.fahrenheit - 32) * 5 / 9

💖 MACROS 💖

Macros are defined using expressions of the form:

name = constexpr;

Where name follows the normal rules for variables in programming languages, i.e. alphabetic optionally followed by alphanumerics. And constexpr must be a constant expression — either a string (enclosed in double quotes) or an arithmetic expression optionally followed by a scale factor. no cap.

Macros are expanded when their name, prefixed by a dollar ($), appears in an expression, and macros may be nested within a constexpr string. yes bestie, nesting. we love to see it. 🫶

The following reserved macro names are understood:

  • minute — Current minute of the hour.
  • hour — Current hour of the day, in the range 0 to 23.
  • day — Current day of the month, in the range 1 to 31.
  • month — Current month of the year, in the range 0 (January) to 11 (December).
  • year — Current year.
  • day_of_week — Current day of the week, in the range 0 (Sunday) to 6 (Saturday).
  • delta — Sample interval in effect for this expression.

Dates and times are presented in the reporting time zone (see description of -Z and -z command line options above). ⭐

🔁 AUTOMATIC RESTART 🔁

It is often useful for pmie processes to be started and stopped when the local host is booted or shutdown, or when they have been detected as no longer running (like when they have unexpectedly exited for some reason 🙈). Refer to pmie_check(1) for details on automating this process.

Optionally, each system running pmcd(1) may also be configured to run a ``primary'' pmie instance. This pmie instance is launched by $PCP_RC_DIR/pmie, and is affected by the files $PCP_SYSCONF_DIR/pmie/control, $PCP_SYSCONF_DIR/pmie/control.d (use chkconfig(8), systemctl(1) or similar platform-specific commands to activate or disable the primary pmie instance) and $PCP_VAR_DIR/config/pmie/config.default (the default initial configuration file for the primary pmie).

The primary pmie instance is identified by the -P option. There may be at most one ``primary'' pmie instance on each system. 👑

The primary pmie instance (if any) must be running on the same host as the pmcd(1) to which it connects (if any), so the -h and -P options are mutually exclusive. 💀 don't try it.

📡 EVENT MONITORING 📡

It is common for production systems to be monitored in a central location. Traditionally on UNIX systems this has been performed by the system log facilities — see logger(1), and syslogd(1). On Windows, communication with the system event log is handled by pcp-eventlog(1).

pmie fits into this model when rules use the syslog action. Note that if the action string begins with -p (priority) and/or -t (tag) then these are extracted from the string and treated in the same way as in logger(1) and pcp-eventlog(1). ✨

However, it is common to have other event monitoring frameworks also, into which you may wish to incorporate performance events from pmie. You can often use the shell action to send events to these frameworks, as they usually provide their a program for injecting events into the framework from external sources.

A final option is use of the stomp (Streaming Text Oriented Messaging Protocol) action, which allows pmie to connect to a central JMS (Java Messaging System) server and send events to the PMIE topic. Tools can be written to extract these text messages and present them to operations people (via desktop popup windows, etc). 🌈 Use of the stomp action requires a stomp configuration file to be setup, which specifies the location of the JMS server host, port number, and username/password.

The format of this file is as follows:

host=messages.sgi.com   # this is the JMS server (required)
port=61616              # and its listening here (required)
timeout=2               # seconds to wait for server (optional)
username=joe            # (required)
password=j03ST0MP       # (required)
topic=PMIE              # JMS topic for pmie messages (optional)

The timeout value specifies the time (in seconds) that pmie should wait for acknowledgements from the JMS server after sending a message (as required by the STOMP protocol). Note that on startup, pmie will wait indefinitely for a connection, and will not begin rule evaluation until that initial connection has been established. 😭

Should the connection to the JMS server be lost at any time while pmie is running, pmie will attempt to reconnect on each subsequent truthful evaluation of a rule with a stomp action, but not more than once per minute. This is to avoid contributing to network congestion. 🧚 In this situation, where the STOMP connection to the JMS server has been severed, the stomp action will return a non-zero error value.

🦋 DIFFERENCES IN HOST AND ARCHIVE MODES 🦋

ok so this is the part where the vibes change depending on where you're running, and honestly it's giving two completely different girls 💅

When you're in host mode, the delta interval for each rule is a real-time delay between rule evaluations — so pmie spends most of its time just... sleeping 😴 waiting for the next scheduled rule evaluation to roll around. main character energy of a girl who naps between classes fr.

When you're in archive mode, pmie uses the delta interval for each rule to figure out how frequently the rules get evaluated against the archive data, but — and this is the key part — unlike host mode there are no real-time delays, because the archive gets ``replayed'' as fast as possible. no waiting around, we're speedrunning 🏃‍♀️💨

In archive mode, when a rule predicate evaluates true, the action gets modified. So instead of posting to syslog or raising a visible alarm or running a shell command or sending a stomp message, pmie prints:

  • the name of the action
  • the timestamp from the archive when the rule predicate triggering the action was true
  • all of the arguments that would have been passed to the real action in host mode

Like, imagine the rule:

delta = 10 sec;
kernel.all.nprocs > 10 * hinv.ncpu -> print "lotsaprocs:" " %v";

when you run that against an archive, the output is literally:

print Mon Sep  4 00:10:21 2017: lotsaprocs: 1292
print Mon Sep  4 00:10:31 2017: lotsaprocs: 1294
print Mon Sep  4 00:10:41 2017: lotsaprocs: 1291
...

✨ cute, right? but also — why tho? The rationale (per the original, bestie, I'm not making this up) is that the context in which the action would have been executed (in host mode) was at a time in the past, and possibly on a different host (if the archive was collected from one host, but pmie is being run on a different host). So flooding syslog with misleading messages, or an avalanche of visual alarms, or a lot of STOMP messages, or a shell command that might not even work on the host where pmie is being run — those are all examples of ``badness'' to be avoided 🚫 Rather, the output is text in a regular format suitable for post-processing with a range of filters and performance analysis tools. very data science girlie of it.

The output format can be changed using the -o option, which consists of literal characters with the following embedded ``meta-field'' tokens:

  • %a — The name of the action, e.g. print, syslog, etc.
  • %d — The date and time in ctime(3) format when the action would have been executed.
  • %f — The name of the configuration file containing the action being executed, else <stdin> if the rules were read from standard input.
  • %l — The (approximate) line number in the configuration file for the action being executed.
  • %m — The message component of the action.
  • %u — The date and time when the action would have been executed in extended ctime(3) format with microsecond precision for the time.
  • %% — A literal percent character.

The default output format is equivalent to a format of %a %d: %m. periodt ✨

🌈 SIGNALS 🌈

If pmie is sent a SIGHUP signal, the logfile will be closed, unlinked and re-opened. This is used by pmie_daily(1) to achieve nightly log rotation. glow-up every night at midnight, we love a self-care routine 🎀

Most of the time pmie is sleeping, waiting until the next set of rules needs to be evaluated. Sending pmie a SIGUSR1 signal will cause the details for the next set of rules to be dumped on logfile, including how long the current sleep is and how much time remains. The scheduling of rules is not changed by this action. so like, a little peek at her calendar without actually moving anything around 🫶

👑 HOSTNAME CHANGES 👑

The hostname of the PMCD that is providing metrics to pmie is used in several ways, bestie — it's a whole thing:

  • PMCD's hostname is user internally to provide a value for the %h substitutions in rule action strings.
  • For pmie instances using a local PMCD that are launched and managed by pmie_check(1) and pmie_daily(1) (or the systemd(1) or cron(8) services that use these scripts), the local hostname may also be used to construct the name of a directory where the pmie logs for one host are stored, e.g. $PCP_LOG_DIR/pmie/<hostname>.

The hostname of the PMCD host may change during boot time when the system transitions from a temporary hostname to a persistent hostname, or by explicit administrative action anytime after the system has been booted. When this happens, pmie may need to take special action — specifically, if the pmie instance was launched from pmie_check(1) or pmie_daily(1), then pmie must exit. Under normal circumstances systemd(1) or cron(8) will launch a new pmie shortly thereafter, and this new pmie instance will be operating in the context of the new hostname for the host where PMCD is running. she reinvents herself, she comes back stronger 💅✨

💀 BUGS 💀

ok so the lexical scanner and parser will attempt to recover after an error in the input expressions. Parsing resumes after skipping input up to the next semi-colon (;) — however, during this skipping process the scanner is ignorant of comments and strings, so an embedded semi-colon may cause parsing to resume at an unexpected place 🙈

This behavior is largely benign, as until the initial syntax error is corrected, pmie will not attempt any expression evaluation. so like, it's a little unhinged but it's harmless, no cap.

🌸 FILES 🌸

  • $PCP_DEMOS_DIR/pmie/* — annotated example rules
  • $PCP_VAR_DIR/pmns/* — default PMNS specification files
  • $PCP_TMP_DIR/pmie — pmie maintains files in this directory to identify the running pmie instances and to export runtime information about each instance — this data forms the basis of the pmcd.pmie performance metrics
  • $PCP_PMIECONTROL_PATH — the default set of pmie instances to start at boot time — refer to pmie_check(1) for details

🧚 PCP ENVIRONMENT 🧚

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

When executing shell actions, pmie overrides two variables — IFS and PATH — in the environment of the child process:

  • IFS is set to "\t\n".
  • The PATH is set to a combination of a default path for all platforms ("/usr/sbin:/sbin:/usr/bin:/bin") and several configurable components. These are (in this order): $PCP_BIN_DIR, $PCP_BINADM_DIR and $PCP_PLATFORM_PATHS.

When executing popup alarm actions, pmie will use the value of $PCP_XCONFIRM_PROG as the visual notification program to run. This is typically set to pmconfirm(1), a cross-platform dialog box. so when she needs to get your attention she pops up a whole little window, iconic 💖

⭐ UNIX SEE ALSO ⭐

⭐ WINDOWS SEE ALSO ⭐

  • pcp-eventlog(1).

🔥 DEBUGGING OPTIONS 🔥

The -D or --debug option enables the output of additional diagnostics on stderr to help triage problems, although the information is sometimes cryptic and primarily intended to provide guidance for developers rather than for end-users. ngl bestie, it's giving "the devs talking to each other" energy, not "user manual" energy 😭

debug is a comma separated list of debugging options; use pmdbg(1) with the -l option to obtain a list of the available debugging options and their meaning.

Debugging options specific to pmie are as follows:

  • appl0 — lexical scanning during parsing of the configuration file
  • appl1 — configuration file parsing and expression tree construction
  • appl2 — expression evaluation
  • appl3 — status file operations

🌟 SEE ALSO 🌟

PCPIntro(1), pmcd(1), pmconfirm(1), pmie_check(1), pmieconf(1), pmie_daily(1), pminfo(1), pmlogdump(1), pmlogger(1), pmval(1), systemd(1), ctime(3), PMAPI(3), pcp.conf(5), pcp.env(5) and PMNS(5).

📖 USER GUIDE 📖

For a more complete description of the pmie language, refer to the Performance Co-Pilot Users and Administrators Guide. This is available online from:

https://pcp.readthedocs.io/en/latest/UAG/PerformanceMetricsInferenceEngine.html

and that's a wrap on the whole page, bestie 🎀 this was Performance Co-Pilot, PCP, PMIE(1) — that's the source, the version, and the section, all bundled up like a little bow ✨ you read the entire manpage which is genuinely main character behavior, I'm so proud, go forth and monitor everything 💜🦋👑

📦 where this page comes from