PCPINTRO(1)

๐Ÿงš PCPINTRO(1) โ€” General Commands Manual ๐Ÿงš

part 1 of 6 ๐Ÿ’–

๐ŸŒธ NAME ๐ŸŒธ

PCPIntro, aka the intro to the Performance Co-Pilot (PCP), your new favourite toolkit ๐Ÿฆ‹

๐ŸŽ€ INTRODUCTION ๐ŸŽ€

okay so the Performance Co-Pilot (PCP) is a toolkit built for keeping an eye on system-level performance ๐Ÿ‘€. It's giving distributed. The services are spread out and scalable enough to handle the most complicated system setups and the nastiest performance problems ๐Ÿ’….

PCP runs on loads of platforms, including (but not limited to) Linux, macOS, Solaris and Windows โœจ. Highkey, you can think of PCP as having two classes of software utility:

  • PCP Collectors โ€” the parts that collect and pull performance data from different sources, like the operating system kernel ๐Ÿง 
  • PCP Monitors โ€” the parts that display data collected from hosts (or archives) that have the PCP Collector installed ๐Ÿ“Š. Lots of monitor tools come with the core PCP release, while other (usually graphical) ones come separately in the PCP GUI package ๐ŸŽจ

This manual entry covers the high-level features and options that most PCP utilities share, on every platform ๐Ÿซถ.

๐Ÿ‘‘ OVERVIEW ๐Ÿ‘‘

The PCP architecture is distributed, which means any PCP tool can be running remotely ๐ŸŒ. On the monitored host (or hosts), each domain of performance metrics needs a Performance Metrics Domain Agent (PMDA) to collect measurements from it. That could be the kernel, a service layer, a database management system, a web server, an application, etc. All the PMDAs are controlled by the Performance Metrics Collector Daemon (pmcd(1)) on the same host ๐Ÿ”.

Client applications (the monitoring tools) connect to pmcd(1), which acts as a router for requests. It forwards each request to the right PMDA and sends the responses back to the clients ๐Ÿ”€. Clients can also access performance data from sets of PCP archives (created using pmlogger(1)) for retrospective analysis ๐Ÿ•ฐ๏ธ.

๐Ÿฅ€ Security philosophy

PCP shares a wealth of performance information within a host and across its networks ๐Ÿซฃ. The security philosophy below is what sets several of the defaults that control how much information gets sent and received.

By default, the information PMCD exposes about a host has about the same level of confidentiality as what a completely unprivileged user on that host can see. So performance data that's freely readable on a machine may be made available by PMCD to the network ๐ŸŒ.

However, the host running PMCD and its network aren't assumed to only run friendly applications ๐Ÿ™…โ€โ™€๏ธ. So write type operations, including from the local host, are not permitted by default ๐Ÿšซ.

These defaults can be overridden (expanded or reduced) in a few ways. You can specify network ACLs in pmcd.conf, activate non-default PMDAs, or use PMCD connections that pass user credentials ๐Ÿ”‘. For example, some PMDAs automatically provide greater information for particular credentialed users or groups ๐Ÿ‘€.

๐Ÿ› ๏ธ Applications

The following performance monitoring applications are mostly console based and typically run straight from the command line. They're just a small subset of the tools in the base PCP package ๐Ÿ’ป.

Each tool or command is documented completely in its own reference page ๐Ÿ“–.

  • pmstat โ€” outputs an ASCII high-level summary of system performance ๐Ÿ“
  • pmie โ€” an inference engine that evaluates predicate-action rules to raise alarms and automate system management tasks ๐Ÿค–
  • pminfo โ€” interrogates specific performance metrics and the metadata that describes them ๐Ÿ”
  • pmlogger โ€” generates PCP archives of performance metrics that are suitable for replay by most PCP tools ๐Ÿ“ผ
  • pmrep โ€” a highly customizable performance metrics reporter with support for lots of different output modes ๐ŸŽ›๏ธ
  • pmval โ€” simple periodic reporting for some or all instances of a performance metric, with optional VCR time control โฏ๏ธ

If the PCP GUI package is installed, these extra tools are available too ๐ŸŽจ:

  • pmchart โ€” displays trends over time for arbitrarily selected performance metrics from one or more hosts ๐Ÿ“ˆ
  • pmtime โ€” a time control utility for coordinating time between multiple tools (including pmchart and pmval) โฐ
  • pmdumptext โ€” produces ASCII reports for arbitrary combinations of performance metrics ๐Ÿงพ

๐Ÿ’… COMMON COMMAND LINE ARGUMENTS ๐Ÿ’…

There's a set of common command line arguments that most PCP tools use consistently. These are the ones to know, bestie ๐Ÿง .

  • -a archive, --archive=archive โ€” pulls performance metric info retrospectively from the PCP archives named by archive, which were previously generated by pmlogger(1) ๐Ÿ•ฐ๏ธ. For archive creation interfaces and format documentation, check out LOGIMPORT(3) and LOGARCHIVE(5) ๐Ÿ“š.

archive is a comma-separated list of names. Each one can be:

  • a directory containing one or more archives, e.g. /path/to/myarchives
  • the base name shared by all the physical files created by one instance of pmlogger(1), e.g. myarchive
  • any one of the physical files, such as:
    • myarchive.meta (the metadata file)
    • myarchive.index (the temporal index)
    • myarchive.0 (the first data volume of the archive)
    • myarchive.0.bz2 or myarchive.0.bz (the first data volume compressed with bzip2(1))
    • myarchive.0.gz, myarchive.0.Z or myarchive.0.z (the first data volume compressed with gzip(1))
    • myarchive.1, myarchive.3.bz2, myarchive.42.gz, etc.
  • -h host, --host=host โ€” unless you point somewhere else with -h (or --host), or at a set of archives with -a (or --archive), the metrics come from the Performance Metrics Collector Daemon (PMCD) on the local host ๐Ÿ . Check out the PMCD HOST SPECIFICATION section later for the many options available when forming a host specification, plus a detailed description of the default local host connection ๐Ÿ”Œ. The -a (or --archive) and -h (or --host) options are mutually exclusive ๐Ÿ™….
    • -s samples, --samples=samples โ€” sets how many samples get retrieved and reported. If samples is 0, or -s (or --samples) isn't specified, the application samples and reports continuously (in real time mode) or until the end of the set of PCP archives (in archive mode) โ™พ๏ธ.
  • -z, --hostzone โ€” switches the reporting timezone to the local timezone at the host that's the source of the performance metrics, as identified by -h (or --host) or -a (or --archive) ๐ŸŒ.
  • -Z timezone, --timezone=timezone โ€” by default, apps report the time of day in the local timezone of the system they run on. This option changes the timezone to timezone, written in the format of the TZ environment variable described in environ(7) ๐Ÿ•.
  • -D debugspec, --debug=debugspec โ€” sets the PCP debugging options to debugspec. This turns on diagnostics and tracing that are most helpful for developers, or when you're trying to figure out why a PCP application is misbehaving ๐Ÿ›. debugspec should be a comma-separated list of debugging option names and/or decimal integers. See pmdbg(1) for the supported option names and values ๐Ÿ“‹.

If there's no live or archive source of metrics, you can kick off a heuristic search for archives for the local host with the -O (or --origin) option. When you use this option without an explicit source of metrics, monitor tools try to use archives from a system archive location such as $PCP_LOG_DIR/pmlogger/`hostname` . Check the TIME WINDOW SPECIFICATION section below for the acceptable syntax for the origin option. A typical invocation in this mode would be -O today or --origin yesterday ๐ŸŒ….

โณ INTERVAL SPECIFICATION AND ALIGNMENT โณ

Most PCP tools work with periodic sampling or reporting. The -t (or --interval) and -A (or --align) options control how long each sample interval is and how the sample times line up โฑ๏ธ.

  • -t interval, --interval=interval โ€” sets the update or reporting interval ๐Ÿ”

The interval argument is a sequence of one or more elements of this form:

number[units]

number is an integer or floating point constant (parsed using strtod(3)). units is optional and can be one of: seconds, second, secs, sec, s, minutes, minute, mins, min, m, hours, hour, h, days, day and d. If the unit is empty, second is assumed ๐Ÿ•’.

Upper case (or mixed case) versions of any of the above are also fine ๐Ÿ” .

Spaces anywhere in the interval are ignored, so 4 days 6 hours 30 minutes, 4day6hour30min, 4d6h30m and 4d6.5h are all equivalent ๐Ÿงฎ.

Multiple specifications are additive. For example, 1hour 15mins 30secs is interpreted as 3600+900+30 seconds โž•.

  • -A align, --align=align โ€” by default, samples aren't necessarily aligned on any natural unit of time. The -A or --align option forces the initial sample onto the boundary of a natural time unit โฐ. For example, -A 1sec, -A 30min and --align 1hour specify alignment on whole seconds, half hours and whole hours respectively.

The align argument follows the same syntax as the interval argument described above ๐Ÿ“.

Heads up: alignment happens by advancing the time as required. -A (or --align) acts as a modifier that advances both the start of the time window (see the next section) and the origin time (if the -O or --origin option is specified) โš ๏ธ

โฐ TIME WINDOW SPECIFICATION โฐ

Okay bestie, so a bunch of PCP tools only care about a certain slice of time ๐Ÿฆ‹ Like "keep monitoring until this moment" for real-time stuff, or "only grab what's between these two points" when you're digging through PCP archives ๐Ÿ” Main character energy, honestly ๐Ÿ’…

If you skip the -O (or --origin) and -A (or --align) options, which set the initial sample time origin and time alignment (see above), the PCP app just grabs the first sample right at the start of the time window ๐ŸŒธ

Here's how you can pick your window ๐ŸŽ€

  • -S starttime, --start=starttime โ€” This sets when the window kicks off. By default it starts immediately in real-time mode, or lines up with the start of the PCP archive set in archive mode. Use -S or --start when you want it to start later than that ๐Ÿ’–

The starttime can come in one of three flavours. (interval works the same way as the -t or --interval option we covered above, and datetime gets explained further down ๐Ÿงš)

  • interval โ€” Count forward from now (real-time mode) or from the beginning of a PCP archive set (archive mode) by just dropping in an interval. For example, -S 30min puts the start exactly 30 minutes from now in real-time mode, or exactly 30 minutes from the start of an archive set.
  • -interval โ€” Count backwards from the end of a PCP archive set by adding a minus sign in front of the interval. The start then sits that interval before the last sample. For example, -S -1hour puts the start exactly one hour before the time of the last sample in the archive set โณ
  • @datetime โ€” Pick an exact calendar date and time (local time in the reporting timezone) by writing the datetime syntax with an at sign in front. Check the datetime section below for all the juicy details ๐Ÿ“…
  • -T endtime, --finish=endtime โ€” This sets when the window closes. By default the end is unbounded in real-time mode, or lines up with the end of a PCP archive set in archive mode. Use -T or --finish if you want it to end earlier ๐ŸŒ™

The endtime can come in one of three flavours too (again, interval matches the -t or --interval option from above, and datetime is explained below ๐Ÿ’œ):

  • interval โ€” Count forward from the start of the time window. For example, -T 2h30m ends the window 2 hours and 30 minutes after the start of the window ๐Ÿ•‘
  • -interval โ€” Count backwards from the time at the end of a PCP archive set by adding a minus sign in front. For example, -T -90m ends the window 90 minutes before the time of the last sample in the archive set ๐Ÿ“
  • @datetime โ€” Pick an exact calendar date and time (local time in the reporting timezone) for the end by putting an at sign in front of the datetime syntax. Scroll down for the deets ๐Ÿ—“๏ธ
  • -O origin, --origin=origin โ€” By default, samples get fetched from the start of the time window (see -S or --start) all the way to the end (see -T or --finish). With -O or --origin, you get to pick a spot inside the window to use as the initial sample time ๐Ÿฆ‹ This is super handy for interactive use of a PCP tool with the pmtime(1) VCR replay facility ๐Ÿ“ผ

The origin argument follows the exact same syntax and semantics as the starttime argument for -T (or --finish). So, for example, --origin -0 puts the starting position at the end of the time window. That's most useful when you want to replay "backwards" through the window ๐Ÿ”™๐Ÿ’…

The datetime situation ๐ŸŽ€

The datetime argument for -O (or --origin), -S (or --start) and -T (or --finish) is built out of these pieces, in this order: date time zone day relative

  • Date โ€” Can be YY-MM-DD, MM/DD/YY, DD Month YYYY, or Month DD YYYY โœจ
  • Time โ€” Can be HH:MM:SS or HH:MM. The HH:MM version can use either the 12 hour clock (with an am or pm suffix) or the 24 hour clock ๐Ÿ•ฐ๏ธ
  • Day โ€” A day of the week spelled out in full, optionally with an ordinal number in front, like second Tuesday ๐Ÿ“†
  • Zone โ€” A time zone value as specified by the tzselect(8) command ๐ŸŒ
  • Relative โ€” A time unit preceded by a cardinal number, like 1 year or 2 months. It can also be preceded by the time word this or last, or followed by the time word ago. And there are four special time words all on their own: yesterday, today, tomorrow, now ๐Ÿ’ซ

Here are some valid datetime strings, bestie:

1996-03-04 13:07:47 EST Mon
1996-03-05 14:07:47 EST -1hour
Mon Mar  4 13:07:47 1996
Mar 4 1996
Mar 4
Mar
13:07:50
13:08

For any low order fields you leave out, PCP fills in defaults: 0 for hours, minutes and seconds, 1 for the day of the month, and Jan for months. That means --start '@ Mar 1996' and --start '@ Mar 1 00:00:00 1996' are totally equivalent, no cap ๐Ÿ™Œ

If any high order fields are missing, they get filled in by starting with the year, month and day from the current time (real-time mode) or from the time at the beginning of the PCP archive set (archive mode). Then the time keeps advancing until it matches the fields you did specify ๐Ÿ”ฎ So say the window starts by default at Mon Mar 4 13:07:47 1996. Then --start @13:10 lands on 13:10:00 on Mon Mar 4, 1996. But --start @10:00 lands on 10:00:00 on Tue Mar 5, 1996, which is the following day ๐Ÿ‘€ (the chaos is real ๐Ÿ˜ญ)

For more precision than datetime(3) gives you, the seconds component can be a floating point number โœจ

If a datetime doesn't include a timezone, there are a few possible interpretations depending on the other command line options you used ๐Ÿงญ The default is the local timezone on the system where the PCP tool is running. A -Z or --timezone option sets an explicit timezone. Otherwise, a -z or --hostzone option switches the timezone to the local timezone at the host that's the source of the performance metrics ๐ŸŒˆ

โœจ PERFORMANCE METRICS - IDENTIFIERS, NAMES, VALUES โœจ

okay so bestie, most platforms have like, many hundreds to several thousand performance metric names ๐Ÿคฏ๐Ÿ’…. PCP keeps track of all of them with an internal ID system that gives every known metric exactly one integer. That integer is the Performance Metric Identifier, or PMID ๐Ÿ”ข. It's not a hard rule, but PMIDs are usually consistent across every system, so one metric tends to have the same PMID everywhere โœจ.

Direct PMID use is a no-go for most users and apps ๐Ÿ™…โ€โ™€๏ธ. It would limit which metrics you can reach, make the code a nightmare to maintain, and force the user interface to be super baroque, and so on ๐Ÿ’€. So enter the Performance Metrics Name Space, or PMNS ๐ŸŒณ. It gives metrics external names and a hierarchic classification. Picture it as a tree ๐ŸŒฒ where every node has a label, and each node points to either a PMID (for leaf nodes) or a set of descendent nodes in the PMNS (for non-leaf nodes).

A node label has to start with an alphabetic character, then can have zero or more alphabetics, digits, and _ (underscore) characters ๐ŸŽ€. Upper and lower case count as different for alphabetic characters, so caps matter, bestie ๐Ÿ™ˆ.

By convention, a metric's name is made by smashing together the node labels along a path from the root to a leaf, with a . as the separator ๐Ÿงฉ. The root node is unlabeled, so every name starts with the label of one of the root's descendents, like kernel.percpu.syscall ๐Ÿ–ฅ๏ธ. Typically (though it's not required) there's at most one name per PMID in a PMNS. For example, kernel.all.cpu.idle and disk.dev.read are the unique names for two different metrics, each with its own PMID ๐Ÿ’œ.

You can name a whole group of related PMIDs by naming a non-leaf node in the tree, like disk ๐ŸŒธ.

The default local PMNS used by pmcd is at $PCP_VAR_DIR/pmns/root ๐Ÿ“‚. But if you set the environment variable PMNS_DEFAULT to the full pathname of a different PMNS, that one becomes the default local PMNS instead โœจ.

Most apps don't touch the local PMNS directly. They import the parts they need from wherever the metrics are fetched: pmcd(1) for live monitoring, or a set of PCP archives for retrospective monitoring ๐Ÿ•ฐ๏ธ.

To explore the PMNS, use pminfo(1). If the PCP GUI package is installed, you can also use the New Chart and Metric Search windows within pmchart(1) ๐Ÿ”.

Some metrics are singular, meaning they have one value per performance metric source, which is one value per host. Available memory and number of context switches are good examples ๐Ÿ . The metric descriptor (metadata) for each metric tells apps that process values for these single-valued metrics that this is the case ๐Ÿ“.

Other metrics have a set of values, called instances, in each implementing performance metric domain ๐Ÿง. Think one value per disk, one per process, one per CPU, or one per activation of a given application. When a metric has multiple instances, the PMNS doesn't show that in the metric name. Instead, a single metric just has an associated set of values ๐Ÿคทโ€โ™€๏ธ. Each value belongs to a member of an instance domain, and every instance has a unique instance identifier within that domain. For example, the "per CPU" instance domain might use the instance identifiers 0, 1, 2, 3, and so on to identify the configured processors in the system ๐Ÿ–ฅ๏ธ.

Internally, instance identifiers are encoded as binary values. But each performance metric domain also supports strings as external names for those identifiers, and those names are what get used at the user interface of the PCP utilities ๐Ÿ’ฌ.

Multiple performance metrics can share one instance domain. For example, per-process metrics under proc all share the same instance domain ๐Ÿ‘ฏ.

PCP makes sure information describing instance domains is exported from the performance metric domains to any apps that need it ๐Ÿ“ค. Apps can also choose to grab values for all instances of a metric, or for some arbitrary subset of the available instances โœ‚๏ธ.

Metric names and the instance domain concept give you two dimensions for modelling performance metrics. It's clear and simple, but sometimes it just doesn't cut it ๐Ÿ˜ค. Say a metric wants higher-dimensional data, like "per CPU" counters for each running process. In that case it's common to create a compound instance, where the name is built from each component with a separator in between. For example, 87245::cpu7 might separate a process ID from a CPU ID to form flattened instance names ๐Ÿง . Those cases also benefit from metric instance labels that explicitly show the separate components. Continuing the example above, the labels {"pid":87245,"cpu":7} might be used ๐ŸŒˆ.


โœจ PERFORMANCE METRIC SPECIFICATIONS โœจ

In config files, and (to a lesser extent) command line options, metric specifications follow these syntax rules in most tools ๐Ÿ“‹. Check the tool-specific manual pages for the exact syntax each one supports ๐Ÿ“–.

If the source of your metrics is real-time from pmcd(1), the accepted syntax is:

host:metric[instance1,instance2,...]

If the source is a set of PCP archives, the accepted syntax is:

archive/metric[instance1,instance2,...]

The host:, archive/ and [instance1,instance2,...] components are all optional ๐Ÿซถ.

The , delimiter in the list of instance names can be swapped out for white space ๐Ÿ“.

Special characters in instance names can be escaped by wrapping the name in double quotes, or by putting a backslash in front of the character ๐Ÿ›ก๏ธ.

White space is ignored everywhere except inside a quoted instance name ๐Ÿซง.

An empty instance is silently ignored. That means [] is the same as no instance at all, while [one,,,two] is parsed as specifying just the two instances one and two ๐Ÿ’ซ.

As a special case, if the host is the single character @, that refers to a PM_CONTEXT_LOCAL source. See pmNewContext(3) ๐Ÿ‘€.


โœจ SECURE PMCD CONNECTIONS โœจ

Since PCP version 3.6.11, a monitor can explicitly request a secure connection to a collector host running pmcd(1) or pmproxy(1) by using the PM_CTXFLAG_SECURE context flag ๐Ÿ”. If the PCP collector host supports this feature (check the pmcd.feature.secure metric to confirm ๐Ÿ•ต๏ธโ€โ™€๏ธ), a TLS/SSL (Transport Layer Security or Secure Sockets Layer) connection can be established. It uses public key cryptography and related techniques ๐Ÿฆ‹. These features aim to prevent eavesdropping and data tampering from a malicious third party. They also provide server-side authentication, meaning a client can confidently identify a server, which can help guard against man-in-the-middle attacks ๐Ÿ›ก๏ธ.

A secure pmcd connection requires certificate-based authentication ๐Ÿ“œ. The security features in pmcd and pmproxy are implemented using the OpenSSL APIs and utilities. The openssl(1) tool can be used to create certificates suitable for establishing trust between PCP monitor and collector hosts ๐ŸŒท.

A complete description is beyond the scope of this document. For the details, check the PCP ENVIRONMENT, FILES and SEE ALSO sections, which include links to tutorials on the steps involved in setting up the available security features ๐Ÿ“š๐Ÿ’–.

โœจ PMCD HOST SPECIFICATION โœจ

Okay so if you don't tell a tool which host to hit, most of them just default to the local host in live update mode ๐Ÿ’ป. In PCP releases since 3.8.4, that means an efficient local protocol gets picked automatically, usually a Unix domain socket ๐Ÿ”Œ. But if you use this (or ask for it explicitly with the unix: host specification we'll get to below), every connection is automatically authenticated ๐Ÿ”. The credentials of whoever is running a client tool get passed along to pmcd(1) and all of its PMDAs on their behalf, so the results can be tailored to each user's privilege level ๐Ÿ‘‘. Main character energy for sure ๐Ÿ’….

You can also point at remote hosts running the pmcd(1) daemon ๐ŸŒ. The most basic version is just a simple host name, possibly with the domain name tacked on if you need it. It can get extended in a bunch of ways to fine-tune the connection to pmcd(1) ๐Ÿฆ‹.

The pmcd port number, plus an optional pmproxy(1) hostname and its port number, can all go in the host specification as well, since PCP version 3.0 ๐ŸŽ€. These supersede (and override) the old-school PMCD_PORT, PMPROXY_HOST and PMPROXY_PORT environment variables, so they win โœจ.

These are all valid hostname specs that connect to pmcd on host nas1.acme.com, with or without a list of ports, with or without a pmproxy(1) connection through a firewall, and with IPv6 and IPv4 addresses too ๐Ÿงš:

$ pcp --host nas1.acme.com:44321,4321@firewall.acme.com:44322
$ pcp --host nas1.acme.com:44321@firewall.acme.com:44322
$ pcp --host nas1.acme.com:44321@firewall.acme.com
$ pcp --host nas1.acme.com@firewall.acme.com
$ pcp --host nas1.acme.com:44321
$ pcp --host [fe80::2ad2:44ff:fe88:e4f1%p2p1]
$ pcp --host 192.168.0.103

Then there are connection attributes ๐Ÿ“. These cover things like username, password (which can be entered interactively and may depend on the authentication mechanism in use), whether to target a specific running container, and whether to use a secure (encrypted) or native (naked) protocol ๐Ÿ™ˆ. Heads up: all the examples above default to the native protocol with no authentication. You can change that, like so:

$ pcp --host pcps://app2.acme.com?container=cae8e6edc0d5
$ pcp --host pcps://nas1.acme.com:44321?username=tanya&method=gssapi
$ pcp --host pcps://nas2.acme.com@firewalls.r.us?method=plain
$ pcp --host pcp://nas3.acme.com
$ pcp --host 192.168.0.103?container=cae8e6edc0d5,method=scram-sha-256
$ pcp --host unix:
$ pcp --host local:

Which authentication method you get, and the other stuff that comes with it like username and maybe password, depends on the SASL2 configuration each (remote) pmcd is using ๐Ÿ”‘. There are tutorials out there on configuring the authentication module(s), but those finer details are outside the scope of this page, so go hunting if you need them ๐Ÿซถ.

Honestly, in every situation you can swap host names for IPv4 or IPv6 addresses directly, as shown above ๐ŸŒˆ. For an IPv6 address, the full address has to be wrapped in square brackets, and the scope (interface) must also be specified ๐Ÿ’œ.

The final local: example from above is now the default for most tools ๐Ÿ‘‘. It's an automatically authenticated local host connection on every platform that supports Unix domain sockets. No password needed, authentication is automatic, and it's the most efficient (lowest overhead) channel going โšก.

Now the difference between unix: and local:, which is kind of a big deal ๐Ÿ‘€. unix: is a strict Unix domain socket specification, so the connection fails if it can't connect that way ๐Ÿšซ. local: is more forgiving, with a fallback to using localhost, meaning a regular Inet socket connection gets used when Unix domain socket connections aren't available ๐Ÿฆ‹.


๐ŸŽ€ FILES ๐ŸŽ€

  • /etc/pcp.conf โ€” The configuration file for the PCP runtime environment. See pcp.conf(5).
  • /etc/pcp/tls.conf โ€” Optionally holds OpenSSL configuration information, including the locations of certificates that provide trusted identification for collector and monitor hosts ๐Ÿ”’.
  • $HOME/.pcp โ€” User-specific directories containing configuration files for customising the various monitor tools, such as pmchart(1) ๐ŸŒธ.
  • $PCP_RC_DIR/pmcd โ€” The script for starting and stopping pmcd(1) โ–ถ๏ธ.
  • $PCP_PMCDCONF_PATH โ€” The control file for pmcd(1) ๐ŸŽ›๏ธ.
  • $PCP_PMCDOPTIONS_PATH โ€” The command line options passed to pmcd(1) when it's started from $PCP_RC_DIR/pmcd. Every command line option line should start with a hyphen as the first character. This file can also contain environment variable settings of the form "VARIABLE=value" ๐Ÿงพ.
  • $PCP_BINADM_DIR โ€” The location of PCP utilities for collecting and maintaining PCP archives, PMDA help text, PMNS files, etc. ๐Ÿ—‚๏ธ
  • $PCP_PMDAS_DIR โ€” The parent directory of the installation directory for Dynamic Shared Object (DSO) PMDAs ๐Ÿงฉ.
  • $PCP_RUN_DIR/pmcd.pid โ€” If pmcd is running, this file contains an ASCII decimal representation of its process ID ๐Ÿ†”.
  • $PCP_LOG_DIR/pmcd โ€” The default location of log files for pmcd(1), and the current directory for running PMDAs ๐Ÿ“. Archives generated by pmlogger(1) are generally found below $PCP_LOG_DIR/pmlogger.
  • $PCP_LOG_DIR/pmcd/pmcd.log โ€” The diagnostic and status log for the currently running pmcd(1) process. This is the first place to look when there are problems with pmcd ๐Ÿ”.
  • $PCP_LOG_DIR/pmcd/pmcd.log.prev โ€” The diagnostic and status log for the previous pmcd(1) instance ๐Ÿ•ฐ๏ธ.
  • $PCP_LOG_DIR/NOTICES โ€” A log of pmcd(1) and PMDA starts, stops, additions and removals ๐Ÿ“ฃ.
  • $PCP_VAR_DIR/config โ€” Contains directories of configuration files for several PCP tools ๐Ÿ› ๏ธ.
  • $PCP_SERVICES_DIR/local โ€” The local script for controlling PCP boot, shutdown and restart actions. This used to live in $PCP_SYSCONF_DIR/pmcd/rc.local, and a symbolic link is installed for that name to keep things compatible ๐Ÿ”—.
  • $PCP_VAR_DIR/pmns โ€” The directory containing the set of PMNS files for all installed PMDAs ๐Ÿ“š.
  • $PCP_VAR_DIR/pmns/root โ€” The ASCII PMNS(5) exported by pmcd(1) by default. This PMNS is the super set of all the other PMNS files installed in $PCP_VAR_DIR/pmns ๐ŸŒŸ.

If the PCP product is installed, these files and directories are relevant too ๐ŸŽ‰:

  • $PCP_LOG_DIR/NOTICES โ€” Along with the pmcd(1) and PMDA activity, this may also be used to log alarms and notices from pmie(1) via pmpost(1) ๐Ÿšจ.
  • $PCP_PMLOGGERCONTROL_PATH โ€” The control file for pmlogger(1) instances launched from $PCP_RC_DIR/pmlogger and/or managed by pmlogger_check(1) and pmlogger_daily(1) as part of a production PCP archive collection setup ๐Ÿ“ผ.

๐Ÿ’– ENVIRONMENT ๐Ÿ’–

Besides the PCP run-time environment and configuration variables described in the PCP ENVIRONMENT section below, the following environment variables apply to all installations ๐ŸŒˆ.

Heads up, bestie ๐Ÿซถ: most uses of these variables are optimised to check the environment only the first time the variable might be used. Since the environment usually isn't checked again, the only safe strategy is to make sure every PCP-related environment variable is set before the first call into any of the PCP libraries โฐ.

  • PCP_ALLOW_BAD_CERT_DOMAIN โ€” When set, clients accept certificates with mismatched domain names without prompting, when they're sent by pmcd or other server components. See PCP_SECURE_SOCKETS ๐Ÿ™ˆ.
  • PCP_ALLOW_SERVER_SELF_CERT โ€” When set, clients accept self-signed certificates without prompting, when they're sent by pmcd or other server components. See PCP_SECURE_SOCKETS ๐Ÿ”.
  • PCP_CONSOLE โ€” When set, this changes the default console from /dev/tty (on Unix) or CON: (on Windows) to the console you specify. The special value none means no console is available for use. This is used wherever console-based tools need to interact with the user, and it's especially used when authentication is happening ๐Ÿ’ฌ.
  • PCP_DEBUG โ€” When set, this is an alternative to the -D command line option (described above) for initialising the diagnostic and debug options in applications that use pmGetOptions(3) to process command line options and arguments ๐Ÿ›. The value is the same as for -D: a comma-separated list of debugging option name(s) and/or decimal integers. See pmdbg(1) for a description of the supported option names and values ๐Ÿ“.
  • PCP_DERIVED_CONFIG โ€” When set, this variable defines a colon-separated list of files and/or directories (the syntax is the same as for the $PATH variable for sh(1)) โœจ. The components are expanded into a list of files like so:
    • If a component is a file, that file is added to the list.
    • If a component is a directory, recursive descent is used to enumerate all the files below it, and those get added to the list ๐Ÿ“‚.
    • Each file in the resulting list is assumed to contain derived metric definitions, following the syntax described in pmLoadDerivedConfig(3), and these are loaded in order ๐Ÿงฎ.
    • Derived metrics let you extend the available metrics with new (derived) metrics using simple arithmetic expressions โž•.
    • If PCP_DERIVED_CONFIG is set, the derived metric definitions get processed automatically each time a new source of performance metrics is established (i.e. each time pmNewContext(3) is called), or when requests are made against the PMNS ๐Ÿ”„.
    • Any component in the $PCP_DERIVED_CONFIG list, or in the expanded list of files, that isn't a file, isn't a directory, or isn't accessible (due to permissions or a bad symbolic link) gets silently ignored ๐Ÿซฅ.
  • PCP_DERIVED_CONTROL โ€” Settings for the control options used when processing derived metrics. Check out pmSetDerivedControl(3) for the details ๐ŸŽ›๏ธ.
  • PCP_IGNORE_MARK_RECORDS: okay so, PCP archives can have time gaps where the data jumps, like when pmcd(1) restarts or when you stitch archives together with pmlogextract(1). PCP marks those discontinuities with a <mark> record, and value interpolation (see pmSetMode(3)) is not supported across <mark> records, because the values before and after might not come from one continuous series ๐Ÿ™ˆ Sometimes you know the data is solid around those marks, so $PCP_IGNORE_MARK_RECORDS lets you suppress the default behaviour โœจ
    • If PCP_IGNORE_MARK_RECORDS is set but has no value, then every <mark> record gets ignored.
    • Otherwise, the value follows the same syntax as the interval argument for the -t option (described above), and <mark> records get ignored if the time gap between the last record before the mark and the first record after it is not more than interval.
  • PCP_SECURE_SOCKETS: when this is set, any monitor tool connection has to be made using the certificate-based secure sockets feature ๐Ÿ”’ If a secure connection can't be established, it just fails. No fallback, bestie ๐Ÿ’€
  • PCP_TLSCONF_PATH: this tells everyone where to find the TLS (Transport Layer Security) configuration settings. PCP client tools, pmcd and pmproxy all read these whenever secure (encrypted) communication is requested ๐Ÿฆ‹
  • PCP_STDERR: lots of PCP tools support this one to control where error messages end up ๐Ÿ’…
    • When it's unset, the default is that "usage" messages and option parsing errors go to standard error. Other messages after the initial startup go to the tool's default destination, which is standard error for ASCII tools or a dialog for GUI tools.
    • If it's set to the literal value DISPLAY, then all messages show up in a dialog. This is what gets used for any tools launched from a Desktop environment ๐ŸŒธ
    • If it's set to any other value, that value is treated as a filename, and all messages get written there ๐Ÿ“
  • PMCD_CONNECT_TIMEOUT: connecting to a remote pmcd(1) on a machine that's still booting can hang for a long time until the remote machine finishes initializing ๐Ÿ˜ญ Most PCP apps and some PCP library routines will abort and return an error if the connection isn't established after a set interval. The default is 5 seconds, but you can change it by setting PMCD_CONNECT_TIMEOUT to a real number of seconds. This is most useful when the remote host is at the end of a slow network that needs longer to connect properly ๐Ÿข
  • PMCD_RECONNECT_TIMEOUT: when a monitor or client app loses its connection to a pmcd(1), it can reconnect by calling a service routine in the PCP library. Reconnect attempts follow a back-off strategy so the network doesn't get flooded with requests ๐ŸŒŠ By default, the back-off delays are 5, 10, 20, 40 and 80 seconds for consecutive reconnection requests from a client. The last delay repeats for any further attempts after the fifth. You can redefine them by setting PMCD_RECONNECT_TIMEOUT to a comma separated list of positive integers. For example, setting it to 1,2 backs off for 1 second, then tries another connection request every 2 seconds after that ๐Ÿงš
  • PMCD_REQUEST_TIMEOUT: monitor and client apps connected to pmcd(1) can "hang" while waiting on a request for performance metrics, metadata or help text ๐Ÿ™ˆ These delays can get really bad if the system running pmcd crashes or the network connection drops. Set PMCD_REQUEST_TIMEOUT to a number of seconds, and requests to pmcd will time out after that many seconds. By default, every application is willing to wait 10 seconds for a response from every pmcd โฑ๏ธ
  • PMCD_WAIT_TIMEOUT: when pmcd(1) is started from $PCP_RC_DIR/pmcd, the readiness check waits up to PMCD_WAIT_TIMEOUT seconds. If pmcd has a long startup time, like on a very large system, you can set PMCD_WAIT_TIMEOUT to allow a maximum wait longer than the default 60 seconds ๐Ÿ˜
  • PMNS_DEFAULT: if this is set, it's interpreted as the full pathname to use as the default local PMNS for pmLoadNameSpace(3). Otherwise, the default local PMNS lives at $PCP_VAR_DIR/pcp/pmns/root on base PCP installations ๐Ÿ—บ๏ธ
  • PCP_COUNTER_WRAP: a lot of performance metrics from PCP agents behave like counters, which means they're supposed to keep going up monotonically ๐Ÿ“ˆ Sometimes one value comes back smaller than the last one fetched. That can happen when a finite precision counter overflows, when the PCP agent gets reset or restarted, or when the agent is exporting values from some instrumentation that has asynchronous discontinuities. Setting PCP_COUNTER_WRAP tells PCP to treat every case of a decreasing "counter" as an overflow, so the values are assumed to have wrapped once between consecutive samples. This wrapping behaviour was the default in earlier PCP versions, but it's been disabled by default in PCP releases from version 1.3 on ๐Ÿšซ
  • PCP_PMDAS_DIR: use this to change the directory that pmcd(1) and pmNewContext(3) search when looking for a daemon or DSO PMDA (for PM_CONTEXT_LOCAL contexts) ๐Ÿ“‚
  • PMCD_PORT: the TCP/IP port(s) pmcd(1) uses to create the socket for incoming connections and requests started out as 4321, then moved to the officially registered port 44321 ๐ŸŽ€ In the current release, both port numbers are used by default as a transitional arrangement. You can override this by setting PMCD_PORT to a different port number, or to a comma-separated list of port numbers. If a non-default port is used when pmcd starts, then every monitoring application connecting to that pmcd must also have PMCD_PORT set in its environment before attempting a connection ๐Ÿ”Œ

If you've got the PCP product installed, these next ones matter for the Performance Metrics Domain Agents (PMDAs) ๐Ÿ‘‘

  • PMDA_LOCAL_PROC: this one's been deprecated and is now ignored, so don't count on it ๐Ÿ’€
    • Right now, if the "proc" PMDA is configured as a DSO for use with pmcd(1) on the local host, all of the "proc" metrics are available to applications using a PM_CONTEXT_LOCAL context.
    • The previous behaviour was that if this variable was set, a context created with the type PM_CONTEXT_LOCAL could access the "proc" PMDA to get performance metrics about individual processes.
  • PMDA_LOCAL_SAMPLE: also deprecated and now ignored ๐Ÿฅ€
    • Right now, if the "sample" PMDA is configured as a DSO for use with pmcd(1) on the local host, all of the "sample" metrics are available to applications using a PM_CONTEXT_LOCAL context.
    • The previous behaviour was that if this variable was set, a PM_CONTEXT_LOCAL context could access the "sample" PMDA, as long as this optional PMDA was installed locally.
  • PCP_PMCDCONF_PATH: this overrides the path to the pmcd(1) configuration file ๐Ÿ› ๏ธ When it's set to $PCP_SYSCONF_DIR/local.conf, a PM_CONTEXT_LOCAL context loads only the DSO PMDAs listed in that file, and pmcd(1) doesn't need to be running. This is the recommended way to keep local context limited to a minimal, well-known set of DSO PMDAs. Pair it with PMNS_DEFAULT (below) for a matching namespace ๐Ÿ’œ
  • PMNS_DEFAULT: this overrides the default Performance Metrics Name Space used when no explicit namespace is specified โœจ When it's set to $PCP_VAR_DIR/pmns/local.root, a PM_CONTEXT_LOCAL context uses the merged namespace for the DSO PMDAs listed in $PCP_SYSCONF_DIR/local.conf, instead of the full system namespace. See also pmns(5) and PMNS_DEFAULT in pcpintro(1).

These next variables matter if you're running pmlogger(1), the PCP archiver ๐Ÿ“ผ

  • PMLOGGER_PORT: this changes the base TCP/IP port number pmlogger(1) uses to create the socket that pmlc(1) instances try to connect to ๐Ÿฆ‹ The default base port is 4330. If you use it, set PMLOGGER_PORT in the environment before pmlogger is executed.
  • PMLOGGER_REQUEST_TIMEOUT: there's a slim chance pmlc(1) "hangs" on a request for information when pmlogger(1) connects, because of a network or pmlogger failure ๐Ÿ˜ฌ Set PMLOGGER_REQUEST_TIMEOUT to a number of seconds, and requests to pmlogger will time out after that many seconds. By default, pmlc is willing to wait forever for a response to each request to a pmlogger. If you use it, set PMLOGGER_REQUEST_TIMEOUT in the environment before pmlc is executed โณ
  • PMIECONF_PATH โ€” If set, pmieconf(1) builds its pmieconf(5) specification (a set of parameterized pmie(1) rules) from every valid pmieconf file it finds below each subdirectory in this colon-separated list of subdirectories. If not set, the default is $PCP_VAR_DIR/config/pmieconf. ๐Ÿงš

๐ŸŒธ PCP ENVIRONMENT ๐ŸŒธ

Okay so, environment variables that start with PCP_ are the boss of the file and directory names PCP uses ๐Ÿ’… Every install has its own local values for these in /etc/pcp.conf. Want a different config file? Set the $PCP_CONF variable and it'll swap in, which is covered in pcp.conf(5) ๐ŸŽ€

For the environment variables that affect PCP tools, check out pmGetOptions(3) ๐Ÿ’–

๐Ÿฆ‹ SEE ALSO ๐Ÿฆ‹

Here's the squad, bestie ๐Ÿ‘‘

If the PCP GUI package is installed, you also need these in your life ๐ŸŒˆ

If the secure sockets extensions have been enabled, these references are also giving main character energy ๐Ÿ”

And don't forget the books, periodt ๐Ÿ“š Performance Co-Pilot User's and Administrator's Guide and Performance Co-Pilot Programmer's Guide both live at https://pcp.readthedocs.io/en/latest/ ๐Ÿซถ


Okay that's a wrap, bestie ๐Ÿฆ‹๐Ÿ’œ This page came to you courtesy of Performance Co-Pilot (PCP), and it's the PCPINTRO(1) page, so you know it's iconic ๐ŸŒธโœจ

๐Ÿ“ฆ where this page comes from