LOGARCHIVE(5)

✨ NAME ✨

bestie, meet LOGARCHIVE(5) — aka LOGARCHIVE — the manual page for Performance Co-Pilot archive formats 💖

---

💅 DESCRIPTION 💅

okay so Performance Co-Pilot (PCP) archives? They store historical values about arbitrary metrics recorded from a single host. Archives are machine independent and self-contained — all metric data and metadata required for off-line or off-site analysis is held within an archive. periodt.

The format is stable so long-term historical storage and processing by PMAPI(3) client tools can happen. Some format variants are supported over time, and currently Versions 2 and 3 are supported. The mandate is that PCP will provide long-term backwards compatibility, so an archive created on any version of PCP can be read on that version of PCP and all subsequent versions of PCP. The exception is Version 1 — it was retired in the PCP Version 2.0 release in May 1998. rip.

Archives may be read by most PCP client tools, using the -a/--archive NAME option, or dumped raw by pmlogdump(1). Archives are created primarily by pmlogger(1), however they can also be created using the LOGIMPORT(3) programming interface.

Archives may be merged, analyzed, modified and subsampled using pmlogreduce(1), pmlogsummary(1), pmlogrewrite(1) and pmlogextract(1). In addition, PCP archives may be examined in sets or grouped together into ``archive folios'', which are created and managed by the mkaf(1) and pmafm(1) tools.

An archive consists of several physical files that share a common arbitrary prefix, e.g. myarchive.

• myarchive.0, myarchive.1, ... — One or more data volumes containing the metric values and any error codes encountered during metric sampling. Typically the largest of the files and may grow very rapidly, depending on the selection of metrics to be logged by pmlogger(1) and the sampling intervals being used.
• myarchive.meta — Information for PMAPI functions such as pmLookupName(3), pmLookupDesc(3), pmLookupLabels(3) and pmLookupInDom(3). The metadata file may grow sporadically as logged metrics, instance domains and labels vary over time.
• myarchive.index — A temporal index, mapping timestamps to byte offsets in the other files.

---

🌈 COMMON FEATURES 🌈

All three types of files have a similar record-based structure, a convention of network byte-order (big-endian) encoding, and 32-bit fields for tagging/padding for those records. Strings are stored as 8-bit characters without assuming a specific encoding, so normally ASCII. See also the __pmLog* types in src/include/pcp/libpcp.h.

📦 RECORD FRAMING 📦

The volume and .meta files are divided into self-identifying records. The vibe is:

• Offset 0, Length 4 — N, length of record, in bytes, including this field
• Offset 4, Length N-8 — record payload, usually starting with a 32-bit record type tag
• Offset N-4, Length 4 — N, length of record (again)

🏷️ ARCHIVE LABEL 🏷️

All three types of files begin with an ``archive label'' header, which identifies the host name, starting timestamp and timezone information; all referring to the host that was the source of the performance data (which may be different to the host where pmlogger(1) was running).

The ``archive label'' format differs between Version 2 and Version 3, with the latter providing enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision) and some additional fields.

💾 Version 2 💾

• Offset 0, Length 4 — tag, PM_LOG_MAGIC | PM_LOG_VERS02=0x50052602
• Offset 4, Length 4 — process id (PID) of pmlogger process that wrote file
• Offset 8, Length 4 — archive start time, seconds part (past UNIX epoch)
• Offset 12, Length 4 — archive start time, microseconds part
• Offset 16, Length 4 — current archive volume number (or -1=.meta, -2=.index)
• Offset 20, Length 64 — name of collection host
• Offset 80, Length 40 — time zone string for collection host ($TZ environment variable)

💾 Version 3 💾

• Offset 0, Length 4 — tag, PM_LOG_MAGIC | PM_LOG_VERS03=0x50052603
• Offset 4, Length 4 — PID of pmlogger process that wrote file
• Offset 8, Length 8 — archive start time, seconds part (past UNIX epoch)
• Offset 16, Length 4 — archive start time, nanoseconds part
• Offset 20, Length 4 — current archive volume number (or -1=.meta, -2=.index)
• Offset 24, Length 4 — archive feature bits
• Offset 28, Length 4 — reserved for future use
• Offset 32, Length 256 — name of collection host
• Offset 288, Length 256 — timezone string for collection host ($TZ environment variable), e.g. AEDT-11
• Offset 544, Length 256 — timezone zoneinfo string for collection host, e.g. :Australia/Melbourne

The ``archive feature bits'' are intended to encode possible future extensions or differences to the on-disk structure or the archive semantics. At this stage there are no such features, but if they are introduced at some point in the future, there will be associated PM_LOG_FEATURE_XXX macros added to the <pcp/pmapi.h> header file.

All fields, except for the ``current archive volume number'', match for all files in a single PCP archive.

💖 ARCHIVE VOLUME (.0, .1, ...) RECORDS 💖

pmResult 🌸

Okay bestie so after the archive label record, an archive volume file contains one or more records, and each one is handing you metric values that correspond to the pmResult from one pmFetch(3) operation. The record size may vary according to number of metrics being fetched and the number of instances in the associated instance domains. It's giving flexible, periodt.

For Version 2 the file size is limited to 2GiB, due to storage of 32-bit byte offsets within the temporal index. For Version 3 the file size is limited to 8191PiB, due to storage of 62-bit byte offsets within the temporal index. Massive glow-up, no cap.

The pmResult format differs between Version 2 and Version 3, with the latter providing enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision). Nanoseconds?? She's precise. 💅

Version 2 ✨

• offset 0, length 4 — timestamp, seconds part (past UNIX epoch)
• offset 4, length 4 — timestamp, microseconds part
• offset 8, length 4 — number of metrics with data following
• offset 12, length M — pmValueSet #0
• offset 12+M, length N — pmValueSet #1
• offset 12+M+N, length ... — ...
• offset NOP, length X — pmValueBlock #0
• offset NOP+X, length Y — pmValueBlock #1
• offset NOP+X+Y, length ... — ...

Version 3 ✨

• offset 0, length 8 — timestamp, seconds part (past UNIX epoch)
• offset 8, length 4 — timestamp, nanoseconds part
• offset 12, length 4 — number of metrics with data following
• offset 16, length M — pmValueSet #0
• offset 16+M, length N — pmValueSet #1
• offset 16+M+N, length ... — ...
• offset NOP, length X — pmValueBlock #0
• offset NOP+X, length Y — pmValueBlock #1
• offset NOP+X+Y, length ... — ...

Records with a number of metrics'' equal to zero are mark records'', and represent interruptions, missing data, or time discontinuities in logging. So basically a mark record is the archive being like "brb, nothing to see here" 🫶 and it's literally the plot twist in your timeline.

---

pmValueSet 🦋

This subrecord represents the values for one metric at one point in time. She's the moment, the whole moment, and nothing but the moment. 💫

• offset 0, length 4 — Performance Metrics Identifier (PMID)
• offset 4, length 4 — number of values
• offset 8, length 4 — value format, PM_VAL_INSITU=0 or PM_VAL_DPTR=1
• offset 12, length M — pmValue #0
• offset 12+M, length N — pmValue #1
• offset 12+M+N, length ... — ...

The metadata describing metrics is found in the .meta file where the entries are not timestamped, as the metadata is assumed to be unchanging throughout an archive. Iconic behaviour honestly — the .meta file is the friend who never changes her profile pic. 🙈

---

pmValue 🎀

This subrecord represents one value for one instance of a metric at one point in time. It is a variant type, depending on the parent pmValueSet's value format field. This allows small numbers to be encoded compactly, but retain flexibility for larger or variable length data to be stored later in the pmResult record in a pmValueBlock subrecord. She contains multitudes, lowkey. ✨

• offset 0, length 4 — internal instance identifier (or PM_IN_NULL=-1 for singular metrics)
• offset 4, length 4 — value (INSITU) or / offset in pmResult to our pmValueBlock (DPTR)

The metadata describing the instance domain for metrics is found in the .meta file. Since the numeric mappings may change during the lifetime of the logging session, it is important to match up the timestamp of the measurement record with the corresponding instance domain record. That is, the instance domain corresponding to a measurement at time T is the instance domain observation for the metric's instance domain with largest timestamp T' <= T.

Not me stressing the timestamp matching: get this wrong and your instance names are just wrong, and that's the kind of chaos nobody wants. Match T' <= T, bestie, always. 💀

---

pmValueBlock 🌈

Instances of this subrecord are placed at the end of the pmValueSet, after all the pmValue subrecords. If (and only if) needed, they are padded at the end to the next 32-bit boundary. Alignment queen 👑

• offset 0, length 1 — value type (same as pmDesc.type)
• offset 1, length 3 — 4 + N, the length of the subrecord
• offset 4, length N — bytes that make up the raw value
• offset 4+N, length 0-3 — padding (not included in the 4+N length field)

Note that for PM_TYPE_STRING, the length includes an explicit NULL terminator byte. For PM_TYPE_EVENT, the value byte string is further structured. Refer to PMDAEVENTARRAY(3) for more information about how arrays of event records are packed inside a pmResult container. Nested data?? She's got layers, iykyk. 🧚

---

🌟 METADATA FILE (.meta) RECORDS 🌟

After the archive label record, the metadata file contains interleaved metric description records, timestamped instance domain records, timestamped label records (for context, instance domain and metric labels) and (help) text records. Unlike the data volumes, these records are not forced to 32-bit alignment. Free-spirited. Unaligned. Living her truth. 🦋

For Version 2 the file size is limited to 2GiB, due to storage of 32-bit byte offsets within the temporal index. For Version 3 the file size is limited to 8191PiB, due to storage of 62-bit byte offsets within the temporal index. Same limits, same energy, still huge.

See also libpcp/src/logmeta.c. go read the source bestie, she's right there 💜

💅 Metric Descriptions 💅

Okay so every single pmDesc record is basically the glow-up backstory for ONE metric in the PCP archive ✨. It's where the metric tells you who it really is: its description, its metadata, the whole vibe.

What's in there, bestie:

• the metric's PMID
• the data type
• the data semantics
• the instance domain identifier — or PM_INDOM_NULL for those singular metrics that only ever have one value 🙈
• and a set of names (one or more of them, she contains multitudes)

Here's the layout, field by field, no cap:

• offset 0, length 4 — tag, TYPE_DESC=1
• offset 4, length 4 — PMID
• offset 8, length 4 — data type (PM_TYPE_*)
• offset 12, length 4 — instance domain identifier
• offset 16, length 4 — metric semantics (PM_SEM_*)
• offset 20, length 4 — units: bit-packed pmUnits
• length 4 — number of alternative names for this PMID
• offset 28, length 4 — N: number of bytes in this name
• offset 32, length N — bytes of the name, no NULL terminator nor padding
• offset 32+N, length 4 — N2: number of bytes in next name
• offset 36+N, length N2 — bytes of the name, no NULL terminator nor padding
• ..., ... — ...

---

🌈 Instance Domains 🌈

A set-valued metric isn't just one value, she's a whole squad — and she's defined over an instance domain 🫶. An instance domain is made of:

• an instance domain identifier — which will have already been mentioned in a prior pmDesc record, iykyk
• a count of the number of instances
• a map that defines the association between internal instance identifiers (integers) and external instance names (strings)

The way instance domains can literally just change over time means the instance domain also needs a timestamp, and the same instance domain can show up multiple times inside the .meta file. Those timestamps are the whole point: they let you search for the temporally correct instance domain when you're decoding pmResult records from the archive data volumes, or when you're answering metadata queries against the instance domain. Timing is everything 💅

The instance domain format is dramatically different between Version 2 and Version 3. Version 3 brings enhanced timestamps (64-bit encoding of the seconds part plus nanosecond precision, she's precise ✨) and introduces a brand new ``delta'' instance domain format that encodes the differences between the previous observation of the instance domain and the current state of it.

🎀 Full Instance Domain - Version 2 🎀

• offset 0, length 4 — tag, TYPE_INDOM_V2=2
• offset 4, length 4 — timestamp, seconds part (past UNIX epoch)
• offset 8, length 4 — timestamp, microseconds part
• offset 12, length 4 — instance domain number
• offset 16, length 4 — N: number of instances in domain, normally >0
• offset 20, length 4 — first instance number
• offset 24, length 4 — second instance number (if appropriate)
• ..., ... — ...
• offset 20+4*N, length 4 — first offset into string table (see below)
• offset 20+4*N+4, length 4 — second offset into string table (etc.)
• ..., ... — ...
• offset 20+8*N, length M — base of string table, containing packed, NULL-terminated instance names

🦋 Full Instance Domain - Version 3 🦋

• offset 0, length 4 — tag, TYPE_INDOM=5
• offset 4, length 8 — timestamp, seconds part (past UNIX epoch)
• offset 12, length 4 — timestamp, nanoseconds part
• offset 16, length 4 — instance domain number
• offset 20, length 4 — N: number of instances in domain, normally >0
• offset 24, length 4 — first instance number
• offset 28, length 4 — second instance number (if appropriate)
• ..., ... — ...
• offset 24+4*N, length 4 — first offset into string table (see below)
• offset 24+4*N+4, length 4 — second offset into string table (etc.)
• ..., ... — ...
• offset 24+8*N, length M — base of string table, containing packed, NULL-terminated instance names

Now the delta'' instance domain record in Version 3? She uses the *exact same physical structure* as the full'' instance domain up there 👆 but with these plot twists:

• The tag is TYPE_INDOM_DELTA=6.
• The ``number of instances in domain'' field becomes the sum of the number of instances added and the number of instances deleted.
• Deleted instances are encoded with the string offset set to -1 and there is no corresponding string table entry.
• Added instances are encoded exactly the same way.

So why does the ``delta'' format even exist? It's giving compact 💀 — it gives a more compact on-disk encoding for instance domains that have a large number of instances and are subject to frequent small changes. The example the manpage gives is the instance domain of process ids, as exported by pmdaproc(1).

For ``full'' instance domain records, the instance domain replaces the previous instance domain: prior records are not searched for instance domain metadata queries after this timestamp. Gone, archived, not coming back to that search.

And in a Version 3 archive, each instance domain must have an initial full'' instance domain record — non-negotiable. After that, subsequent records for the same instance domain can be the full'' variant or the delta'' variant, dealer's choice. Also important: any instance mentioned in the prior observation of an instance domain that is *not* mentioned in the delta'' instance domain record is assumed to continue to exist for the current observation of the instance domain. She's still there, just quiet 🤫

---

⭐ Labels for Contexts, Instance Domains and Metrics ⭐

Instances of this record (pmLogLabelSet) provide sets of label-name:label-value pairs associated with labels of the context, instance domains and individual performance metrics — go read pmLookupLabels(3) for the further details, bestie, that's where the lore lives.

Any instance domain identifier will have already been mentioned in a prior pmDesc record.

Because brand new labels can just appear during an archiving session (main character energy, honestly), these records are timestamped and must be searched when decoding pmResult records from the archive data volumes. The pmLogLabelSet format differs between Version 2 and Version 3, with the latter providing enhanced timestamps (64-bit encoding of the seconds part and nanosecond precision). Version 3 stays winning 💅

Version 2 💖

okay bestie, here's the Version 2 record layout — every byte mapped out, no cap:

• Offset 0, Length 4 — tag, TYPE_LABEL_V2=3
• Offset 4, Length 4 — timestamp, seconds part (past UNIX epoch)
• Offset 8, Length 4 — timestamp, microseconds part
• Offset 12, Length 4 — label type (PM_LABEL_* type macros.)
• Offset 16, Length 4 — numeric identifier - domain, PMID, etc or PM_IN_NULL=-1 for context labels
• Offset 20, Length 4 — N: number of label sets in this record, usually 1 except in the case of instances
• Offset 24, Length 4 — offset to the start of the JSONB labels string
• Offset 28, Length L1 — first labelset array entry (see below)
• Offset 28+L1, Length LN — N-th labelset array entry (see below)
• Offset 28+L1+...LN, Length M — concatenated JSONB strings for all labelsets

---

Version 3 ✨

And here's the glow-up for Version 3 — same vibe, new layout:

• Offset 0, Length 4 — tag, TYPE_LABEL=7
• Offset 4, Length 8 — timestamp, seconds part (past UNIX epoch)
• Offset 12, Length 4 — timestamp, nanoseconds part
• Offset 16, Length 4 — label type (PM_LABEL_* type macros.)
• Offset 20, Length 4 — numeric identifier - domain, PMID, etc or PM_IN_NULL=-1 for context labels
• Offset 24, Length 4 — N: number of label sets in this record, usually 1 except in the case of instances
• Offset 28, Length 4 — offset to the start of the JSONB labels string
• Offset 32, Length L1 — first labelset array entry (see below)
• Offset 32+L1, Length LN — N-th labelset array entry (see below)
• Offset 32+L1+...LN, Length M — concatenated JSONB strings for all labelsets

Records of this form replace the existing labels for a given label type: prior records are not searched for resolving that class of label in measurements after this timestamp. periodt.

The individual labelset array entries are variable length, depending on the number of labels present within that set. These entries contain the instance identifiers (in the case of type PM_LABEL_INSTANCES labels), lengths and offsets of each label name and value, and also any flags set for each label. iykyk

• Offset 0, Length 4 — instance identifier (or PM_IN_NULL=-1)
• Offset 4, Length 4 — length of JSONB label string
• Offset 8, Length 4 — N: number of labels in this labelset
• Offset 12, Length 2 — first label name offset
• Offset 14, Length 1 — first label name length
• Offset 15, Length 1 — first label flags (e.g. optionality)
• Offset 16, Length 2 — first label value offset
• Offset 18, Length 2 — first label value length
• Offset 20, Length 2 — second label name offset (if appropriate)
• ..., ... — ...

---

Help Text 🫶

This (pmLogText) record stores help text associated with a metric or an instance domain - as provided by pmLookupText(3) and pmLookupInDomText(3). so helpful, honestly.

The metric identifier and instance domain identifier will have already been mentioned in a prior pmDesc record. like, they're already in the group chat.

• Offset 0, Length 4 — tag, TYPE_TEXT=4
• Offset 4, Length 4 — text and identifier type (PM_TEXT_* macros.)
• Offset 8, Length 4 — numeric identifier - PMID or instance domain
• Offset 12, Length M — help text string, arbitrary text

💾 INDEX FILE (.index) RECORDS 💾

okay bestie so after that archive label record, the temporal index file is just a plainly concatenated, unframed lil group of tuples 💅 — and those tuples are the ones matching up timestamps to the byte offsets over in the volume and .meta files. and here's the thing: these records are fixed size, fixed format, and they are not wrapped in the standard length/payload/length wrapper 🙅‍♀️. nope. they take up the entire remainder of the .index file after the archive label record. whole thing. it's giving "no packaging, just vibes" ✨

so what's the point of this temporal index file? it's the rapid way of seeking to a particular point in time inside an archive — for both the performance metric values and the associated metadata. it's the fast-travel spell of the archive world 🧚

also peep libpcp/src/logutil.c. 👀

now, the index format is different between Version 2 and Version 3 — and V3 is the upgraded glow-up: it's got enhanced timestamps (64-bit encoding of the seconds part, plus nanosecond precision 🌸) and 64-bit byte offsets. slay.

---

🌈 Version 2

• 0 offset, length 4 — timestamp, seconds part (past UNIX epoch)
• 4 offset, length 4 — timestamp, microseconds part
• 8 offset, length 4 — archive volume number (0...N)
• 12 offset, length 4 — byte offset in .meta file
• 16 offset, length 4 — byte offset in archive volume file

---

🦋 Version 3

• 0 offset, length 8 — timestamp, seconds part (past UNIX epoch)
• 8 offset, length 4 — timestamp, nanoseconds part
• 12 offset, length 4 — archive volume number (0...N)
• 16 offset, length 8 — byte offset in .meta file
• 24 offset, length 8 — byte offset in archive volume file

---

now for the tea ☕ the temporal index is optional. it exists only to speed up time-based random access to metrics and their metadata, which means index records get emitted only intermittently — like, whenever it feels like it. so an archive reader program should not presume any particular rate of data flow into the index 🫶 do not assume. do not predict. just vibe.

that said, the common events that may trigger a new temporal index record are:

• changes in instance domains
• switching over to a new archive volume
• starting or stopping logging

and there's one reliable invariant though, and bestie this one matters: for each index entry, there are to be no meta or archive volume records with a timestamp after that in the index, but physically before the associated byte offset in the index. 📌 that's the rule. no cap.

---

📁 FILES 📁

several PCP tools create archives in standard locations, and here they are:

• $HOME/.pcp/pmlogger — default location for the interactive chart recording mode in pmchart(1)
• $PCP_LOG_DIR/pmlogger — default location for the pmlogger_daily(1) and pmlogger_check(1) scripts

---

👀 SEE ALSO 👀

go read these, fr fr, they're the whole friend group 💖

mkaf(1), PCPIntro(1), pmafm(1), pmchart(1), pmdaproc(1), pmlogdump(1), pmlogger(1), pmlogger_check(1), pmlogger_daily(1), pmlogreduce(1), pmlogrewrite(1), pmlogsummary(1), LOGIMPORT(3), PMAPI(3), pmLookupDesc(3), pmLookupInDom(3), pmLookupInDomText(3), pmLookupLabels(3), pmLookupName(3), pmLookupText(3), pcp.conf(5) and pcp.env(5).

---

and that's a wrap on LOGARCHIVE(5) 🎀 you made it to the end, iconic behavior honestly 👑 this came from Performance Co-Pilot, it's the LOGARCHIVE(5) page, and that's the version and date stamped on the bottom of the original. mwah 💋 go forth and read your archives bestie, you've got this 🌟