acct_gather.conf(5)

📦
version 26.05.5.1 · 7 versions
, in arch rolling, epel 10, nixpkgs unstable, ubuntu stonking

✨ NAME ✨

acct_gather.conf — the Slurm config file for the acct_gather plugins 💖 (yes bestie, it's a (5) manual page, we're in config-file territory)

🌸 DESCRIPTION 🌸

okay so acct_gather.conf is a UTF8 formatted file, and it defines the parameters that Slurm's acct_gather-related plugins are gonna use. And the way this works — it always lives in the same directory as the slurm.conf. no exceptions, they're roommates, they're a package deal 🫶

Parameter names are case insensitive, but parameter values are case sensitive. so you can yell ENERGYIPMIFREQUENCY or whisper energyipmifrequency, whatever, but the values? those gotta be exact. lowercase where lowercase belongs.

Anything after a # is a comment, all the way to the end of the line. And heads up: each line in the file is capped at 1024 characters. don't be writing a novel on one line.

Changes to the configuration file take effect upon restart of the Slurm daemons. So like… restart. ✨

The following acct_gather.conf parameters are defined to control the general behavior of the various plugins in Slurm.

Now here's the tea ☕ — acct_gather.conf is different from other Slurm .conf files. Each plugin defines which options are available. And every plugin you want loaded has to be specified in slurm.conf under these config entries:

  • AcctGatherEnergyType (plugin type=acct_gather_energy)
  • AcctGatherInterconnectType (plugin type=acct_gather_interconnect)
  • AcctGatherFilesystemType (plugin type=acct_gather_filesystem)
  • AcctGatherProfileType (plugin type=acct_gather_profile)
BIG WARNING BESTIE — if the respective plugin for an option is not loaded, then that option is unknown to Slurm, and that causes the daemon to fatal on initialization. 💀 And if you decide to change plugin types in slurm.conf, make sure you also change the related options in acct_gather.conf. keep them in sync fr fr.

👑 acct_gather_energy/gpu 👑

Required entry in slurm.conf:

AcctGatherEnergyType=acct_gather_energy/gpu

This plugin doesn't read any options from acct_gather.conf. zero, zilch, nothing.

Dataset provided by the plugin is: Energy.

🧚 acct_gather_energy/IPMI 🧚

Required entry in slurm.conf:

AcctGatherEnergyType=acct_gather_energy/ipmi

Options used for acct_gather_energy/ipmi are as follows:

  • EnergyIPMIFrequency=<number> — this parameter is the number of seconds between BMC access samples. Ideally it should be higher than or equal to JobAcctGatherFrequency, otherwise the JobAcctGather plugin will get repeated values in successive polls. (so like… keep it ≥ bestie, or you're just sampling the same vibe twice 😭)
  • EnergyIPMICalcAdjustment=<yes|no> — if set to "yes", the consumption between the last BMC access sample and a step consumption update is approximated to get more accurate task consumption. The adjustment is made at the step start and each time the consumption is updated, including the step end. The approximations are not accumulated — only the first and last adjustments are used to calculated the consumption. The default is "no".
  • EnergyIPMIPowerSensors=<key=values> — optionally specify the ids of the sensors to used. Multiple <key=values> can be set with ";" separators. The key "Node" is mandatory and is used to know the consumed energy for nodes (scontrol show node) and jobs (sacct). Other keys are optional and are named by administrator. These keys are useful only when profile is activated for energy to store power (in watt) of each key. <values> are integers except when using DCMI. Multiple values can be set with "," separators. The sum of the listed sensors is used for each key. EnergyIPMIPowerSensors is optional, default value is "Node=<value>" where "<value>" is the id of the first power sensor returned by ipmi-sensors.

i.e.

EnergyIPMIPowerSensors=Node=16,19,23,26;Socket0=16,23;Socket1=19,26;SSUP=23,26;KNC=16,19
EnergyIPMIPowerSensors=Node=29,32;SSUP0=29;SSUP1=32
EnergyIPMIPowerSensors=Node=1280

Data Center Manageability Interface — acct_gather_energy/ipmi supports gathering power data through DCMI IPMI extension commands. When configured, the ipmi plugin will query the DCMI using the "System Power mode" or the "Enhanced System Power Statistics mode" flags depending on the configuration.

To configure one or the other, the special sensor values DCMI or DCMI_ENHANCED can be used, for example:

EnergyIPMIPowerSensors=Node=DCMI
EnergyIPMIPowerSensors=Node=DCMI_ENHANCED

The following acct_gather.conf parameters are defined to control the IPMI config default values for libipmiconsole.

  • EnergyIPMIUsername=USERNAME — Specify BMC Username.
  • EnergyIPMIPassword=PASSWORD — Specify BMC Password.

Datasets provided by the plugin have name: <IPMI_SENSOR_LABEL>Power.

📝 NOTES:

This plugin requires the freeipmi development files to be installed and linkable at configure time. The plugin will not build otherwise. When building the RPM, rpmbuild ... --with freeipmi can be specified to explicitly check for these dependencies.

🦋 acct_gather_energy/rapl 🦋

Required entry in slurm.conf:

AcctGatherEnergyType=acct_gather_energy/rapl

This plugin doesn't read any options from acct_gather.conf. nope, nothing here.

Dataset provided by the plugin is: Power.

🌈 acct_gather_energy/XCC 🌈

Required entry in slurm.conf:

AcctGatherEnergyType=acct_gather_energy/xcc

Options used for acct_gather_energy/xcc include only in-band communications with XClarity Controller, thus a reduced set of configurations is supported:

  • EnergyIPMIFrequency=<number> — this parameter is the number of seconds between XCC access samples. Default is 30 seconds.
  • EnergyIPMITimeout=<number> — Timeout, in seconds, for initializing the IPMI XCC context for a new gathering thread. Default is 10 seconds.

Datasets provided by the plugin are: Energy, CurrPower.

🍓 acct_gather_filesystem/lustre 🍓

Required entry in slurm.conf:

AcctGatherFilesystemType=acct_gather_filesystem/lustre

This plugin doesn't read any options from acct_gather.conf. not a single one.

Datasets provided by the plugin are: Reads, ReadMB, Writes, WriteMB.

💜 acct_gather_profile/HDF5 💜

Required entry in slurm.conf:

AcctGatherProfileType=acct_gather_profile/hdf5

Options used for acct_gather_profile/hdf5 are as follows:

  • ProfileHDF5Dir=<path> — this parameter is the path to the shared folder into which the acct_gather_profile plugin will write detailed data (usually as an HDF5 file). The directory is assumed to be on a file system shared by the controller and all compute nodes. This is a required parameter. ⚠️
  • ProfileHDF5Default — a comma-delimited list of data types to be collected for each job submission. Allowed values are:
    • All — All data types are collected. (Cannot be combined with other values.)
    • None — No data types are collected. This is the default. (Cannot be combined with other values.)
    • Energy — Energy data is collected.
    • Filesystem — File system (Lustre) data is collected.
    • Network — Network (InfiniBand) data is collected.
    • Task — Task (I/O, Memory, ...) data is collected.

🌊 acct_gather_profile/InfluxDB

Bestie, this one is the glow-up sibling of HDF5. Same tea, different delivery service fr fr.

Required entry in slurm.conf:

AcctGatherProfileType=acct_gather_profile/influxdb

The InfluxDB plugin hands you the exact same info as the HDF5 plugin, except instead of writing files it beams everything over to your configured InfluxDB server. iconic

It's designed against the 1.x protocol of InfluxDB, so listen up: if your site is running a v2.x InfluxDB server, you're gonna need to configure a v1.x compatibility endpoint along with the correct user and password authorization. Token authentication is not currently supported. No cap, that's just how it is. 🙈

🎀 Options:

  • ProfileInfluxDBDatabase — The InfluxDB v1.x database name where profiling info gets written. (In the v2.x world? This is the bucket name where profiling info gets written.)
  • ProfileInfluxDBDefault — A comma-delimited list of data types to be collected for each job submission. The allowed values are, and yes they're giving menu:
    • All — literally every data type gets collected. Cannot be combined with other values.
    • None — no data types get collected. This is the default. Cannot be combined with other values either.
    • Energy — energy data gets collected.
    • Filesystem — file system (Lustre) data gets collected.
    • Network — network (InfiniBand) data gets collected.
    • Task — task (I/O, Memory, ...) data gets collected.
  • ProfileInfluxDBExtraTags=<tags> — A comma-delimited list of extra tags to attach to every InfluxDB measurement. Heads up: this requires ProfileInfluxDBFlags=grouped_fields. The valid tags are:
    • cluster — the Slurm cluster name.
    • sluid — the SLUID of the job.
    • uid — the user ID of the user who ran the job.
    • user — the user name of the user who ran the job.
  • ProfileInfluxDBFlags=<flags> — A comma-delimited list of flags that tune how the InfluxDB plugin behaves. The valid flags are:
    • grouped_fields — emits all fields from a single sample as one InfluxDB measurement line, with the dataset name (e.g., "task", "network", "filesystem", "energy") serving as the measurement, instead of one measurement per field. Why do we care? Because this reduces InfluxDB series cardinality and kills the ambiguity that happens when identical field names come from different plugins. clarity queen behavior ✨
  • ProfileInfluxDBFrequency=<seconds> — How often, in seconds, data should be sent to the InfluxDB server. Lowkey important detail: in practice, profile data gets sent no more often than every JobAcctGatherFrequency seconds. A value of 0 disables buffering in the plugin, so data is sent to the InfluxDB server as soon as it's collected. In all cases, data is sent to InfluxDB whenever the plugin's internal buffer is full and also at the end of each job step. Default is 30 seconds.
  • ProfileInfluxDBHost=<hostname>:<port> — The hostname of the machine where the InfluxDB instance is running, plus the port used by the HTTP API. That port is the one configured through the bind-address influxdb.conf option in the [http] section. Example time:
ProfileInfluxDBHost=myinfluxhost:8086
  • ProfileInfluxDBPass — Password for the username configured in ProfileInfluxDBUser. Required in v2.x and optional in v1.x InfluxDB. 🔐
  • ProfileInfluxDBRTPolicy — The InfluxDB v1.x retention policy name for the database configured in the ProfileInfluxDBDatabase option. If you leave it unset, InfluxDB writes to the default retention policy.

For InfluxDB v2.x, this value gets passed on the v1-compatibility /write endpoint and, together with ProfileInfluxDBDatabase, selects a DBRP mapping that resolves to a v2 bucket. Recent v2.x releases auto-create a default DBRP mapping per bucket, so leaving this option unset is usually enough. convenient 💅

  • ProfileInfluxDBUser — The InfluxDB username that should be used to gain access to the database configured in ProfileInfluxDBDatabase. Required in v2.x and optional in v1.x InfluxDB. This is only needed if InfluxDB v1.x is configured with authentication enabled in the [http] config section and a user has been granted at least WRITE access to the database. See also ProfileInfluxDBPass.
  • ProfileInfluxDBTimeout=<seconds> — The maximum time in seconds that an HTTP query to the InfluxDB server can take. After this timeout hits, the data is discarded. ⚠️ Be aware: a long timeout can drain your nodes if the InfluxDB server is unresponsive and, when terminating the job, the last dataset takes more than UnkillableStepTimeout to be sent. Internally that option sets the CURLOPT_TIMEOUT library option. Default is 10 seconds.
⚠️ warning bestie — that ProfileInfluxDBTimeout and the silently-discarded writes below are real. Data loss is not a vibe.

📝 NOTES:

This plugin requires the libcurl development files to be installed and linkable at configure time. If they're not? The plugin will not build. Periodt. 💀

Info on how to install and configure InfluxDB and manage databases, retention policies and such lives on the official webpage.

Collected information is written from every compute node where a job runs to the InfluxDB instance listening on the ProfileInfluxDBHost. To avoid overloading that InfluxDB instance with a million incoming connection requests, the plugin uses an internal buffer that fills up with samples. Once the buffer is full, an HTTP API write request goes out and the buffer is emptied to hold subsequent samples. A final request also goes out when a task ends, even if the buffer isn't full. 🦋

Failed HTTP API write requests are silently discarded. Translate that for me: collected profile information sitting in the plugin buffer is lost if it can't be written to the InfluxDB database for any reason. gone. poof. 😭

Plugin messages get logged along with the slurmstepd logs to SlurmdLogFile. For troubleshooting, the move is to temporarily bump the slurmd debug level to debug3 and add Profile to the debug flags. You can do this by setting the slurm.conf SlurmdDebug and DebugFlags respectively, or dynamically through scontrol setdebug and setdebugflags. 🕵️♀️

Grafana can be used to create charts based on the data held by InfluxDB. That kind of tool lets you create dashboards, tables and other graphics using the stored time series. aesthetic data visualization moment 📊

🛰️ acct_gather_interconnect/OFED

Required entry in slurm.conf:

AcctGatherInterconnectType=acct_gather_interconnect/ofed

Options used for acct_gather_interconnect/ofed are as follows:

  • InfinibandOFEDPort=<number> — This parameter represents the port number of the local Infiniband card that we're willing to monitor. The default port is 1.

Datasets provided by the plugin: PacketsIn, PacketsOut, InMB, OutMB 📦

🧵 acct_gather_interconnect/sysfs

Required entry in slurm.conf:

AcctGatherInterconnectType=acct_gather_interconnect/sysfs

Options used for acct_gather_interconnect/sysfs are as follows:

  • SysfsInterfaces=<interfaces> — Comma-separated list of interface names to collect statistics from. Usage from all listed interfaces will be summed together, and is not broken down individually. one big smoothie, not separate cups 🍓

Datasets provided by the plugin: PacketsIn, PacketsOut, InMB, OutMB 📦

🌈 EXAMPLE

###
# Slurm acct_gather configuration file
###
# Parameters for acct_gather_energy/impi plugin
EnergyIPMIFrequency=10
EnergyIPMICalcAdjustment=yes
#
# Parameters for acct_gather_profile/hdf5 plugin
ProfileHDF5Dir=/app/slurm/profile_data
# Parameters for acct_gather_interconnect/ofed plugin
InfinibandOFEDPort=1

👑 COPYING

Copyright (C) 2012-2013 Bull. Copyright (C) 2012-2022 SchedMD LLC. Produced at Bull (cf, DISCLAIMER).

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

🫶 SEE ALSO

And that's the whole page, bestie — acct_gather.conf(5) from the Slurm 26.05 era, filed under Slurm Configuration File, brought to you with love by the slurm.conf(5) cinematic universe. Go configure something cute. 💖✨🎀

📦 where this page comes from