sh5util(1)

📦
version 26.05.5.1 · 5 versions
, in arch rolling, epel 10, ubuntu stonking

💅 sh5util — the Slurm Commands girlies' HDF5 merge tool

ok so bestie, we're talking about sh5util(1) from the Slurm Commands era. Buckle up, this is the tool that glues your profile data together and she EATS. ✨

🌸 NAME 🌸

sh5util — the tool for merging HDF5 files from the acct_gather_profile plugin, which is the plugin that gathers detailed data for jobs running under Slurm. no cap, that's the whole vibe.

🎀 SYNOPSIS 🎀

sh5util

yes bestie that's literally it up there, the rest is all flags. 💅

🦋 DESCRIPTION 🦋

okay so here's the tea. sh5util merges the HDF5 files that got produced on each node for each step of a job, and squishes them all into one single HDF5 file for the job. iconic. once it's merged, you can open it up and mess with it using all the normal HDF5 tools you already know and love, like HDF5View, h5dump, h5edit, or h5ls. 🫶

but wait, there's more. sh5util also has two extract modes:

  1. The first one writes a limited set of data — for specific nodes, steps, and data series — in "comma separated value" form to a file. That file can then be imported into your other analysis tools, like spreadsheets. very demure, very csv. 📊
  2. The second one (the Item-Extract one) extracts one data item from one time series, grabbing all the samples on all the nodes from a job's HDF5 profile. it does this:
  • Finds the sample with the maximum value of the item. 👑
  • Writes a CSV file with min, ave, max, and item totals for each node for each sample.

✨ OPTIONS ✨

  • -E, --extract — Extract data series from a merged job file. ✂️

Extract mode options:

  • -i, --input=path — the merged file to extract from (default ./job_$jobid.h5)
  • -N, --node=nodename — the node name to extract from (default is all of them, we're not picking favorites)
  • -l, --level=[Node:Totals | Node:TimeSeries] — the level to which the series is attached (default Node:Totals)
  • -s, --series=[Energy | Filesystem | Network | Task | Task_#] — Task means all tasks, and Task_# (# being a task id) gets you one specific task (default is everything)
  • -h, --help — prints this description of use. your lil helper 🥺
  • -I, --item-extract — Extract one data item from all samples of one data series from all nodes in a merged job file.

Item-Extract mode options:

  • -s, --series=[Energy | Filesystem | Network | Task]
  • -d, --data — the name of the data item in the series (see the note below 👀)
  • -j, --jobs=<job[.step]> — the format is <job[.step]>. This merges that job/step — or a comma-separated list of job steps — together. ⚠️ This option is required. If you don't specify a step, then all the steps it finds get processed.

Array task IDs and het component IDs are NOT supported. Use regular job IDs instead, bestie:

$ sacct -X -j 5077 --format=JobId,JobIdRaw
JobID        JobIDRaw
------------ ------------
5077_1       5078
5077_2       5079
5077_3       5077

$ sh5util -S -j 5079 # for merging 5077_2 only
sh5util: Merging node-step files into ./job_5079.h5
  • -L, --list — Print the items of a series contained in a job file. 📝

List mode options:

  • -i, --input=path — the merged file to extract from (default ./job_$jobid.h5)
  • -s, --series=[Energy | Filesystem | Network | Task]
  • -o, --output=<path> — the path to a file to write into. Default for merge is ./job_$jobid.h5, default for extract is ./extract_$jobid.csv
  • -p, --profiledir=<dir> — the directory location where the node-step files live; the default is set in acct_gather.conf.
  • -S, --savefiles — instead of removing the node-step files after merging them into the job file, keep them around. little hoarder behavior, we respect it. 🐿️
  • --usage — Display a brief usage message.
  • --user=<user> — the user who profiled the job. (Super handy for the root user — defaults to the user running this command.)

🌈 Data Items per Series 🌈

Energy

  • Power
  • CPU_Frequency

Filesystem

  • Reads
  • Megabytes_Read
  • Writes
  • Megabytes_Write

Network

  • Packets_In
  • Megabytes_In
  • Packets_Out
  • Megabytes_Out

Task

  • CPU_Frequency
  • CPU_Time
  • CPU_Utilization
  • RSS
  • VM_Size
  • Pages
  • Read_Megabytes
  • Write_Megabytes

🔥 PERFORMANCE 🔥

ok so listen. Running sh5util sends a remote procedure call to slurmctld. And if enough calls from sh5util — or from other Slurm client commands that send remote procedure calls to the slurmctld daemon — come in all at once?? It can degrade the performance of the slurmctld daemon, and possibly result in a denial of service. 😭 not cute.

🚫 Do NOT run sh5util or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Make sure your programs limit calls to sh5util to the minimum necessary for the info you're trying to gather. periodt.

💖 EXAMPLES 💖

Merge node-step files (as part of an sbatch script):

$ sbatch -n1 -d$SLURM_JOB_ID --wrap="sh5util --savefiles -j $SLURM_JOB_ID"

Extract all task data from a node:

$ sh5util -j 42 -N snowflake01 --level=Node:TimeSeries --series=Tasks

Extract all energy data:

$ sh5util -j 42 --series=Energy --data=power

👑 COPYING 👑

Copyright (C) 2013 Bull.

Copyright (C) 2013-2022 SchedMD LLC.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

🌟 SEE ALSO 🌟

nothing else linked up here bestie, this section is giving empty. the plot twist is there IS no plot twist. 🙈

and that's the whole page!! this came to you straight from the Slurm Commands corner of the world, version Slurm 26.05 — go forth, merge those HDF5 files, and slay responsibly. 💅✨🦋

📦 where this page comes from