HB_REPORT(8)

📦
version 1.0.12 · 7 versions
, in ubuntu stonking

okay bestie, deep breath, we're doing the hb_report(8) page 💖 it's from the Pacemaker documentation and it's a big one, so this is part one. pull up a chair, grab a strawberry 🍓, and let's get into it.

🌸 NAME 🌸

hb_report — create report for CRM based clusters (Pacemaker)

like, it's literally the "gather everything, we need receipts" tool 🕵️♀️

✨ SYNOPSIS ✨

hb_report -f {time|"cts:"testnum} [-t time] [-u user] [-l file] [-n
nodes] [-E files] [-p patt] [-L patt] [-e prog] [-MSDCZAQVsvhd] [dest]

🦋 DESCRIPTION 🦋

so hb_report(1) is the utility that collects all the information — logs, configuration files, system info, the whole vibe ✨ — that's relevant to Pacemaker (CRM) over whatever period of time you tell it. it's giving "detective, but make it tarball" 🔥

👑 OPTIONS 👑

every single one, no skips, periodt 💅

  • dest — the report name. it can also contain a path where to put the report tarball. if you leave it out, the tarball gets created in the current directory and it's named "hb_report-current_date", for instance hb_report-Wed-03-Mar-2010.
  • -d — don't build the compressed tar, just leave the result sitting there in a directory. no zip, just vibes 📁
  • -f { time | "cts:"testnum } — the start time from which to collect logs. the time is in the format used by the Date::Parse perl module. for cts tests, say the "cts:" string followed by the test number. this option is required — no cap, you cannot skip it.
  • -t time — the end time to which to collect logs. defaults to now ⏰
  • -n nodes — a list of space separated hostnames (cluster members). hb_report may try to find out the set of nodes by itself, but if it's running on the loghost — which, as it usually is the case, does not belong to the cluster — that can be difficult. also, OpenAIS doesn't contain a list of nodes, and if Pacemaker isn't running, there's just no way to find it out automatically. this option is cumulative (i.e. use -n "a b" or -n a -n b).
  • -l file — log file location. if, for whatever reason, hb_report can't find the log files, you can specify its absolute path 🗂️
  • -E files — extra log files to collect. this option is cumulative. by default, /var/log/messages are collected along with the cluster logs.
  • -M — don't collect extra log files, only the file containing messages from the cluster subsystems. main character log only, we're not doing side quests 🙅♀️
  • -L patt — a list of regular expressions to match in log files for analysis. this option is additive (default: "CRIT: ERROR:").
  • -p patt — additional patterns to match parameter names which contain sensitive information. this option is additive (default: "passw.*"). we're redacting, not spilling 🔒
  • -Q — Quick run. gathering some system information can be expensive. with this option, such operations are skipped and thus information collecting is sped up. the operations considered I/O or CPU intensive: verifying installed packages content, sanitizing files for sensitive information, and producing dot files from PE inputs. ⚡
  • -A — this is an OpenAIS cluster. hb_report has some heuristics to find the cluster stack, but that's not always reliable. by default, hb_report assumes it is run on a Heartbeat cluster.
  • -u user — the ssh user. hb_report will try to login to other nodes without specifying a user, then as "root", and finally as "hacluster". if you have another user for administration over ssh, please use this option.
  • -X ssh-options — extra ssh options. these get added to every ssh invocation. alternatively, use $HOME/.ssh/config to setup the desired ssh connection options.
  • -S — single node operation. run hb_report only on this node and don't try to start slave collectors on other members of the cluster. under normal circumstances this option is not needed. use it if ssh(1) doesn't work to other nodes. 🏠
  • -Z — if the destination directory exists, remove it instead of exiting (this is default for CTS). careful, bestie, it deletes things 💀
  • -V — print the version, including the last repository changeset.
  • -v — increase verbosity. normally used to debug unexpected behaviour. 🐛
  • -h — show usage and some examples.
  • -D (obsolete) — don't invoke the editor to fill the description text file.
  • -e prog (obsolete) — your favourite text editor. defaults to $EDITOR, vim, vi, emacs, or nano, whichever is found first.
  • -C (obsolete) — remove the destination directory once the report has been put in a tarball.

🎀 EXAMPLES 🎀

example the first: the overnight backup saga

last night during the backup there were several warnings encountered (logserver is the log host):

logserver# hb_report -f 3:00 -t 4:00 -n "node1 node2" report

this collects everything from all nodes from 3am to 4am last night. the files are compressed to a tarball report.tar.bz2. ate, no crumbs 🔥

example the second: just found a problem during testing

# note the current time
node1# date
Fri Sep 11 18:51:40 CEST 2009
node1# /etc/init.d/heartbeat start
node1# nasty-command-that-breaks-things
node1# sleep 120 #wait for the cluster to settle
node1# hb_report -f 18:51 hb1

# if hb_report can't figure out that this is corosync
node1# hb_report -f 18:51 -A hb1

# if hb_report can't figure out the cluster members
node1# hb_report -f 18:51 -n "node1 node2" hb1

the files are compressed to a tarball hb1.tar.bz2 ✨

🔥 INTERPRETING RESULTS 🔥

the compressed tar archive is the final product of hb_report. here's one example of its content, for a CTS test case on a three node OpenAIS cluster:

$ ls -RF 001-Restart

001-Restart:
analysis.txt     events.txt  logd.cf       s390vm13/  s390vm16/
description.txt  ha-log.txt  openais.conf  s390vm14/

001-Restart/s390vm13:
STOPPED  crm_verify.txt  hb_uuid.txt  openais.conf@   sysinfo.txt
cib.txt  dlm_dump.txt    logd.cf@     pengine/        sysstats.txt
cib.xml  events.txt      messages     permissions.txt

001-Restart/s390vm13/pengine:
pe-input-738.bz2  pe-input-740.bz2  pe-warn-450.bz2
pe-input-739.bz2  pe-warn-449.bz2   pe-warn-451.bz2

001-Restart/s390vm14:
STOPPED  crm_verify.txt  hb_uuid.txt  openais.conf@   sysstats.txt
cib.txt  dlm_dump.txt    logd.cf@     permissions.txt
cib.xml  events.txt      messages     sysinfo.txt

001-Restart/s390vm16:
STOPPED  crm_verify.txt  hb_uuid.txt  messages        sysinfo.txt
cib.txt  dlm_dump.txt    hostcache    openais.conf@   sysstats.txt
cib.xml  events.txt      logd.cf@     permissions.txt

okay so the top directory holds information that pertains to the cluster or event as a whole. files with exactly the same content on all nodes also live at the top, with per-node links created (as it is in this example the case with openais.conf and logd.cf). 🔗

the cluster log files are named ha-log.txt regardless of the actual log file name on the system. if it's found on the loghost, then it's placed in the top directory. if not, the top directory ha-log.txt contains all nodes' logs merged and sorted by time. files named messages are excerpts of /var/log/messages from nodes.

most files are copied verbatim, or they contain the output of a command. for instance, cib.xml is a copy of the CIB found in /var/lib/heartbeat/crm/cib.xml. crm_verify.txt is the output of the crm_verify(8) program.

some files are the result of a more involved processing:

  • analysis.txt — a set of log messages matching user defined patterns (may be provided with the -L option).
  • events.txt — a set of log messages matching event patterns. it should provide information about major cluster motions without unnecessary details. these patterns are devised by the cluster experts. currently, the patterns cover membership and quorum changes, resource starts and stops, fencing (stonith) actions, and cluster starts and stops. events.txt is always generated for each node. in case the central cluster log was found, also combined for all nodes. 👀
  • permissions.txt — one of the more common problem causes are file and directory permissions. hb_report looks for a set of predefined directories and checks their permissions. any issues are reported here. 🚨
  • backtraces.txt — gdb generated backtrace information for cores dumped within the specified period.
  • sysinfo.txt — various release information about the platform, kernel, operating system, packages, and anything else deemed to be relevant. the static part of the system.
  • sysstats.txt — output of various system commands such as ps(1), uptime(1), netstat(8), and ifconfig(8). the dynamic part of the system.

description.txt should contain a user supplied description of the problem, but since it is very seldom used, it will be dropped from the future releases. rip description.txt, you were iconic 💔

🫶 PREREQUISITES 🫶

🔐 ssh

  • it is not strictly required, but you won't regret having a password-less ssh. it's not too difficult to setup and will save you a lot of time. if you can't have it — for example because your security policy doesn't allow such a thing, or you just prefer menial work — then you'll have to resort to the semi-manual semi-automated report generation. see below for instructions.
  • if you need to supply a password for your passphrase/login, then always use the -u option.
  • for extra ssh(1) options, if you're too lazy to setup $HOME/.ssh/config, use the -X option. do not forget to put the options in quotes. 🫶

🧑⚖️ sudo

if the ssh user (as specified with the -u option) is other than root, then hb_report uses sudo to collect the information which is readable only by the root user. in that case it is required to setup the sudoers file properly. the user (or group to which the user belongs) should have the following line:

<user> ALL = NOPASSWD: /usr/sbin/hb_report

see the sudoers(5) man page for more details. 👑

⏰ Times

in order to find files and messages in the given period, and to parse the -f and -t options, hb_report uses perl and one of the Date::Parse or Date::Manip perl modules. note that you need only one of these. furthermore, on nodes which have no logs and where you don't run hb_report directly, no date parsing is necessary. in other words, if you run this on a loghost then you don't need these perl modules on the cluster nodes.

on rpm based distributions, you can find Date::Parse in perl-TimeDate, and on Debian and its derivatives in libtimedate-perl. 📦

💀 Core dumps

to backtrace core dumps, gdb is needed, plus the packages with the debugging info. the debug info packages may be installed at the time the report is created. let's hope that you will need this really seldom. 🍀

⏰ TIMES ⏰

okay so typing out times is lowkey the most annoying part of anything, ever. that's literally why they picked a perl module to do it — those things are so chill about dates. you can either go read the instructions on the Date::Parse examples page at http://search.cpan.org/dist/TimeDate/lib/Date/Parse.pm#EXAMPLE_DATES, or just trust your gut and try stuff like this:

3:00          (today at 3am)
15:00         (today at 3pm)
2007/9/1 2pm  (September 1st at 2pm)
Tue Sep 15 20:46:27 CEST 2009 (September 15th etc)

hb_report will (probably) complain if it can't figure out what you mean, bestie. it's not psychic, it's a script 💀

try to delimit the event as close as possible so your report doesn't balloon into a whole saga, but still leave a minute or two around it for good measure. just in case.

-f is not optional. periodt. and do NOT forget to quote your dates when they have spaces in them, or the shell will absolutely eat them for breakfast 🙈

🔒 SHOULD I SEND ALL THIS TO THE REST OF INTERNET? 🔒

so. by default, the sensitive data in CIB and PE files is not mangled by hb_report, because mangling it makes PE input files mostly useless. like, what's the point of a file with all the juicy parts blacked out, you know?

but if you truly have no other option and you gotta send the report to a public mailing list, and you do not want the sensitive data riding along, use the -s option. without -s, hb_report will issue a warning if it finds information that should not be exposed — think of it as your friend going "girl, are you SURE about this" 💅

by default, parameters matching passw.* are considered sensitive. use the -p option to specify additional regular expressions to match variable names which may contain information you don't want to leak. for example:

# hb_report -f 18:00 -p "user.*" -p "secret.*" /var/tmp/report

Heartbeat's ha.cf is always sanitized. logs and other files are not filtered. sorry, that's just how it is 😭

📜 LOGS 📜

ngl, finding syslog logs can be tricky. the scheme they use is: log a unique message on all nodes, then go look it up in the usual syslog locations. this procedure is not foolproof — especially if your syslog files live in some non-standard directory (main character behavior, honestly).

they look in /var/log /var/logs /var/syslog /var/adm /var/log/ha /var/log/cluster. and if the logs still can't be found, please just supply their location yourself:

# hb_report -f 5pm -l /var/log/cluster1/ha-log -S /tmp/report_node1

if you have different log locations on different nodes — well, perhaps you'd like to make them the same and make life easier for everybody? ✨ just a thought 🫶

files starting with "ha-" are preferred. and in case syslog sends messages to more than one file, if one of them is named ha-log or ha-debug, those will be favoured over syslog or messages. they're the chosen ones, basically 👑

hb_report also supports archived logs in case the period you specified extends that far back in the past. the archives must reside in the same directory as the current log, and their names must be prefixed with the name of the current log (syslog-1.gz or messages-20090105.bz2).

if there's no separate log for the cluster, possibly unrelated messages from other programs get included too. they don't filter logs — they just pick a segment for the period you specified. simple as that 💖

🛠️ MANUAL REPORT COLLECTION 🛠️

so your ssh doesn't work. it happens to the best of us 😭 in that case, you'll have to run this procedure on all the nodes. use -S so that hb_report doesn't even bother with ssh:

# hb_report -f 5:20pm -t 5:30pm -S /tmp/report_node1

and if you also have a log host which is not in the cluster, then you'll have to copy the log to one of the nodes and tell us where it is:

# hb_report -f 5:20pm -t 5:30pm -l /var/tmp/ha-log -S /tmp/report_node1

easy! look at you, doing it manually, iconic 💅

⚙️ OPERATION ⚙️

hb_report collects files and other information in a fairly straightforward way. the most complex tasks are discovering the log file locations (if syslog is used, which is the most common case) and coordinating the operation on multiple nodes. that's the drama of it all 🌈

🎀 the cast 🎀

the instance of hb_report running on the host where it was invoked is the master instance. instances running on other nodes are slave instances. the master instance communicates with slave instances by ssh.

and there are multiple ssh invocations per run, so it is essential that the ssh works without password — i.e. with the public key authentication and authorized_keys. no cap, if your ssh is asking for a password mid-run, everything falls apart 💀

🌸 the three phases 🌸

the operation consists of three phases. each phase must finish on all nodes before the next one can commence. no cutting in line, bestie ✋

phase one

consists of logging unique messages through syslog on all nodes. this is the shortest of all phases. a lil appetizer 🍓

phase two

this is the most involved. during this phase all local information is collected, which includes:

  • logs (both current and archived if the start time is far in the past)
  • various configuration files (corosync, heartbeat, logd)
  • the CIB (both as xml and as represented by the crm shell)
  • pengine inputs (if this node was the DC at any point in time over the given period)
  • system information and status
  • package information and status
  • dlm lock information
  • backtraces (if there were core dumps)

phase three

this one is collecting information from all nodes and analyzing it. the analyzis consists of the following tasks:

  • identify files equal on all nodes which may then be moved to the top directory
  • save log messages matching user defined patterns (defaults to ERRORs and CRITical conditions)
  • report if there were coredumps and by whom
  • report crm_verify(8) results
  • save log messages matching major events to events.txt
  • in case logging is configured without loghost, node logs and events files are combined using a perl utility

💀 BUGS 💀

finding logs may at times be extremely difficult, depending on how weird the syslog configuration is. it would be nice to ask the syslog-ng developers to provide a way to find out the log destination based on facility and priority. like... that would slay, actually 🧚

if you think you found a bug, please rerun with the -v option and attach the output to bugzilla. receipts, bestie, we need receipts 🔥

hb_report can function in a satisfactory way only if ssh works to all nodes using authorized_keys (without password). this is not negotiable, it's the whole foundation 🙈

there are way too many options. yeah. they said it. it's canon 😭

✍️ AUTHOR ✍️

Written by Dejan Muhamedagic, dejan@suse.de

🌟 RESOURCES 🌟

👀 SEE ALSO 👀

Date::Parse(3)

📄 COPYING 📄

Copyright (C) 2007-2009 Dejan Muhamedagic. Free use of this software is granted under the terms of the GNU General Public License (GPL).

okay so that's literally the whole girlie 🌈 this was HB_REPORT(8) from the hb_report source, dated 2026-08-10 — go forth, collect your reports, and may your ssh never ask you for a password 💖✨🎀 💅

📦 where this page comes from