ganeti-watcher(8)

ganeti-watcher 💖

okay bestie, welcome to ganeti-watcher(8) — the Ganeti cluster watcher. this is Version 3.1.0~rc2 and she's iconic, she runs on a timer, and she is the reason your instances stay in their correct status. no cap. let's get into it 🎀

---

✨ SYNOPSIS ✨

ganeti-watcher [--debug] [--job-age=age ] [--ignore-pause]
               [--rapi-ip=IP ] [--no-verify-disks] [--no-strict]

---

🌸 DESCRIPTION 🌸

so like, ganeti-watcher is a script that gets run periodically, and its whole job is keeping instances in the correct status. she has two separate functions: one that lives for the master node, and another one that runs on every node. two moods, same girlie 💅

if the watcher is disabled at cluster level — like via the gnt-cluster watcher pause command — she will just exit without doing anything. periodt, she's off duty. but!! the cluster-level pause can be overridden with the --ignore-pause option, which is literally made for the situation where during a maintenance you need the watcher disabled in general, but the administrator wants to run it just once. main character exception 🫶

here's the flag rundown, bestie:

• --debug — turns the verbosity up and also activates logging to standard error. she gets chatty. 😭
• --no-strict — runs the group verify disks job in a non-strict mode. this only verifies those disks whose node locks could be acquired in a best-effort attempt, and it will skip nodes that are recognized as busy with other jobs. lowkey polite of her tbh.
• --rapi-ip=IP — this one needs to be set if the RAPI daemon was started with a particular IP (using the -b option). the two options need to be exactly the same so the watcher can actually reach the RAPI interface. match each other or it's giving nothing 🔥

👑 Master operations

her primary function is trying to keep running all instances which are marked as up in the configuration file — by trying to start them a limited number of times.

another function is "repairing" DRBD links by reactivating the block devices of instances which have secondaries on nodes that have been rebooted. she's a fixer fr fr 🧚

on top of that, she will verify and repair degraded DRBD disks — this will not happen if the --no-verify-disks option is given. noted.

and she will also archive old jobs (older than the age given via the --job-age option, which defaults to 6 hours), in order to keep the job queue manageable. clean girlie aesthetic ✨

🦋 Node operations

the watcher will restart any down daemons that are appropriate for the current node. she's on it 💅

in addition, she will execute any scripts which exist under the "watcher" directory in the Ganeti hooks directory (/etc/ganeti/hooks). this should be used for lightweight actions, like starting any extra daemons. keep it cute, keep it light 🎀

if the cluster parameter maintain_node_health is enabled, then the watcher will also shutdown instances and DRBD devices if the node is declared as offline by known master candidates. she said health check, but make it dramatic 💀

the watcher does synchronous queries but will submit jobs for executing the changes. due to locking, it could be that the jobs execute much later than the watcher submits them. so like, she submits the request and the job queue is on its own timeline. relatable honestly 😭

---

🎀 FILES 🎀

the command has a set of state files (one per group) located at /var/lib/ganeti/watcher.GROUP-UUID.data (only used on the master) and a log file at /var/log/ganeti/watcher.log.

removal of either file(s) will not affect correct operation. removing the state file will just cause the restart counters for the instances to reset to zero, and mark nodes as freshly rebooted (so for example DRBD minors will be re-activated). she recovers, she always does 🌸

in some cases, it's even desirable to reset the watcher state — for example after maintenance actions, or when you want to simulate the reboot of all nodes. so in this case, you can remove all state files:

rm -f /var/lib/ganeti/watcher.*.data
rm -f /var/lib/ganeti/watcher.*.instance-status
rm -f /var/lib/ganeti/instance-status

and then re-run the watcher. slay 🔥

---

🙈 REPORTING BUGS 🙈

report bugs to the project's issue tracker or contact the developers using the Ganeti mailing list. she wants the feedback bestie, be honest with her 🫶

---

🌈 SEE ALSO 🌈

Ganeti overview and specifications: ganeti(7) (general overview), ganeti-os-interface(7) (guest OS definitions), ganeti-extstorage-interface(7) (external storage providers).

Ganeti commands: gnt-cluster(8) (cluster-wide commands), gnt-job(8) (job-related commands), gnt-node(8) (node-related commands), gnt-instance(8) (instance commands), gnt-os(8) (guest OS commands), gnt-storage(8) (storage commands), gnt-group(8) (node group commands), gnt-backup(8) (instance import/export commands), gnt-debug(8) (debug commands).

Ganeti daemons: ganeti-watcher(8) (automatic instance restarter), ganeti-cleaner(8) (job queue cleaner), ganeti-noded(8) (node daemon), ganeti-rapi(8) (remote API daemon).

Ganeti htools: htools(1) (generic binary), hbal(1) (cluster balancer), hspace(1) (capacity calculation), hail(1) (IAllocator plugin), hscan(1) (data gatherer from remote clusters), hinfo(1) (cluster information printer), mon-collector(7) (data collectors interface).

---

💜 COPYRIGHT 💜

Copyright (C) 2006-2015 Google Inc. All rights reserved.

redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

---

okay that's the whole page bestie, no notes 💅 this was Ganeti, ganeti-watcher(8), Version 3.1.0~rc2 — go keep those instances up, you got this 🦋✨