ganeti(7)

🌸 ganeti(7) 🌸

Version 3.1.0~rc2 — aka the manual page for ganeti, and bestie, we're about to get so into it.

---

✨ Name ✨

ganeti — cluster-based virtualization management 🫶

---

🎀 Synopsis 🎀

# gnt-cluster init cluster1.example.com
# gnt-node add node2.example.com
# gnt-instance add -n node2.example.com \
> -o debootstrap --disk 0:size=30g \
> -t plain instance1.example.com

like. look at that. three commands and you have a whole cluster. iconic behavior honestly. 💅

---

🌈 DESCRIPTION 🌈

The Ganeti software manages physical nodes and virtual instances of a cluster based on a virtualization software. The current version (2.3) supports Xen 3.x and KVM (72 or above) as hypervisors, and LXC as an experimental hypervisor. 🧚

yes we said "experimental" for LXC. that word is doing a lot of heavy lifting there. proceed accordingly.

🌟 Quick start 🌟

First you must install the software on all the cluster nodes, either from sources or (if available) from a package. The next step is to create the initial cluster configuration, using gnt-cluster init.

Then you can add other nodes, or start creating instances. 🦋

literally it's giving: install ➡️ gnt-cluster init ➡️ go off. no cap.

🏗️ Cluster architecture 🏗️

In Ganeti 2.0, the architecture of the cluster is a little more complicated than in 1.2. The cluster is coordinated by a master daemon (ganeti-masterd(8)), running on the master node. Each node runs (as before) a node daemon, and the master has the RAPI daemon running too.

👑 Node roles 👑

Each node can be in one of the following states:

• master — Only one node per cluster can be in this role, and this node is the one holding the authoritative copy of the cluster configuration and the one that can actually execute commands on the cluster and modify the cluster state. See more details under Cluster configuration. (main character energy, obviously. there can only be one.)
• master_candidate — The node receives the full cluster configuration (configuration file and jobs) and can become a master via the gnt-cluster master-failover command. Nodes that are not in this state cannot transition into the master role due to missing state.
• regular — This the normal state of a node. (just vibing. doing its job. no drama. 💜)
• drained — Nodes in this state are functioning normally but cannot receive new instances, because the intention is to set them to offline or remove them from the cluster.
• offline — These nodes are still recorded in the Ganeti configuration, but except for the master daemon startup voting procedure, they are not actually contacted by the master. This state was added in order to allow broken machines (that are being repaired) to remain in the cluster but without creating problems.

🚩 Node flags 🚩

Nodes have two flags which govern which roles they can take:

• master_capable — The node can become a master candidate, and furthermore the master node. When this flag is disabled, the node cannot become a candidate; this can be useful for special networking cases, or less reliable hardware.
• vm_capable — The node can host instances. When enabled (the default state), the node will participate in instance allocation, capacity calculation, etc. When disabled, the node will be skipped in many cluster checks and operations.

🔧 Node Parameters 🔧

The ndparams refer to node parameters. These can be set as defaults on cluster and node group levels, but they take effect for nodes only.

Currently we support the following node parameters:

• oob_program — Path to an executable used as the out-of-band helper. It needs to implement the corresponding interface; in particular, in needs to support the power-on, power-off, power-cycle, power-status, and health commands. The full specification can be found in the Ganeti Node OOB Management Framework design document (implemented in Ganeti 2.4). Design documents are also available online on http://docs.ganeti.org/.
• spindle_count — This should reflect the I/O performance of local attached storage (e.g. for "file", "plain" and "drbd" disk templates). It doesn't have to match the actual spindle count of (any eventual) mechanical hard-drives, its meaning is site-local and just the relative values matter.
• exclusive_storage — When this Boolean flag is enabled, physical disks on the node are assigned to instance disks in an exclusive manner, so as to lower I/O interference between instances. This parameter cannot be set on individual nodes, as its value must be the same within each node group. The Partitioned Ganeti design document (implemented in Ganeti 2.9) contains more details.
• ovs — When this Boolean flag is enabled, OpenvSwitch will be used as the network layer. This will cause the initialization of OpenvSwitch on the nodes when added to the cluster. Per default this is not enabled.
• ovs_name — When ovs is enabled, this parameter will represent the name of the OpenvSwitch to generate and use. This will default to switch1.
• ovs_link — When ovs is enabled, a OpenvSwitch will be initialized on new nodes and will have this as its connection to the outside. This parameter is not set per default, as it depends very much on the specific setup.
• ssh_port — The port used for SSH connections to nodes belonging to a group. The user is responsible for properly configuring the ports of SSH daemons on machines prior to adding them as Ganeti nodes or when modifying the parameter value of an existing group. Note that using non-standard SSH ports and downgrading to an older Ganeti version that doesn't support ssh_port will break the cluster.

that last one is not a vibe, that's a warning. non-standard ssh_port + downgrade = broken cluster. don't do it bestie. 😭

⚡ Hypervisor State Parameters ⚡

Using --hypervisor-state you can set hypervisor specific states.

The format is: hypervisor:option=value.

Currently we support the following hypervisor state values:

• mem_total — Total node memory, as discovered by this hypervisor
• mem_node — Memory used by, or reserved for, the node itself; note that some hypervisors can report this in an authoritative way, other not
• mem_hv — Memory used either by the hypervisor itself or lost due to instance allocation rounding; usually this cannot be precisely computed, but only roughly estimated
• cpu_total — Total node cpu (core) count; usually this can be discovered automatically
• cpu_node — Number of cores reserved for the node itself; this can either be discovered or set manually. Only used for estimating how many VCPUs are left for instances

Note that currently this option is unused by Ganeti; values will be recorded but will not influence the Ganeti operation.

so you can set it. it just... doesn't do anything yet. recorded and ignored, we love a decorative parameter. 💀

💾 Disk State Parameters 💾

Using --disk-state you can set disk specific states.

The format is: storage_type/identifier:option=value. Where we currently just support lvm as storage type. The identifier in this case is the LVM volume group. By default this is xenvg.

Currently we support the following values here:

• disk_total — Total disk size (usually discovered automatically)
• disk_reserved — Reserved disk size; this is a lower limit on the free space, if such a limit is desired
• disk_overhead — Disk that is expected to be used by other volumes (set via reserved_lvs); usually should be zero

Note that currently this option is unused by Ganeti; values will be recorded but will not influence the Ganeti operation.

same energy as above. cute, recorded, zero impact. 🎀

📦 Cluster configuration 📦

The master node keeps and is responsible for the cluster configuration. On the filesystem, this is stored under the /var/ganeti/lib directory, and if the master daemon is stopped it can be backed up normally.

The master daemon will replicate the configuration database called config.data and the job files to all the nodes in the master candidate role. It will also distribute a copy of some configuration values via the ssconf files, which are stored in the same directory and start with a ssconf_ prefix, to all nodes.

🧾 Jobs 🧾

All cluster modification are done via jobs. A job consists of one or more opcodes, and the list of opcodes is processed serially. If an opcode fails, the entire job is failed and later opcodes are no longer processed. A job can be in one of the following states:

• queued — The job has been submitted but not yet processed by the master daemon.
• waiting — The job is waiting for for locks before the first of its opcodes.
• canceling — The job is waiting for locks, but is has been marked for cancellation. It will not transition to running, but to canceled.
• running — The job is currently being executed.
• canceled — The job has been canceled before starting execution.
• success — The job has finished successfully. 🌟
• error — The job has failed during runtime, or the master daemon has been stopped during the job execution.

---

✨ Common command line features ✨

🎀 Options 🎀

Okay bestie, listen — so many Ganeti commands hand you these same options. Wanna know if a specific command has them? Just ask it with --help. Iconic behavior fr fr. 💅

gnt-... command [--dry-run] [--priority {low | normal | high}]
[--submit] [--print-jobid]

• --dry-run — vibe check first: it lets you peek at whether an operation would succeed, no commitment, no mess. 🦋
• --priority — sets the priority for the opcodes the command submits. Your lil' to-do list ordering, basically.
• --submit — yeets the job off to the master daemon and then does not wait around for it to finish. The job ID gets shown to you so you can go stalk it with gnt-job info later. So real.
• --reason — lets you write down why you submitted this job. It gets inherited by every job this job creates, which makes it way easier to trace why any given job even exists. Some reason strings are ✨special✨ and mean something specific:
  • rate-limit:n:label — assigns the job to a rate-limiting bucket identified by the combo of (n, label). And yes, rate-limit:4:mylabel and rate-limit:5:mylabel are different buckets — don't get it twisted. n has to be a positive integer, and label is any arbitrary ASCII string. The job scheduler then makes sure that, for each bucket, at most n jobs belonging to that bucket are running in parallel. No cap. (okay, actually, literally capped at n.) 🔥

Heads up: those special-case reason strings have to be in exactly the specified format. If anything comes before them — even whitespace — they stop being special and just become normal reasons with no effect. 😭

• --print-jobid — makes the command print the job id as the first line on stdout, so other programs can parse it without drama. Parsing-friendly queen. 👑

🌸 Defaults 🌸

For some commands you can set default command line arguments through environment variables. Just assign the arguments as a string to the matching variable. The variable name format is binary_command — where binary is the gnt-* script name in ALL UPPER CASE with dashes turned into underscores, and command is the command you're invoking on that script. 🧚

Right now the supported commands are gnt-node list, gnt-group list and gnt-instance list. So yeah, you can set your default flags with GNT_NODE_LIST, GNT_GROUP_LIST and GNT_INSTANCE_LIST. Set it and forget it, periodt. 💜

🦋 Debug options 🦋

If the variable FORCE_LUXI_SOCKET is set, it overrides the socket that command-line tools (gnt-*) use for LUXI connections. This is mostly for debugging, and lowkey some operations won't work at all — like, say, if you point this variable at the confd-supplied query socket and then try to submit a job? Yeah. That's not gonna go well. 💀

• If it's set to the value master, it connects to the correct path for the master daemon — even if, for example, split queries are enabled and this is a query operation.
• If it's set to query, it will always (try to) connect to the query socket, even if split queries are disabled.
• Otherwise, the value is treated as a filesystem path to the socket to use. 🛣️

---

🎀 Field formatting 🎀

Lots of Ganeti commands share one framework for tabular listing of resources (think gnt-instance list, gnt-node list, gnt-group list, gnt-debug locks, etc.). For those commands, special states get shown as a special symbol in terse mode, or a string in verbose mode:

• *, (offline) — this node is marked offline, so we can't query it for data. Persistent until the node is de-offlined. She's on do-not-disturb mode. 🚫
• ?, (nodata) — Ganeti expected an answer from this entity, but the cluster RPC call failed and/or no valid answer came back. Usually there's more info in the node daemon log (if the node is alive) or the master daemon log. This one is transient — re-running the command might give you a whole different result. 🙈
• -, (unavail) — this field just doesn't make sense for this entity. Example: asking a down instance for its current memory live usage, or asking a non-vm_capable node for disk/memory data. Persistent — until the entity's state changes via Ganeti commands, the result won't change. 🫠
• ??, (unknown) — this field is not known. (Note: that's different from the entity being unknown.) Either you mis-typed the field name, or you're using a field the running Ganeti master daemon doesn't know about. Persistent — re-running won't fix it, bestie. 😭

💅 Key-value parameters 💅

Several options take parameters shaped like key=value,key=value,... or category:key=value,.... Think hypervisor parameters, backend parameters, and friends. For these, you can use values that contain commas — escape them with a backslash. And because shells are dramatic, that backslash needs to be doubled if you're not single-quoting: 😤

# gnt-instance modify -H kernel_path=an\\,example instance1
# gnt-instance modify -H kernel_path='an\,example' instance1

Also, these non-string parameters can be passed:

• To pass the boolean value True, just mention the key — leave out the equality sign and any value.
• To pass the boolean value False, again just mention the key, but prefix it with no_.
• To pass the special None value, again just mention the key, but prefix it with a single - sign.

🌈 Query filters 🌈

Most commands that list resources (instances, nodes, whatever) support filtering. The filter language is kinda like Python expressions with a few Perl elements sprinkled in. It is not generic, though — each condition has to be a field name and a value (except for boolean checks), and a field can never be compared to another field. Keywords are case-sensitive, so don't get cute. ⭐

Examples — syntax details are down below:

• List webservers:

gnt-instance list --filter 'name =* "web*.example.com"'

• List instances with three or six virtual CPUs whose primary nodes reside in groups starting with the string "rack":

gnt-instance list --filter
  '(be/vcpus == 3 or be/vcpus == 6) and pnode.group =~ m/^rack/'

• Nodes hosting primary instances:

gnt-node list --filter 'pinst_cnt != 0'

• Nodes which aren't master candidates:

gnt-node list --filter 'not master_candidate'

• Short version for globbing patterns:

gnt-instance list '*.site1' '*.site2'

Syntax in pseudo-BNF:

<quoted-string> ::= /* String quoted with single or double quotes,
                       backslash for escaping */

<integer> ::= /* Number in base-10 positional notation */

<re> ::= /* Regular expression */

/*
  Modifier "i": Case-insensitive matching, see
  http://docs.python.org/library/re#re.IGNORECASE

  Modifier "s": Make the "." special character match any character,
  including newline, see http://docs.python.org/library/re#re.DOTALL
*/
<re-modifiers> ::= /* empty */ | i | s

<value> ::= <quoted-string> | <integer>

<condition> ::=
  { /* Value comparison */
    <field> { == | != | < | <= | >= | > } <value>

    /* Collection membership */
    | <value> [ not ] in <field>

    /* Regular expressions (recognized delimiters
       are "/", "#", "^", and "|"; backslash for escaping)
    */
    | <field> { =~ | !~ } m/<re>/<re-modifiers>

    /* Globbing */
    | <field> { =* | !* } <quoted-string>

    /* Boolean */
    | <field>
  }

<filter> ::=
  { [ not ] <condition> | ( <filter> ) }
  [ { and | or } <filter> ]

Operators, bestie, here's the whole lineup:

• == — Equality
• != — Inequality
• < — Less than
• <= — Less than or equal
• > — Greater than
• >= — Greater than or equal
• =~ — Pattern match using regular expression
• !~ — Logically negated from =~
• =* — Globbing, see glob(7), though only * and ? are supported
• !* — Logically negated from =*
• in, not in — Collection membership and negation

---

💜 Common daemon functionality 💜

Every Ganeti daemon re-opens its log file(s) when it gets a SIGHUP signal. So you can use logrotate(8) to rotate Ganeti's log files. She said "fresh logs, who dis". 🌸

😭 REPORTING BUGS 😭

Found a bug? Report it to the project's issue tracker, or hit up the developers on the Ganeti mailing list. We're friendly, promise. 🫶

⭐ SEE ALSO ⭐

Ganeti overview and specifications: ganeti(7) (general overview), ganeti-os-interface(7) (guest OS definitions), ganeti-extstorage-interface(7) (external storage providers).

Ganeti commands: gnt-cluster(8) (cluster-wide commands), gnt-job(8) (job-related commands), gnt-node(8) (node-related commands), gnt-instance(8) (instance commands), gnt-os(8) (guest OS commands), gnt-storage(8) (storage commands), gnt-group(8) (node group commands), gnt-backup(8) (instance import/export commands), gnt-debug(8) (debug commands).

Ganeti daemons: ganeti-watcher(8) (automatic instance restarter), ganeti-cleaner(8) (job queue cleaner), ganeti-noded(8) (node daemon), ganeti-rapi(8) (remote API daemon).

Ganeti htools: htools(1) (generic binary), hbal(1) (cluster balancer), hspace(1) (capacity calculation), hail(1) (IAllocator plugin), hscan(1) (data gatherer from remote clusters), hinfo(1) (cluster information printer), mon-collector(7) (data collectors interface).

👑 COPYRIGHT 👑

Copyright (C) 2006-2015 Google Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

---

and that's a wrap on ganeti(7), bestie!! 💖 this whole thing is the Ganeti manual, version 7, and that lil' footer at the bottom is just the manual name and its section number doin' their thing. go forth and cluster responsibly, you absolute icon. 🎀✨🦋