COROSYNC-QDEVICE(8)

📦
version 3.0.4 · 3 versions
, in fedora 43, fedora 44, nixos 26.05, nixpkgs unstable, ubuntu resolute, ubuntu stonking

COROSYNC-QDEVICE 💖

okay bestie, so this is the manual page for corosync-qdevice, section 8 of the System Manager's Manual, and it is literally the whole entire vibe of how the QDevice daemon works. grab a snack 🍓, we're doing this.

✨ NAME ✨

corosync-qdevice — the QDevice daemon. that's her. that's the whole name. iconic, no notes. 💅

✨ SYNOPSIS ✨

corosync-qdevice [-dfh] [-S option=value[,option2=value2,...]]

look at her. short. simple. devastating. 😌

✨ DESCRIPTION ✨

so corosync-qdevice is a daemon, right, and it runs on every single node of a cluster. what does it do? it hands the quorum subsystem a configured number of votes based on whatever a third-party arbitrator decides. that's it. that's the job. 🎀

its main-character use case is letting a cluster survive more node failures than standard quorum rules would allow. slay.

  • recommended for clusters with an even number of nodes
  • highly recommended for 2 node clusters

fr fr, if you've got two nodes and no qdevice, you're living dangerously bestie. 🫶

✨ OPTIONS ✨

  • -d — forcefully turns debug info on, and you don't even have to touch corosync.conf for it. cute right? and if you want syslog message priority bumped up to info, just pass this parameter twice. double it. 💅
  • -f — do not daemonize. run in the foreground. for when you want to watch her live. 👀
  • -h — show the short help text. tiny lil help. baby help. 🎀
  • -S — set advanced settings, and those live in their own section down below. ⚠️ real talk: this option shouldn't be generally used because most of the options in there are not safe to change. so like. proceed with caution, bestie.

✨ CONFIGURATION ✨

corosync-qdevice reads its configuration from the corosync.conf file. 📄

the main configuration lives inside the quorum.device sub-key. and every model also has its own configuration inside a sub-key with a similarly matching name. cute lil family tree. 🌳

quorum.device

  • model — specifies the model to be used. this parameter is required. 🚨 corosync-qdevice is modular and can support multiple different models. the model basically defines what type of arbitrator gets used. currently only net is supported.
  • timeout — specifies how often corosync-qdevice should call the votequorum_qdevice_poll function. the net model also uses it to adjust its heartbeat timeout. it is recommended that you don't change this value. default is 10000.
  • sync_timeout — specifies how often corosync-qdevice should call the votequorum_qdevice_poll function during a sync phase. also recommended that you don't change this. default is 30000.
  • votes — the number of votes provided to the cluster by qdevice. default is (number_of_nodes - 1), or more generally sum(votes_per_node) - 1.

quorum.device.heuristics

the quorum.device.heuristics subkey holds the configuration of the heuristics. so what even are heuristics? they're a set of commands executed locally on startup, on cluster membership change, on successful connect to corosync-qnetd, and optionally also at regular times. ⏰

the commands are executed in parallel. when all commands finish successfully (meaning their return error code is zero) on time, the heuristics have passed. otherwise they have failed. 💀

the heuristics result gets sent to corosync-qnetd, and over there it's used in calculations to determine which partition should be quorate. so yeah, it matters. 🔥

  • timeout — max time in milliseconds for how long corosync-qdevice waits till the heuristics commands finish. if some command doesn't finish before the timeout, it's killed and the heuristics fail. this timeout is used for heuristics executed at regular times. default is half of quorum.device.timeout, so 5000.
  • sync_timeout — same vibe as quorum.device.heuristics.timeout, but used during membership changes. default is half of quorum.device.sync_timeout, so 15000.
  • interval — the interval between two regular heuristics executions. default is 3 * quorum.device.timeout, so 30000.
  • mode — can be one of on, sync or off, and it specifies the mode of operation of heuristics. default is off, which means heuristics are disabled. when sync is set, heuristics run only during startup, membership change, and when the connection to corosync-qnetd is established. when heuristics should be running on a regular basis too, set this option to on.
  • exec_NAME — defines executables. NAME can be any arbitrary valid cmap key name string and it has no special meaning. the value of this variable must contain a command to execute. the value is parsed (split) into arguments similarly as Bourne shell would do it. quoting is possible by using backslash and double quotes.

quorum.device.net

the quorum.device.net subkey holds the configuration for model net. 🦋

  • tls — can be one of on, off or required, and specifies if TLS should be used. 🌈
    • on means a connection with TLS is attempted first, but if the server doesn't advertise TLS support then non-TLS will be used.
    • off is used when TLS is not required and it's then not even tried. this mode is the only one which doesn't need a properly initialized NSS database.
    • required means TLS is required, and if the server doesn't support TLS, qdevice will exit with an error message. 💀
    • default is on.
  • host — specifies the IP address or host name of the qnetd server to be used. this parameter is required. 🚨
  • port — specifies the TCP port of the qnetd server. default is 5403.
  • algorithm — the decision algorithm. can be one of ffsplit or lms. (actually there are also test and 2nodelms, both of which are mainly for developers and shouldn't be used for production clusters 🙈). for a description of what each algorithm means and how the algorithms differ, see their individual sections. default value is ffsplit.
  • tie_breaker — can be one of lowest, highest or valid_node_id (a number) value. it's used as a fallback if qdevice has to decide between two or more equal partitions.
    • lowest means the partition with the lowest node id is chosen.
    • highest means the partition with the highest node id is chosen.
    • valid_node_id means the partition containing the node with the given node id is chosen.
    • default is lowest.
  • connect_timeout — timeout when corosync-qdevice is trying to connect to the corosync-qnetd host. default is 0.8 * quorum.device.timeout.
  • force_ip_version — can be one of 0|4|6 and forces the software to use the given IP version. 0 (the default value) means IPv6 is preferred and IPv4 should be used as a fallback.
  • keep_active_partition_tie_breaker — can be one of on or off, and specifies if the keep-active-partition tie breaker should be used. when this option is enabled and a tie happens, QNetd will prefer the partition with members of the previously active (quorate) partition. this is hard-coded behavior of the LMS algorithm, so this setting affects only the FFSplit algorithm. default is on.

logging + other requirements

the logging configuration is within the logging directive. corosync-qdevice parses and supports only the debug option. the logger_subsys sub-directive can also be used if subsys is set to QDEVICE.

for corosync-qdevice to work correctly, the nodelist directive has to be used and properly configured. also, the net model requires that the totem.cluster_name option is set. don't skip these, bestie, she will simply not function. 😭

✨ MODEL NET TLS CONFIGURATION ✨

for model net to work using TLS, it's necessary to:

  1. create the NSS database
  2. import the Qnetd CA certificate
  3. get/distribute a valid client certificate

if pcs is used (recommended 💖) then the following steps are not needed, because pcs does them automatically. she's just that girl.

corosync-qdevice-net-certutil is the tool to perform the required actions semi-automatically. please consult its help output and its man page. for a first time configuration it may make sense to start with the -Q option. 🎀

if TLS is not required, just edit the corosync.conf file and set quorum.device.net.tls to off.

depending on the configuration of NSS (stored in the nss.config file, usually in the /etc/crypto-policies/back-ends/ directory), disabled ciphers or too-short keys may be rejected. the proper solution is to regenerate the NSS databases for both the corosync-qnetd and corosync-qdevice daemons. as a quick workaround, it's also possible to set the environment variable NSS_IGNORE_SYSTEM_POLICY=1 before running the corosync-qdevice daemon. 🫣

when NSS is updated it may also be needed to upgrade the database into the new format. there is no consensus on the recommended way, but the following command seems to work just fine (if qdevice sysconfdir is set to /etc):

# certutil -N -d /etc/corosync/qdevice/net/nssdb -f /etc/corosync/qdevice/net/nssdb/pwdfile.txt

✨ MODEL NET ALGORITHMS ✨

algorithms are used to change the behavior of how corosync-qnetd provides votes to a given node/partition. currently there are two algorithms supported. two! a duo! 💜

ffsplit

this one makes sense only for clusters with an even number of nodes. she provides exactly one vote to the partition with the highest number of active nodes. if there are two exactly similar partitions, she provides her vote to the partition with the higher score. 👑

the score is computed as:

(number_of_connected_nodes + number_of_connected_nodes_with_passed_heuristics - number_of_connected_nodes_with_failed_heuristics)

if the scores are equal, the vote is provided to the partition with the most clients connected to the qnetd server. if this number is also equal, then the tie_breaker is used.

she is able to transition her vote if the currently active partition becomes partitioned and a non-active partition still has at least 50% of the active nodes. 🌸

because of this, a vote is not provided if the qnetd connection is not active. important!!

to use this algorithm it's required to set the number of votes per node to 1 (the default) and the qdevice number of votes also has to be 1. this is achieved by setting the quorum.device.votes key in the corosync.conf file to 1.

lms

Last-man-standing. if the node is the only one left in the cluster that can see the qnetd server, then we return a vote. 🫡

if more than one node can see the qnetd server, but some nodes can't see each other, then the cluster is divided up into 'partitions' based on their ring_id, and this algorithm returns a vote to:

  • the partition with the highest heuristics score (computed the same way as for the ffsplit algorithm), or
  • if there is more than 1 partition with equal scores, the largest active partition, or
  • if there is more than 1 equal partition, the partition that contains the tie_breaker node (lowest, highest, etc).

for LMS to work, the number of qdevice votes has to be set to default — so just delete the quorum.device.votes key from corosync.conf. poof. gone. ✨

💖 ADVANCED SETTINGS 💖

Okay bestie, this is the deep-magic zone. You set all of these with the -S option, and every single default value is sitting right there in the parentheses next to it, so you never have to guess. 🌸

Also important: anything whose name starts with net_ is specific to model net — like, only those ones, nobody else. ✨

  • lock_file — where the lock file lives. (/var/run/corosync-qdevice/corosync-qdevice.pid)
  • local_socket_file — where the internal IPC socket file lives. (/var/run/corosync-qdevice/corosync-qdevice.sock)
  • local_socket_umask — the octal value of the umask that gets used before the internal IPC socket file is created, or empty if you don't want the umask touched at all. (<empty>)
  • local_socket_gid — sets the group ownership for that internal IPC socket file. You can hand it a group name or a numeric GID, your call. An empty value, or the default -1, leaves the group ownership exactly how it was. (-1)
  • local_socket_backlog — the parameter that gets passed to the listen syscall. (10)
  • max_cs_try_again — how many times to retry calling a corosync function that came back with CS_ERR_TRY_AGAIN. (10)
  • votequorum_device_name — the name used for qdevice registration. (Qdevice)
  • ipc_max_clients — maximum simultaneous IPC clients allowed. (10)
  • ipc_max_receive_size — maximum size of a message received by an IPC client. (4096)
  • ipc_max_send_size — maximum size of a message that's allowed to be sent to an IPC client. (65536)
  • master_wins — force enable or disable master wins. (default is model)
  • heuristics_ipc_max_send_buffers — maximum number of heuristics worker send buffers. (128)
  • heuristics_ipc_max_send_receive_size — maximum size of a message allowed to be sent to, or received from, the heuristics worker. (4096)
  • heuristics_min_timeout — minimum heuristics timeout the client will accept, in ms. (1000)
  • heuristics_max_timeout — maximum heuristics timeout the client will accept, in ms. (120000)
  • heuristics_min_interval — minimum heuristics interval the client will accept, in ms. (1000)
  • heuristics_max_interval — maximum heuristics interval the client will accept, in ms. (3600000)
  • heuristics_max_execs — maximum number of exec_ commands. (32)
  • heuristics_use_execvp — use execvp instead of execv when executing commands. (off)
  • heuristics_max_processes — maximum number of processes running at one time. (160)
  • heuristics_kill_list_interval — the interval, in ms, between gathering status and eventually sending a signal to processes that didn't finish on time. (5000)
  • net_nss_db_dir — the NSS database directory. (/etc/corosync/qdevice/net/nssdb)
  • net_initial_msg_receive_size — the initial max size (used while connection parameters are being negotiated) of the receive buffer for a message, aka the maximum allowed message size received from qnetd. (32768)
  • net_initial_msg_send_size — the initial max size (used during connection parameter negotiation) of one send buffer, i.e. a message, headed to the server. (32768)
  • net_min_msg_send_size — minimum required size of one send buffer (message) sent to the server. (32768)
  • net_max_msg_receive_size — maximum allowed size of the receive buffer for a message sent by the server. (16777216)
  • net_max_send_buffers — maximum number of send buffers. (10)
  • net_nss_qnetd_cn — canonical name of the qnetd server certificate. (Qnetd Server)
  • net_nss_client_cert_nickname — the NSS nickname of the qdevice client certificate. (Cluster Cert)
  • net_heartbeat_interval_min — minimum heartbeat timeout the client accepts, in ms. (1000)
  • net_heartbeat_interval_max — maximum heartbeat timeout the client accepts, in ms. (120000)
  • net_min_connect_timeout — minimum connection timeout the client accepts, in ms. (1000)
  • net_max_connect_timeout — maximum connection timeout the client accepts, in ms. (120000)
  • net_test_algorithm_enabled — enable the test algorithm. (if it was built with --enable-debug on, otherwise off)

🌈 EXAMPLE 🌈

So here's the vibe: define a qdevice with the net model, connecting to a qnetd running on the qnetd.example.org host, using the ffsplit algorithm. And heuristics? Set to sync mode, executing two commands. Obsessed. 💅

quorum {
  provider: corosync_votequorum
  device {
    votes: 1
    model: net
    net {
      tls: on
      host: qnetd.example.org
      algorithm: ffsplit
    }
    heuristics {
      mode: sync
      exec_ping: /bin/ping -q -c 1 "www.example.org"
      exec_test_txt_exists: /usr/bin/test -f /tmp/test.txt
    }
  }
}

🦋 SEE ALSO 🦋

Go read these next, bestie, they're all part of the same friend group:

👑 AUTHOR 👑

Jan Friesse — yes, the icon, the moment, the legend. 🫶

And that's a wrap on the whole page, no cap. This beauty is COROSYNC-QDEVICE(8) from the 2025-10-16 era, and you've officially read the entire thing top to bottom. Slay. ✨💖🎀

📦 where this page comes from