COROSYNC_CONF(5)

📦
version 3.1.10 · 5 versions
, in epel 9, fedora 43, fedora 44, ubuntu stonking

💖 NAME 💖

okay so bestie, corosync.conf is the corosync executive configuration file. This page is COROSYNC_CONF(5) from the Corosync Cluster Engine Programmer's Manual 👑.

✨ SYNOPSIS ✨

/etc/corosync/corosync.conf

🌸 DESCRIPTION 🌸

The corosync.conf file instructs the corosync executive about various parameters needed to control the corosync executive. Empty lines and lines starting with # character are ignored. The configuration file consists of bracketed top level directives. The possible directive choices are:

  • totem { } — This top level directive contains configuration options for the totem protocol. 🐾
  • logging { } — This top level directive contains configuration options for logging. 📝
  • quorum { } — This top level directive contains configuration options for quorum. 🗳️
  • nodelist { } — This top level directive contains configuration options for nodes in cluster. 🧑‍🤝‍🧑
  • system { } — This top level directive contains configuration options related to system. 🖥️
  • resources { } — This top level directive contains configuration options for resources. 🧺
  • nozzle { } — This top level directive contains configuration options for a libnozzle device. 🚿

Corosync supports multiple types of network transports for communication between the nodes in the cluster. There are three types of transports:

  1. KNET. This is a default and recommended transport introduced in Corosync 3. It provides several advantages over the UDP and UDPU transports, including better performance, link-level redundancy, automatic link recovery, and native IP compression and encryption. 🔥
  2. UDPU. This is for unicast communication. This transport is deprecated. 💀
  3. UDP. This is for multicast communication. This transport is deprecated and highly discouraged to use. 🚫

The interface sub-directive of totem is optional for UDP and KNET transports.

For KNET, multiple interface subsections define parameters for each KNET link on the system. 🦋

For UDPU an interface section is not needed and it is recommended that the nodelist is used to define cluster nodes.

Now the interface options, bestie 💅:

  • linknumber — This specifies the link number for the interface. When using the KNET protocol, each interface should specify separate link numbers to uniquely identify to the membership protocol which interface to use for which link. The linknumber must start at 0. For UDP the only supported linknumber is 0.
  • knet_link_priority — This specifies the priority for the link when KNET is used in 'passive' mode. (see link_mode below)
  • knet_ping_interval — This specifies the interval between KNET link pings. knet_ping_interval and knet_ping_timeout are a pair, if one is specified the other should be too, otherwise one will be calculated from the token timeout and one will be taken from the config file. (default is token timeout / (knet_pong_count*2))
  • knet_ping_timeout — If no ping is received within this time, the KNET link is declared dead. knet_ping_interval and knet_ping_timeout are a pair, if one is specified the other should be too, otherwise one will be calculated from the token timeout and one will be taken from the config file. (default is token timeout / knet_pong_count)
  • knet_ping_precision — How many values of latency are used to calculate the average link latency. (default 2048 samples)
  • knet_pong_count — How many valid ping/pongs before a link is marked UP. (default 2)
  • knet_transport — Which IP transport KNET should use. valid values are "udp" (or "sctp" if supported by the installed version of kronosnet). (default: udp) NOTE: even if currently included, sctp is deprecated and will be removed in kronoset v2.0
  • bindnetaddr (UDP only) — This specifies the network address the corosync executive should bind to when using UDP transport.

bindnetaddr (UDP only) should be an IP address configured on the system, or a network address.

For example, if the local interface is 192.168.5.92 with netmask 255.255.255.0, you should set bindnetaddr to 192.168.5.92 or 192.168.5.0. If the local interface is 192.168.5.92 with netmask 255.255.255.192, set bindnetaddr to 192.168.5.92 or 192.168.5.64, and so forth.

This may also be an IPV6 address, in which case IPV6 networking will be used. In this case, the exact address must be specified and there is no automatic selection of the network interface within a specific subnet as with IPv4.

If IPv6 networking is used, the nodeid field in nodelist must be specified.

  • broadcast (UDP only) — This is optional and can be set to yes. If it is set to yes, the broadcast address will be used for communication. If this option is set, mcastaddr should not be set.
  • mcastaddr (UDP only) — This is the multicast address used by corosync executive. The default should work for most networks, but the network administrator should be queried about a multicast address to use. Avoid 224.x.x.x because this is a "config" multicast address. 🙈

This may also be an IPV6 multicast address, in which case IPV6 networking will be used. If IPv6 networking is used, the nodeid field in nodelist must be specified.

It's not necessary to use this option if cluster_name option is used. If both options are used, mcastaddr has higher priority.

  • mcastport — This specifies the UDP port number. Exact meaning depends on used transport.

For the KNET transport, it is used as both the binding and sending port. For additional links, the default value is computed as (mcastport + linknumber).

For the UDPU transport, mcastport specifies the bind port. Corosync also allocates and binds a random sending port for each remote node.

For the UDP transport, it is used as a bind port for mcastaddr. It is possible to use the same multicast address on a network with the corosync services configured for different UDP ports. Please note corosync uses two UDP ports mcastport (for mcast receives) and mcastport - 1 (for mcast sends). If you have multiple clusters on the same network using the same mcastaddr please configure the mcastports with a gap.

The default is 5405.

  • ttl (UDP only) — This specifies the Time To Live (TTL). If you run your cluster on a routed network then the default of "1" will be too small. This option provides a way to increase this up to 255. The valid range is 0..255.

Within the totem directive, there are seven configuration options of which one is required, five are optional, and one is required when IPV6 is configured in the interface subdirective. The required directive controls the version of the totem configuration. The optional option unless using IPV6 directive controls identification of the processor. The optional options control secrecy and authentication, the network mode of operation and maximum network MTU field.

  • version — This specifies the version of the configuration file. Currently the only valid version for this directive is 2.
  • clear_node_high_bit — This configuration option is optional and is only relevant when no nodeid is specified. Some corosync clients require a signed 32 bit nodeid that is greater than zero however by default corosync uses all 32 bits of the IPv4 address space when generating a nodeid. Set this option to yes to force the high bit to be zero and therefore ensure the nodeid is a positive signed 32 bit integer.
WARNING: Cluster behavior is undefined if this option is enabled on only a subset of the cluster (for example during a rolling upgrade). 😭
  • crypto_model — This specifies which cryptographic library should be used by KNET. Supported values depend on the libknet build and on the installed cryptography libraries. Typically nss and openssl will be available but gcrypt and others could also be allowed.

The default is nss.

  • crypto_hash — This specifies which HMAC authentication should be used to authenticate all messages. Valid values are none (no authentication), md5, sha1, sha256, sha384 and sha512. Encrypted transmission is only supported for the KNET transport.

The default is none.

  • crypto_cipher — This specifies which cipher should be used to encrypt all messages. Valid values are none (no encryption), aes256, aes192 and aes128. Enabling crypto_cipher, requires also enabling of crypto_hash. Encrypted transmission is only supported for the KNET transport.

The default is none.

  • secauth — This implies crypto_cipher=aes256 and crypto_hash=sha256, unless those options are explicitly set. Encrypted transmission is only supported for the KNET transport.

The default is off.

  • keyfile — This specifies the fully qualified path to the shared key used to authenticate and encrypt data used within the Totem protocol.

The default is /etc/corosync/authkey.

  • key — Shared key stored in configuration instead of authkey file. This option has lower precedence than keyfile option so it's used only when keyfile is not specified. Using this option is not recommended for security reasons. 🔒
  • link_mode — This specifies the Kronosnet mode, which may be passive, active, or rr (round-robin). passive: the active link with the highest priority (highest number) will be used. If one or more links share the same priority the one with the lowest link ID will be used. active: All active links will be used simultaneously to send traffic. link priority is ignored. rr: Round-Robin policy. Each packet will be sent to the next active link in order.

If only one interface directive is specified, passive is automatically chosen.

The maximum number of interface directives that is allowed with Kronosnet is 8. For other transports it is 1. 🌈

  • netmtu — This sets the maximum packet length that corosync will send, bestie. It's mainly for the UDPU (and UDP) transport, where it specifies the network maximum transmit size, but it can also be used with the KNET transport, where it defines the maximum length of packets passed to the KNET layer. If you wanna set the network MTU manually for KNET, use the knet_mtu option instead. 💅

For UDPU (and UDP), if you set this beyond 1500 — the regular frame MTU — you need ethernet devices that support large, or jumbo, frames. If any device in the network doesn't support large frames, the protocol will not operate properly. The hosts must also have their mtu size set from 1500 to whatever frame size you specify here. 😭

Heads up: some NICs or switches claim large frame support, but they actually support 9000 MTU as the maximum frame size including the IP header. Setting netmtu and host MTUs to 9000 will cause totem to use the full 9000 bytes of the frame. Then Linux adds an 18 byte header, moving the full frame size to 9018. As a result, some hardware will not operate properly with this size of data. A netmtu of 8982 seems to work for the few large frame devices that have been tested. Some manufacturers claim large frame support when in fact they support frame sizes of 4500 bytes. 💀

When sending multicast traffic, if the network frequently reconfigures, chances are that some device in the network doesn't support large frames. 🦋

Choose hardware carefully if you're intending to use large frame support. 👑

The default is 1500 for UDPU (and UDP) and 65536 for the KNET transport.

  • transport — This directive controls the transport mechanism used. The default is knet (for KNET). The transport type can also be set to udpu (for UDPU) or udp (for UDP). Only KNET allows crypto or multiple interfaces per node. 🔥
  • cluster_name — This specifies the name of the cluster and it's used for automatic generating of multicast address. ✨
  • config_version — This specifies the version of the config file. It's converted to an unsigned 64-bit int. By default it's 0. This option is used to prevent joining old nodes with not up-to-date configuration. If the value is not 0, and the node is going for the first time (only for the first time — join after split doesn't follow this rule) from single-node membership to multiple-nodes membership, other nodes' config_versions are collected. If the current node's config_version is not equal to the highest of the collected versions, corosync is terminated. 💀
  • ip_version — This specifies the version of IP to ask the DNS resolver for. The value can be one of: ipv4 (look only for an IPv4 address), ipv6 (check only IPv6 address), ipv4-6 (look for all address families and use the first IPv4 address found in the list if there is such an address, otherwise use the first IPv6 address), and ipv6-4 (look for all address families and use the first IPv6 address found in the list if there is such an address, otherwise use the first IPv4 address). 🌈

The default (if unspecified) is ipv6-4 for KNET and UDPU transports and ipv4 for UDP transport.

The KNET transport supports IPv4 and IPv6 addresses concurrently, provided they are consistent on each link. 🫶

  • ip_dscp — This specifies the dscp (differentiated services code point) value to be used in the IP header's type of service (TOS) field according to RFC 2474. Allowed values are 0-63 and symbolic dscp names: cs0, cs1, cs2, cs3, cs4, cs5, cs6, cs7, af11, af12, af13, af21, af22, af23, af31, af32, af33, af41, af42, af43 and ef. Value 0 disables the dscp support. Use of the TOS field then depends on the used transport. If ip_dscp is set to a value not equal to zero, and an appropriate socket option (IP_TOS) is available, the value is put into the upper six bits of the TOS header field. 💅

Within the totem directive, there are several configuration options which are used to control the operation of the protocol. It is generally not recommended to change any of these values without proper guidance and sufficient testing. Some networks may require larger values if suffering from frequent reconfigurations. Some applications may require faster failure detection times, which can be achieved by reducing the token timeout. 🎀

  • token — This timeout is used directly or as a base for real token timeout calculation (explained in the token_coefficient section). Token timeout specifies in milliseconds until a token loss is declared after not receiving a token. This is the time spent detecting a failure of a processor in the current configuration. Reforming a new configuration takes about 50 milliseconds in addition to this timeout. 😭

For the real token timeout used by totem, it's possible to read the cmap value of the runtime.config.totem.token key. 🔑

Be careful to use the same timeout values on each of the nodes in the cluster, or unpredictable results may occur. 💀

The default is 3000 milliseconds.

  • token_warning — Specifies the interval between warnings that the token has not been received. The value is a percentage of the token timeout and can be set to 0 to disable warnings. 🦋

The default is 75%.

  • token_coefficient — This value is used only when the nodelist section is specified and contains at least 3 nodes. If so, the real token timeout is then computed as token + (number_of_nodes - 2) * token_coefficient. This allows the cluster to scale without manually changing the token timeout every time a new node is added. This value can be set to 0, resulting in effective removal of this feature. ✨

The default is 650 milliseconds.

  • token_retransmit — This timeout specifies in milliseconds after how long before receiving a token the token is retransmitted. This will be automatically calculated if token is modified. It is not recommended to alter this value without guidance from the corosync community. 🙈

The minimum is 30 milliseconds. If not set and an error occurs, make sure token / (token_retransmits_before_loss_const + 0.2) is more than 30. 🤓

The default is 238 milliseconds for a two nodes cluster. Three or more nodes reference token_coefficient.

  • knet_compression_model — Type of compression used by Kronosnet. Supported values depend on the libknet build and on the installed compression libraries. Typically zlib and lz4 will be available, but bzip2 and others could also be allowed. The default is 'none'. 💅
  • knet_compression_threshold — Tells KNET to NOT compress any packets that are smaller than the value indicated. Default 100 bytes. 😤

Set to 0 to reset to the default. Set to 1 to compress everything. 🔥

  • knet_compression_level — Many compression libraries allow tuning of compression parameters. For example 0 or 1 ... 9 are commonly used to determine the level of compression. This value is passed unmodified to the compression library, so it is recommended to consult the library's documentation for more detailed information. 📚
  • hold — This timeout specifies in milliseconds how long the token should be held by the representative when the protocol is under low utilization. It is not recommended to alter this value without guidance from the corosync community. 🫶

The default is 180 milliseconds.

  • token_retransmits_before_loss_const — This value identifies how many token retransmits should be attempted before forming a new configuration. It is also used for token_retransmit and hold calculations. 😌

The default is 4 retransmissions.

  • join — This timeout specifies in milliseconds how long to wait for join messages in the membership protocol. 🎀

The default is 50 milliseconds.

  • send_join — This timeout specifies in milliseconds an upper range between 0 and send_join to wait before sending a join message. For configurations with less than 32 nodes, this parameter is not necessary. For larger rings, this parameter is necessary to ensure the NIC is not overflowed with join messages on formation of a new ring. A reasonable value for large rings (128 nodes) would be 80msec. Other timer values must also change if this value is changed. Seek advice from the corosync mailing list if trying to run larger configurations. 📧

The default is 0 milliseconds.

  • consensus — This timeout specifies in milliseconds how long to wait for consensus to be achieved before starting a new round of membership configuration. The minimum value for consensus must be 1.2 * token. This value will be automatically calculated at 1.2 * token if the user doesn't specify a consensus value. 🤝

For two node clusters, a consensus larger than the join timeout but less than token is safe. For three node or larger clusters, consensus should be larger than token. There is an increasing risk of odd membership changes, which still guarantee virtual synchrony, as node count grows if consensus is less than token. 😬

The default is 3600 milliseconds.

  • merge — This timeout specifies in milliseconds how long to wait before checking for a partition when no multicast traffic is being sent. If multicast traffic is being sent, the merge detection happens automatically as a function of the protocol. 🌸

The default is 200 milliseconds.

  • downcheck — This timeout specifies in milliseconds how long to wait before checking that a network interface is back up after it has been downed. 🔌

The default is 1000 milliseconds.

  • fail_recv_const — This constant specifies how many rotations of the token without receiving any of the messages when messages should be received may occur before a new configuration is formed. 💀

The default is 2500 failures to receive a message.

  • seqno_unchanged_const — This constant specifies how many rotations of the token without any multicast traffic should occur before the hold timer is started. ⏳

The default is 30 rotations.

  • heartbeat_failures_allowed — [HeartBeating mechanism] this turns on the optional HeartBeating mechanism, which exists purely to detect failures faster. ⚠️ real talk: if your network is lossy, turning this on can make you declare failures that never actually happened, because the whole mechanism leans on the network to do the heartbeating.

So the rule of thumb bestie: reach for this if you want better failure detection in low-to-medium utilized networks. 🌸

The value is the number of heartbeat failures the system should tolerate before it declares heartbeat failure — like 3. And if you don't set it, or you set it to 0, the heartbeat mechanism is just not engaged at all, and token rotation stays the failure detection method.

The default is 0 (disabled).

  • max_network_delay — [HeartBeating mechanism] this one is, in milliseconds, roughly how long your network takes to move one packet from one machine to another. 🦋 This value is for system engineers to set — please don't touch it if you're not sure, because it directly affects the heartbeat-based failure detection mechanism.

The default is 50 milliseconds.

  • window_size — the max number of messages that can be sent on one token rotation. If every processor performs equally well, this could be big (300), but that would introduce higher latency from origination to delivery in very large rings. To keep latency down in large rings (16+), the defaults are the safe compromise. 👑

If one or more slow processors are hanging out among the fast ones, window_size should be no larger than 256000 / netmtu, otherwise you overflow the kernel receive buffers. You get told about this by a retransmit list showing up in the notification logs. No data is lost, but performance drops while those errors are happening.

The default is 50 messages.

  • max_messages — the max number of messages one processor may send on receipt of the token. Heads up: max_messages is limited to 256000 / netmtu to prevent overflow of the kernel transmit buffers. 💅

The default is 17 messages.

  • miss_count_const — this defines the max number of times, on receipt of a token, that a message is checked for retransmission before a retransmission actually happens. This parameter is worth tweaking for switches that delay multicast packets compared to unicast packets. (The default setting works great for basically all modern switches, ngl.)

The default is 5 messages.

  • knet_pmtud_interval — how often the KNET PMTUd runs to look for network MTU changes. Value in seconds, default: 30.
  • knet_mtu — switch between manual and automatic MTU discovery. A value of 0 means automatic; any other value sets a manual MTU. In a setup with multiple interfaces, please specify the lowest MTU of the selected interfaces. 🧚

The default value is 0.

  • block_unlisted_ips — lets UDPU and KNET drop packets from IP addresses that corosync doesn't know about (nodes that don't exist in the nodelist). Value is yes or no.

This feature is mainly to protect against nodes with outdated configurations joining after a cluster split. Another use case is allowing the atomic merge of two independent clusters. ✨

Changing the default value is not recommended — the overhead is tiny, and an existing cluster may fail if corosync is started on an unlisted node with an old configuration.

The default value is yes.

  • cancel_token_hold_on_retransmit — lets Corosync hold the token by representative when there's too much retransmit traffic, so the network can process the increased load without getting overloaded. The mechanism used is the same as described for the hold directive.

Some deployments would rather never hold the token when there are retransmit messages — if that's you, set this option to yes.

The default value is no.

Within the logging directive there are several configuration options, and all of them are optional. 💖

The following 3 options are valid only for the top level logging directive:

  • timestamp — specifies that a timestamp gets placed on all log messages. It can be one of off (no timestamp), on (second precision timestamp) or hires (millisecond precision timestamp — only when supported by LibQB).

The default is hires (or on if hires is not supported).

  • fileline — specifies that file and line should be printed.

The default is off.

  • function_name — specifies that the code function name should be printed.

The default is off.

  • blackbox — specifies that blackbox functionality should be enabled.

The default is on.

The following options are valid both for the top level logging directive and they can be overridden in logger_subsys entries. 👑

  • to_stderr, to_logfile, to_syslog — these specify the destination of logging output. Any combination of these options may be specified. Valid options are yes and no.

The default is syslog and stderr.

Please note: if you're using to_logfile and you want to rotate the file, use logrotate(8) with the option copytruncate. eg.

/var/log/corosync.log {
     missingok
     compress
     notifempty
     daily
     rotate 7
     copytruncate
}
  • logfile — if the to_logfile directive is set to yes, this option specifies the pathname of the log file.

No default.

  • logfile_priority — specifies the logfile priority for this particular subsystem. Ignored if debug is on. Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

  • syslog_facility — specifies the syslog facility type used for any messages sent to syslog. options are daemon, local0, local1, local2, local3, local4, local5, local6 & local7.

The default is daemon.

  • syslog_priority — specifies the syslog level for this particular subsystem. Ignored if debug is on. Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

  • debug — specifies whether debug output is logged for this particular logger. It can also contain the value trace, which is the highest level of debug information.

The default is off.

Within the logging directive, logger_subsys directives are optional. 🎀

Within the logger_subsys sub-directive, all of the above logging configuration options are valid and can be used to override the default settings. The subsys entry, described below, is mandatory to identify the subsystem.

  • subsys — specifies the subsystem identity (name) for which logging is specified. This is the name used by a service in the log_init() call. E.g. 'CPG'. This directive is required.

Within the quorum directive it's possible to specify the quorum configuration options. The following option is required to activate the quorum service:

  • provider — specifies the algorithm to use. At the time of writing only corosync_votequorum is supported. See votequorum(5) for configuration options.

Within the nodelist directive it's possible to specify specific information about nodes in the cluster. 🌈 The directive can contain only the node sub-directive, which specifies every node that should be a member of the membership, and where non-default options are needed. Every node must have at least the ring0_addr field filled in.

Every node that should be a member of the membership must be specified.

Possible options are:

  • ringX_addr — specifies the IP or network hostname address of the particular node. X is a link number.
  • nodeid — this configuration option is required for each node for Kronosnet mode. It's a 32 bit value specifying the node identifier delivered to the cluster membership service. The node identifier value of zero is reserved and should not be used. If KNET is set, this field must be set.
  • name — this option is used mainly with the KNET transport to identify the local node. It's also used by client software (pacemaker). The algorithm for identifying the local node is as follows:
    1. Looks up $HOSTNAME in the nodelist
    2. If this fails, strip the domain name from $HOSTNAME and look that up in the nodelist
    3. If this fails, look in the nodelist for a fully-qualified name whose short version matches the short version of $HOSTNAME
    4. If all this fails, then search the interfaces list for an address that matches a name in the nodelist

Within the system directive it's possible to specify system options. 🦋

Possible options are:

  • qb_ipc_type — specifies the type of IPC to use. Can be one of native (default), shm and socket. native means one of shm or socket, depending on what's supported by the OS. On systems with support for both, SHM is selected. SHM is generally faster, but it needs to allocate a ring buffer file in /dev/shm.
  • sched_rr — should be set to yes (default) if corosync should try to set round robin realtime scheduling with maximal priority to itself. When setting of the scheduler fails, it falls back to setting maximal priority.
  • priority — sets the priority of the corosync process. Valid only when sched_rr is set to no. Can be either a numeric value with similar meaning as nice(1) or max / min meaning maximal / minimal priority (so minimal / maximal nice value).
  • move_to_root_cgroup — can be one of yes (Corosync always moves itself to root cgroup), no (Corosync never tries to move itself to root cgroup) or auto (Corosync first checks if sched_rr is enabled, and if so, it tries to set round robin realtime scheduling with maximal priority to itself. If setting of priority fails, corosync tries to move itself to root cgroup and retries setting of priority).

This feature is available only for systems with cgroups v1 with RT sched enabled — that's Linux with the CONFIG_RT_GROUP_SCHED kernel option — and cgroups v2.

ngl it's worth noting that currently (May 3 2021) cgroup2 doesn't yet support control of realtime processes, and the cpu controller can only be enabled when all RT processes are in the root cgroup (and yes, that only applies for kernels with CONFIG_RT_GROUP_SCHED enabled). So when move_to_root_cgroup is disabled, the kernel is compiled with CONFIG_RT_GROUP_SCHED and systemd is used, it may be impossible to make systemd options like CPUQuota work correctly until corosync is stopped. 😭

Also, when moving to the root cgroup is enforced and used together with cgroup2 and systemd, it makes it impossible (most of the time) for journald to add systemd-specific metadata (most importantly _SYSTEMD_UNIT) properly, because corosync is moved out of the cgroup created by systemd. Which means you can't filter corosync logged messages based on that metadata (like using -u or the _SYSTEMD_UNIT=UNIT pattern), and running systemctl status doesn't display (all) corosync log messages either. 💀 And it gets worse: journald caches pid for some time (approx. 5 sec) so the initial corosync messages have correct metadata. Sneaky.

  • allow_knet_handle_fallback — if KNET handle creation fails using privileged operations, allow fallback to creating the KNET handle using unprivileged operations. Defaults to no, meaning if privileged KNET handle creation fails, corosync will refuse to start.

The KNET handle will always be created using privileged operations if possible — setting this to yes only allows fallback to unprivileged operations. That fallback may result in performance issues, but if you're running in an unprivileged environment, e.g. as a normal user or in an unprivileged container, this may be required. ✨

  • state_dir — existing directory where corosync should chdir into. Corosync stores important state files and blackboxes there. 🗂️

The default is the value of the environment variable STATE_DIRECTORY or /var/lib/corosync.

🎀 Options inside the resources directive 🎀

Within the resources directive it is possible to specify options for resources.

Possible option is:

  • watchdog_device — (valid only if Corosync was compiled with watchdog support) the watchdog device to use, for example /dev/watchdog. If unset, empty or "off", no watchdog is used.

In a cluster with properly configured power fencing a watchdog provides no additional value. On the other hand, slow watchdog communication may incur multi-second delays in the Corosync main loop, potentially breaking down membership. 👀 IPMI watchdogs are particularly notorious in this regard: read about kipmid_max_busy_us in IPMI.txt in the Linux kernel documentation.

🦋 The nozzle directive 🦋

Within the nozzle directive it is possible to specify options for a libnozzle device. This is a pseudo ethernet device that routes network traffic through a channel on the corosync KNET network (NOT cpg or any corosync internal service) to other nodes in the cluster. This allows applications to take advantage of KNET features such as multipathing, automatic failover, link switching etc. Note that libnozzle is not a reliable transport, but you can tunnel TCP through it for reliable communications.

libnozzle also supports optional interface up/down scripts that are kept under a /etc/corosync/updown.d/ directory. See the KNET documentation for more information.

Only one nozzle device is allowed. Periodt.

The nozzle stanza takes several options:

  • name — the name of the network device to be created. On Linux this may be any name at all, other platforms have restrictions on the name.
  • ipaddr — the IP address (IPv6 or IPv4) of the interface. The bottom part of this address will be replaced by the local node's nodeid in conjunction with ipprefix. So, e.g. ipaddr: 192.168.1.0 ipprefix: 24 will make nodeids 1, 2, 5 use IP addresses 192.168.1.1, 192.168.1.2 & 192.168.1.5. If a prefix length of 16 is used then the bottom two bytes will be filled in with nodeid numbers. IPv6 addresses must end in '::', the nodeid will be added after the two colons to make the local IP address. Only one IP address is currently supported in the corosync.conf file. Additional IP addresses can be added in the ifup script if necessary.
  • ipprefix — specifies the IP address prefix for the nozzle device (see above).
  • macaddr — specifies the MAC address prefix for the nozzle device. As for the IP address, the bottom part of the MAC address will be filled in with the node id. In this case no prefix applies — the bottom two bytes of the MAC address will always be overwritten with the node id. So specifying macaddr: 54:54:12:24:12:12 on nodeid 1 will result in it having a MAC address of 54:54:12:24:00:01. 💅

🌈 TO ADD A NEW NODE TO THE CLUSTER 🌈

For example, to add a node with address 10.24.38.108 with nodeid 3. The node has the name NEW (in DNS or /etc/hosts) and is not currently running corosync. The current corosync.conf nodelist looks like this:

nodelist {
    node {
        nodeid: 1
        ring0_addr: 10.24.38.101
        name: node1
    }
    node {
        nodeid: 2
        ring0_addr: 10.24.38.102
        name: node2

    }
}

Add a new entry for the node below the existing nodes. Node entries don't have to be in nodeid order, but it will help keep you sane. 🧠 So the nodelist now looks like this:

nodelist {
    node {
        nodeid: 1
        ring0_addr: 10.24.38.101
        name: node1
    }
    node {
        nodeid: 2
        ring0_addr: 10.24.38.102
        name: node2

    }
    node {
        nodeid: 3
        ring0_addr: 10.24.38.108
        name: NEW

    }
}

This file must then be copied onto all three nodes — the existing two nodes, and the new one. On one of the existing corosync nodes, tell corosync to re-read the updated config file into memory:

corosync-cfgtool -R

This command only needs to be run on one node in the cluster. You may then start corosync on the NEW node and it should join the cluster. If this doesn't work as expected, then check that communications between all three nodes is working, and check the syslog files on all nodes for more information. 🕵️ It's important to note that the key bit of information about a node failing to join might be on a different node than you expect. Rude, but true.

🌸 TO REMOVE A NODE FROM THE CLUSTER 🌸

This is the reverse procedure to 'Adding a node' above. First you need to shut down the node you will be removing from the cluster.

corosync-cfgtool -H

Then delete the nodelist stanza from corosync.conf and finally update corosync on the remaining nodes by running

corosync-cfgtool -R

on one of them. 🫶

💜 ADDRESS RESOLUTION 💜

corosync resolves ringX_addr names/IP addresses using the getaddrinfo(3) call with respect of the totem.ip_version setting.

getaddrinfo() uses a sophisticated algorithm to sort node addresses into a preferred order, and corosync always chooses the first address in that list of the required family. Which means it is essential that your DNS or /etc/hosts files are correctly configured so that all addresses for ringX appear on the same network (or are reachable with minimal hops) and over the same IP protocol. If this is not the case then some nodes might not be able to join the cluster. 😬 It is possible to override the search order used by getaddrinfo() using the configuration file /etc/gai.conf(5) if necessary, but this is not recommended.

If there is any doubt about the order of addresses returned from getaddrinfo(), then it might be simpler to just use IP addresses (v4 or v6) in the ringX_addr field. fr fr, saves you so much pain.

🍓 FILES 🍓

  • /etc/corosync/corosync.conf — the corosync executive configuration file.

⭐ SEE ALSO ⭐

corosync_overview(7), votequorum(5), corosync-qdevice(8), logrotate(8) getaddrinfo(3) gai.conf(5)

okay bestie that's the whole page, you made it to the end, I'm so proud of you 🎉💖 This one's from the corosync Man Page, dated 2025-06-12, and it lives at COROSYNC_CONF(5) — go configure your cluster, you've got this, slay 👑✨

📦 where this page comes from