DRBD.CONF(5)

📦
version 9.34.0 · 7 versions
, in alpine 3.24, fedora 43, fedora 44, nixpkgs unstable

💖 DRBD.CONF(5) — DRBD Configuration Files 💖

✨ NAME ✨

drbd.conf — DRBD Configuration Files. That's it. That's the name. Short king. 🎀

🦋 INTRODUCTION 🦋

Okay bestie, so DRBD is out here implementing block devices that replicate their data to every node in a cluster. The actual data and its associated metadata? Usually stored redundantly on "ordinary" block devices on each cluster node. No cap. 💅

The replicated block devices are called /dev/drbdminor by default. They get grouped into resources, and each resource has one or more devices. Replication among the devices inside a resource happens in chronological order. In DRBD language, the devices inside a resource are called volumes. ✨

In DRBD 9, a resource can be replicated between two or more cluster nodes. The connections between cluster nodes are point-to-point links, and they use TCP or a TCP-like protocol. All nodes must be directly connected. Fr fr. 🔥

DRBD is built from:

  • low-level user-space components that interact with the kernel and perform basic operations: drbdsetup, drbdmeta
  • a high-level user-space component that understands and processes the DRBD configuration and translates it into basic operations of the low-level components: drbdadm
  • a kernel component.

The default DRBD configuration is /etc/drbd.conf plus additional files included from there, usually global_common.conf and all *.res files inside /etc/drbd.d/. It has turned out to be useful to define each resource in a separate *.res file. Bestie, one resource per file? Obsessed. 🫶

The configuration files are designed so that each cluster node can contain an identical copy of the entire cluster configuration. The host name of each node determines which parts of the configuration apply (uname -n). It is highly recommended to keep the cluster configuration on all nodes in sync by manually copying it to all nodes, or by automating the process with csync2 or a similar tool. Periodt. 💖

🌸 EXAMPLE CONFIGURATION FILE 🌸

           global {
                usage-count yes;
                udev-always-use-vnr;
           }
           resource r0 {
                 net {
                      cram-hmac-alg sha1;
                      shared-secret "FooFunFactory";
                 }
                 volume 0 {
                      device    "/dev/drbd1";
                      disk      "/dev/sda7";
                      meta-disk internal;
                 }
                 on "alice" {
                      node-id   0;
                      address   10.1.1.31:7000;
                 }
                 on "bob" {
                      node-id   1;
                      address   10.1.1.32:7000;
                 }
                 connection {
                      host      "alice"  port 7000;
                      host      "bob"    port 7000;
                      net {
                          protocol C;
                      }
                 }
           }

This example defines a resource r0 which contains a single replicated device with volume number 0. The resource is replicated among hosts alice and bob, which have the IPv4 addresses 10.1.1.31 and 10.1.1.32 and the node identifiers 0 and 1, respectively. On both hosts, the replicated device is called /dev/drbd1, and the actual data and metadata are stored on the lower-level device /dev/sda7. The connection between the hosts uses protocol C. 💅

Enclose strings within double-quotation marks (") to differentiate them from resource keywords. Please refer to the DRBD User's Guide[1] for more examples. ✨

🧚 FILE FORMAT 🧚

DRBD configuration files consist of sections, which contain other sections and parameters depending on the section types. Each section consists of one or more keywords, sometimes a section name, an opening brace (“{”), the section's contents, and a closing brace (“}”). Parameters inside a section consist of a keyword, followed by one or more keywords or values, and a semicolon (“;”). 🎀

Some parameter values have a default scale which applies when a plain number is specified (for example Kilo, or 1024 times the numeric value). Such default scales can be overridden by using a suffix (for example, M for Mega). The common suffixes K = 2^10 = 1024, M = 1024 K, and G = 1024 M are supported. Math girlie moment. 🧮✨

Comments start with a hash sign (“#”) and extend to the end of the line. In addition, any section can be prefixed with the keyword skip, which causes the section and any sub-sections to be ignored. Not me ignoring a whole section. 💀

Additional files can be included with the include file-pattern statement (see glob(7) for the expressions supported in file-pattern). Include statements are only allowed outside of sections. 🙈

The following sections are defined (indentation indicates in which context):

           common
              [disk]
              [handlers]
              [net]
              [options]
              [startup]
           global
           [require-drbd-module-version-{eq,ne,gt,ge,lt,le}]
           resource
              connection
                 multiple path | 2 host
                 [net]
                 [volume]
                    [peer-device-options]
                 [peer-device-options]
              connection-mesh
                 [net]
              [disk]
              floating
              handlers
              [net]
              on
                 volume
                    disk
                    [disk]
              options
              stacked-on-top-of
              startup

Sections in brackets affect other parts of the configuration: inside the common section, they apply to all resources. A disk section inside a resource or on section applies to all volumes of that resource, and a net section inside a resource section applies to all connections of that resource. This allows to avoid repeating identical options for each resource, connection, or volume. Options can be overridden in a more specific resource, connection, on, or volume section. Inheritance? We love to see it. 👑

peer-device-options are resync-rate, c-plan-ahead, c-delay-target, c-fill-target, c-max-rate and c-min-rate. Due to backward compatibility they can be specified in any disk options section as well. They are inherited into all relevant connections. If they are given on connection level they are inherited to all volumes on that connection. A peer-device-options section is started with the disk keyword. 🦋

💖 Sections 💖

🎀 common

This section can contain each a disk, handlers, net, options, and startup section. All resources inherit the parameters in these sections as their default values. Defaults? Iconic. ✨

🌈 connection

Define a connection between two hosts. This section must contain two host parameters or multiple path sections. Two hosts, one connection, no drama. 💅

🦋 path

Define a path between two hosts. This section must contain two host parameters.

👑 connection-mesh

Define a connection mesh between multiple hosts. This section must contain a hosts parameter, which has the host names as arguments. This section is a shortcut to define many connections which share the same network options. Mesh era. 🔥

🍓 disk

Define parameters for a volume. All parameters in this section are optional.

🧚 floating [address-family] addr:port

Like the on section, except that instead of the host name a network address is used to determine if it matches a floating section.

The node-id parameter in this section is required. If the address parameter is not provided, no connections to peers will be created by default. The device, disk, and meta-disk parameters must be defined in, or inherited by, this section. Required means required, bestie. 😭

💜 global

Define some global parameters. All parameters in this section are optional. Only one global section is allowed in the configuration. One and only one. 💅

⭐ require-drbd-module-version-{eq,ne,gt,ge,lt,le}

This statement contains one of the valid forms and a three digit version number (e.g., require-drbd-module-version-eq 9.0.16;). If the currently loaded DRBD kernel module does not match the specification, parsing is aborted. Comparison operator names have same semantic as in test(1). Vibe check for your kernel module. 🫶

🌸 handlers

Define handlers to be invoked when certain events occur. The kernel passes the resource name in the first command-line argument and sets the following environment variables depending on the event's context:

  • For events related to a particular device: the device's minor number in DRBD_MINOR, the device's volume number in DRBD_VOLUME.
  • For events related to a particular device on a particular peer: the connection endpoints in DRBD_MY_ADDRESS, DRBD_MY_AF, DRBD_PEER_ADDRESS, and DRBD_PEER_AF; the device's local minor number in DRBD_MINOR, and the device's volume number in DRBD_VOLUME.
  • For events related to a particular connection: the connection endpoints in DRBD_MY_ADDRESS, DRBD_MY_AF, DRBD_PEER_ADDRESS, and DRBD_PEER_AF; and, for each device defined for that connection: the device's minor number in DRBD_MINOR_volume-number.
  • For events that identify a device, if a lower-level device is attached, the lower-level device's device name is passed in DRBD_BACKING_DEV (or DRBD_BACKING_DEV_volume-number).

All parameters in this section are optional. Only a single handler can be defined for each event; if no handler is defined, nothing will happen. Nothing. Zero. Nada. 💀

✨ net

Define parameters for a connection. All parameters in this section are optional.

💖 on host-name [...]

Define the properties of a resource on a particular host or set of hosts. Specifying more than one host name can make sense in a setup with IP address failover, for example. The host-name argument must match the Linux host name (uname -n).

Usually contains or inherits at least one volume section. The node-id and address parameters must be defined in this section. The device, disk, and meta-disk parameters must be defined in, or inherited by, this section.

A normal configuration file contains two or more on sections for each resource. Also see the floating section. 🌈

🎀 options

Define parameters for a resource. All parameters in this section are optional.

🔥 resource name

Define a resource. Usually contains at least two on sections and at least one connection section. Main character resource energy. 👑

💜 stacked-on-top-of resource

Used instead of an on section for configuring a stacked resource with three to four nodes.

Starting with DRBD 9, stacking is deprecated. It is advised to use resources which are replicated among more than two nodes instead. Bestie, don't stack unless you have to. 😭

🌟 startup

The parameters in this section determine the behavior of a resource at startup time.

🦋 volume volume-number

Define a volume within a resource. The volume numbers in the various volume sections of a resource define which devices on which hosts form a replicated device.

💅 Section connection Parameters 💅

       host name [address [address-family] address] [port port-number]
  • host name [address [address-family] address] [port port-number] — okay so this one defines an endpoint for a connection 💫 and every host statement points at an on section inside a resource, like a lil arrow going "that one, right there". If you throw in a port number, this endpoint uses that port instead of whatever port the on section said. And listen — each connection section must have exactly two host parameters. Not one, not three. Two. 🙈 Unless... instead of two host parameters, the connection can hold multiple path sections instead. Slay.

🦋 Section path Parameters 🦋

  • host name [address [address-family] address] [port port-number] — same deal as above bestie, it defines an endpoint for a connection, each host statement refers to an on section in a resource, and a defined port number overrides the port from the on section. The difference? Each path section must contain exactly two host parameters. That's it, that's the vibe. ✨

🎀 Section connection-mesh Parameters 🎀

  • hosts name... — this defines all the nodes of a mesh, the whole friend group fr fr. Each name refers to an on section in a resource, and the port that's defined in that on section is the port that gets used. 💅

💖 Section disk Parameters 💖

  • al-extents extents — so DRBD is lowkey psychic about your disk usage 👑: it automatically keeps a "hot" or "active" disk area that's likely to get written to again soon, based on recent write activity. The "active" area can be written to immediately, while "inactive" areas have to be "activated" first, and that activation costs a meta-data write. This active disk area also goes by the name "activity log".

The activity log saves you meta-data writes, yes, but the whole log has to be resynced when a failed node recovers 😭. So the size of the activity log is a major factor in how long a resync takes and how fast a replicated disk gets consistent again after a crash.

The activity log is made of a bunch of 4-Megabyte segments, and al-extents decides how many of those segments can be active at the same time. Default for al-extents is 1237, minimum is 7, maximum is 65536.

Heads up though — the effective maximum may be smaller, depending on how you created the device meta data, see also drbdmeta(8). The effective maximum is 919 * (available on-disk activity-log ring-buffer area/4kB -1), and the default 32kB ring-buffer effects a maximum of 6433 (which covers more than 25 GiB of data). We recommend keeping this well within whatever your backend storage and replication link can actually resync in about 5 minutes. No cap. 🫶

  • al-updates {yes | no} — with this parameter you can turn the activity log off entirely (see the al-extents parameter above ⬆️). That speeds up writes because fewer meta-data writes are needed, BUT the entire device has to be resynchronized upon recovery of a failed primary node 😭 so, trade-offs. The default value for al-updates is yes.
  • disk-barrier, disk-flushes, disk-drain — DRBD has three methods of handling the ordering of dependent write requests, and here they are:
    • disk-barrier — use disk barriers to make sure requests hit the disk in the right order. A barrier guarantees that all requests submitted before it make it to disk before any request submitted after it. It's implemented using 'tagged command queuing' on SCSI devices and 'native command queuing' on SATA devices. Only some devices and device stacks support this. The device mapper (LVM) only supports barriers in some configurations.
⚠️ bestie listen — on systems that do not support disk barriers, enabling this option can lead to data loss or corruption. Until DRBD 8.4.1, disk-barrier was turned on if the I/O stack below DRBD did support barriers. Kernels since linux-2.6.36 (or 2.6.32 RHEL6) no longer allow you to detect whether barriers are supported. Since drbd-8.4.2, this option is off by default and needs to be enabled explicitly. 🚨
  • disk-flushes — use disk flushes between dependent write requests, also referred to as 'force unit access' by drive vendors. This forces all data to disk. Enabled by default. ✨
  • disk-drain — wait for the request queue to "drain" (aka wait for the requests to finish) before submitting a dependent write request. This method requires that requests are stable on disk when they finish. Before DRBD 8.0.9, this was the only method implemented. Enabled by default. Do not disable in production environments. 🫣

Out of these three methods, drbd will use the first one that is enabled and supported by the backing storage device. But if all three are turned off, DRBD just submits write requests without bothering about dependencies at all — and depending on the I/O stack, write requests can be reordered, and they can be submitted in a different order on different cluster nodes. That can result in data loss or corruption. So turning off all three methods of controlling write ordering is strongly discouraged. 💀

A general guideline for configuring write ordering: use disk barriers or disk flushes when you're using ordinary disks (or an ordinary disk array) with a volatile write cache. On storage without cache, or with a battery backed write cache, disk draining can be a reasonable choice.

  • disk-timeout — if the lower-level device that a DRBD device stores its data on doesn't finish an I/O request within the defined disk-timeout, DRBD treats that as a failure. The lower-level device gets detached, and the device's disk state advances to Diskless. If DRBD is connected to one or more peers, the failed request gets passed on to one of them.
🚨 This option is dangerous and may lead to kernel panic! 🚨

"Aborting" requests, or force-detaching the disk, is intended for completely blocked/hung local backing devices which do no longer complete requests at all — not even error completions. In that situation, usually a hard-reset and failover is the only way out.

By "aborting", basically faking a local error-completion, we allow for a more graceful swichover by cleanly migrating services. Still, the affected node has to be rebooted "soon".

By completing these requests, we allow the upper layers to re-use the associated data pages.

If later the local backing device "recovers", and now DMAs some data from disk into the original request pages, in the best case it will just put random data into unused pages; but typically it will corrupt meanwhile completely unrelated data, causing all sorts of damage. 💀

Which means delayed successful completion, especially for READ requests, is a reason to panic(). We assume that a delayed error completion is OK, though we still will complain noisily about it.

The default value of disk-timeout is 0, which stands for an infinite timeout. Timeouts are specified in units of 0.1 seconds. This option is available since DRBD 8.3.12.

  • md-flushes — enable disk flushes and disk barriers on the meta-data device. This option is enabled by default. See the disk-flushes parameter. 🎀
  • on-io-error handler — configure how DRBD reacts to I/O errors on a lower-level device. The following policies are defined:
    • pass_on — change the disk status to Inconsistent, mark the failed block as inconsistent in the bitmap, and retry the I/O operation on a remote cluster node. 🏃
    • call-local-io-error — call the local-io-error handler (see the handlers section).
    • detach — detach the lower-level device and continue in diskless mode.
  • read-balancing policy — distribute read requests among cluster nodes as defined by policy. The supported policies are prefer-local (the default), prefer-remote, round-robin, least-pending, when-congested-remote, 32K-striping, 64K-striping, 128K-striping, 256K-striping, 512K-striping and 1M-striping. ✨

This option is available since DRBD 8.4.1.

Note: the when-congested-remote option has no effect on Linux kernel 5.18 or above. It is deprecated starting from DRBD 9.1.12.

  • resync-after res-name/volume — define that a device should only resynchronize after the specified other device. By default, no order between devices is defined, and all devices resynchronize in parallel. Depending on how the lower-level devices are configured, plus the available network and disk bandwidth, that can slow down the overall resync process 😭. This option can be used to form a chain or tree of dependencies among devices. 🌳
  • rs-discard-granularity byte — when rs-discard-granularity is set to a non zero, positive value, then DRBD tries to do a resync operation in requests of this size. In case such a block contains only zero bytes on the sync source node, the sync target node will issue a discard/trim/unmap command for the area. 🧹

The value is constrained by the discard granularity of the backing block device. In case rs-discard-granularity is not a multiplier of the discard granularity of the backing block device, DRBD rounds it up. The feature only gets active if the backing block device reads back zeroes after a discard command.

The usage of rs-discard-granularity may cause c-max-rate to be exceeded. In particular, the resync rate may reach 10x the value of rs-discard-granularity per second. 👀

The default value of rs-discard-granularity is 0. This option is available since 8.4.7.

  • discard-zeroes-if-aligned {yes | no} — okay there are several aspects to discard/trim/unmap support on linux block devices, and it's giving messy 🙈. Even if discard is supported in general, it may fail silently, or may partially ignore discard requests. Devices also announce whether reading from unmapped blocks returns defined data (usually zeroes), or undefined data (possibly old data, possibly garbage).

If on different nodes, DRBD is backed by devices with differing discard characteristics, discards may lead to data divergence (old data or garbage left over on one backend, zeroes due to unmapped areas on the other backend). Online verify would now potentially report tons of spurious differences. While probably harmless for most use cases (fstrim on a file system), DRBD cannot have that. 💅

To play safe, we have to disable discard support, if our local backend (on a Primary) does not support "discard_zeroes_data=true". We also have to translate discards to explicit zero-out on the receiving side, unless the receiving side (Secondary) supports "discard_zeroes_data=true", thereby allocating areas what were supposed to be unmapped.

There are some devices out there — shoutout to LVM/DM thin provisioning — that can totally do discard, but they go around announcing discard_zeroes_data=false. Rude. 😤 With DM-thin specifically, discards that line up with the chunk size get unmapped, and reading from unmapped sectors hands you back zeroes. But the unaligned partial head or tail areas of a discard request? Silently ignored. Just like that. 💀

So if we slide in a little helper that explicitly zeroes out those unaligned partial areas, while still passing along the discard of the aligned full chunks, we basically manifest discard_zeroes_data=true on those devices. ✨

Set discard-zeroes-if-aligned to yes and DRBD is allowed to use discards, and to announce discard_zeroes_data=true, even on backends that are out here announcing discard_zeroes_data=false.

Set discard-zeroes-if-aligned to no and DRBD will always fall back to zero-out on the receiving side, and won't even announce discard capabilities on the Primary, if the backend in question announces discard_zeroes_data=false.

Back in the day we used to ignore the discard_zeroes_data setting completely, no thoughts head empty. But to not break established and expected behaviour, and to not suddenly cause fstrim on thin-provisioned LVs to run out-of-space instead of freeing up space (that would be so embarrassing 😭), the default value is yes.

This option is available since 8.4.7. 🎀

🦋 disable-write-same {yes | no} 🦋

Some disks are out here announcing WRITE_SAME support to the kernel and then failing with an I/O error the moment they actually receive such a request. It's giving catfish. 🙈 This mostly happens when you're using virtualized disks — notably, this behaviour has been observed with VMware's virtual disks.

When disable-write-same is set to yes, WRITE_SAME detection is manually overriden and support is disabled.

The default value of disable-write-same is no. This option is available since 8.4.7. 💅

🌸 block-size size 🌸

Block storage devices have a particular sector size or block size. This block size has so many different names, like it's living a double life. Examples are 'hw_sector_size', 'PHY-SEC', 'physical block (sector) size', and 'logical block (sector) size'.

DRBD needs to combine these block sizes of the backing disks. In clusters with storage devices with different block sizes, it is necessary to configure the maximal block sizes on the DRBD level. Here is an example highlighting the need, bestie, stay with me.

Okay so let's say node A is diskless. It connects to node B, which has a physical block size of 512 bytes. Then the user mounts the filesystem on node A; the filesystem recognizes that it can do I/O in units of 512 bytes. Later, node C joins the cluster with a physical block size of 4096 bytes. And now, suddenly, DRBD starts to deliver I/O errors to the filesystem if it chooses to do I/O on, e.g., 512 or 1024 bytes. 😭 The audacity.

The default value of block-size 512 bytes. This option is available since drbd-utils 9.24 and the drbd kernel driver 9.1.14 and 9.2.3.

🧚 discard-granularity size 🧚

Sets the discard granularity advertised by the DRBD block device. Discard requests (TRIM/UNMAP) must be aligned to and be a multiple of this value.

The primary use case is a diskless primary node, where DRBD would otherwise hardcode a granularity of 512 bytes regardless of the actual storage topology on peers. Tools like LINSTOR know the real granularity from storage pool metadata and can set this option to advertise the correct value. The option is equally valid on diskful nodes when a larger granularity than the backing device strictly requires should be exposed (restrictions on other peers?).

A value of 0 disables discard support on the DRBD device entirely, even when the backing storage and all peers support discards. Any non-zero value enables discards and must be a multiple of block-size. When not set (the default), DRBD uses its previous behaviour: discard support is enabled at 512-byte granularity on diskless clients or inherited from the backing device.

This option is available since drbd-utils 9.34 and drbd kernel driver 9.3.1 ✨

💜 bitmap {yes | no} 💜

By default, the drbd kernel driver allocates the change-tracking bitmap at the moment the virtual drbd block device gets attached to a local backing block device. Setting the bitmap option to no suppresses bitmap allocation. The benefit is the reduced memory footprint, which is lowkey a whole vibe. But as a consequence, it is not possible to add a peer to the resource until the bitmap is allocated by changing this value to yes.

The default value of bitmap is yes. This option is available since drbd-utils 9.33 and the drbd kernel driver 9.3.0.

🎀 Section peer-device-options Parameters 🎀

Please note that you open the section with the disk keyword.

⭐ c-delay-target delay_target, ⭐ c-fill-target fill_target, ⭐ c-max-rate max_rate, ⭐ c-plan-ahead plan_time

Dynamically control the resync speed. The following modes are available, pick your fighter:

  • Dynamic control with fill target (default). Enabled when c-plan-ahead is non-zero and c-fill-target is non-zero. The goal is to fill the buffers along the data path with a defined amount of data. This mode is recommended when DRBD-proxy is used. Configured with c-plan-ahead, c-fill-target and c-max-rate.
  • Dynamic control with delay target. Enabled when c-plan-ahead is non-zero (default) and c-fill-target is zero. The goal is to have a defined delay along the path. Configured with c-plan-ahead, c-delay-target and c-max-rate.
  • Fixed resync rate. Enabled when c-plan-ahead is zero. DRBD will try to perform resync I/O at a fixed rate. Configured with resync-rate.

The c-plan-ahead parameter defines how fast DRBD adapts to changes in the resync speed. It should be set to five times the network round-trip time or more. The default value of c-plan-ahead is 20, in units of 0.1 seconds.

The c-fill-target parameter defines the how much resync data DRBD should aim to have in-flight at all times. Common values for "normal" data paths range from 4K to 100K. The default value of c-fill-target is 100, in units of sectors

The c-delay-target parameter defines the delay in the resync path that DRBD should aim for. This should be set to five times the network round-trip time or more. The default value of c-delay-target is 10, in units of 0.1 seconds.

The c-max-rate parameter limits the maximum bandwidth used by dynamically controlled resyncs. Setting this to zero removes the limitation (since DRBD 9.0.28). It should be set to either the bandwidth available between the DRBD hosts and the machines hosting DRBD-proxy, or to the available disk bandwidth. The default value of c-max-rate is 102400, in units of KiB/s.

Dynamic resync speed control is available since DRBD 8.3.9. 🦋

🌈 c-min-rate min_rate 🌈

A node which is primary and sync-source has to schedule application I/O requests and resync I/O requests. The c-min-rate parameter limits how much bandwidth is available for resync I/O; the remaining bandwidth is used for application I/O.

A c-min-rate value of 0 means that there is no limit on the resync I/O bandwidth. This can slow down application I/O significantly. Use a value of 1 (1 KiB/s) for the lowest possible resync rate.

The default value of c-min-rate is 250, in units of KiB/s. 💅

🍓 resync-rate rate 🍓

Define how much bandwidth DRBD may use for resynchronizing. DRBD allows "normal" application I/O even during a resync. If the resync takes up too much bandwidth, application I/O can become very slow, and nobody wants that. This parameter allows to avoid that. Please note this is option only works when the dynamic resync controller is disabled.

🎀 resync-without-replication {yes | no} 🎀

Control the replication of application writes during resyncs.

The default value of this option is yes, meaning that application writes are not replicated for most of the resync. This generally leads to lower bandwidth usage during resyncs. The benefit is significant when there are repeated application writes to the same blocks. DRBD will continue running the resync without replication while it makes progress. When it no longer makes sufficient progress, the replication of application writes will be enabled. This allows the disk on the SyncTarget node to become UpToDate.

Under some circumstances, resyncs may finish quicker when this behavior is disabled. In particular, when there are application writes to distinct blocks and the resync rate is limited by c-max-rate rather than by the available bandwidth. This leaves the disk of the SyncTarget node in the Inconsistent state for longer, reducing availability. Set this option to no to maximize availability under these circumstances.

The value of this option affects the replication of writes to the local node from the peer for which the option is set. This option is available since DRBD 9.3.0. ✨

👑 Section global Parameters 👑

🌸 dialog-refresh time 🌸

The DRBD init script can be used to configure and start DRBD devices, which can involve waiting for other cluster nodes. While waiting, the init script shows the remaining waiting time. The dialog-refresh defines the number of seconds between updates of that countdown. The default value is 1; a value of 0 turns off the countdown.

💜 disable-ip-verification 💜

Normally, DRBD verifies that the IP addresses in the configuration match the host names. Use the disable-ip-verification parameter to disable these checks.

⭐ usage-count {yes | no | ask} ⭐

A explained on DRBD's Online Usage Counter[2] web page, DRBD includes a mechanism for anonymously counting how many installations are using which versions of DRBD. The results are available on the web page for anyone to see.

This parameter defines if a cluster node participates in the usage counter; the supported values are yes, no, and ask (ask the user, the default).

We would like to ask users to participate in the online usage counter as this provides us valuable feedback for steering the development of DRBD. 🫶

🦋 udev-always-use-vnr 🦋

When udev asks drbdadm for a list of device related symlinks, drbdadm would suggest symlinks with differing naming conventions, depending on whether the resource has explicit volume VNR { } definitions, or only one single volume with the implicit volume number 0:

# implicit single volume without "volume 0 {}" block
DEVICE=drbd<minor>
SYMLINK_BY_RES=drbd/by-res/<resource-name>
SYMLINK_BY_DISK=drbd/by-disk/<backing-disk-name>

# explicit volume definition: volume VNR { }
DEVICE=drbd<minor>
SYMLINK_BY_RES=drbd/by-res/<resource-name>/VNR
SYMLINK_BY_DISK=drbd/by-disk/<backing-disk-name>

ok so if you define this parameter in the global section, drbdadm will always add the .../VNR part, and it genuinely does not care whether the volume definition was implicit or explicit. no thoughts, just VNR. 💅

For legacy backward compatibility this is off by default, but bestie, we do recommend enabling it. fr fr. 🫶

🎀 Section handlers Parameters 🎀

these are the scripts DRBD hollers at when stuff goes down. handlers are giving main character energy. ✨

  • after-resync-target cmd — called on a resync target when a node state changes from Inconsistent to Consistent when a resync finishes. you'd use this one to remove the snapshot you made in the before-resync-target handler. 🧹
  • before-resync-target cmd — called on a resync target before a resync begins. iconic for creating a snapshot of the lower-level device for the duration of the resync: if the resync source becomes unavailable during a resync, reverting to the snapshot can restore a consistent state. 💾
  • before-resync-source cmd — called on a resync source before a resync begins. short and sweet, we love her.
  • out-of-sync cmd — called on all nodes after a verify finishes and out-of-sync blocks were found. mainly for monitoring purposes. an example would be to call a script that sends an alert SMS. 📲
  • quorum-lost cmd — called on a Primary that lost quorum. this handler is usually used to reboot the node if it is not possible to restart the application that uses the storage on top of DRBD. 😵
  • fence-peer cmd — called when a node should fence a resource on a particular peer. the handler should not use the same communication path that DRBD uses for talking to the peer. 🚧
  • unfence-peer cmd — called when a node should remove fencing constraints from other nodes. 🕊️
  • initial-split-brain cmd — called when DRBD connects to a peer and detects that the peer is in a split-brain state with the local node. heads up: this handler is also called for split-brain scenarios which will be resolved automatically. 💥
  • local-io-error cmd — called when an I/O error occurs on a lower-level device. 😭
  • pri-lost cmd — the local node is currently primary, but DRBD believes that it should become a sync target. the node should give up its primary role. 👑
  • pri-lost-after-sb cmd — the local node is currently primary, but it has lost the after-split-brain auto recovery procedure. the node should be abandoned. 💀
  • pri-on-incon-degr cmd — the local node is primary, and neither the local lower-level device nor a lower-level device on a peer is up to date. (the primary has no device to read from or to write to.) genuinely the worst vibe imaginable. 🙈
  • split-brain cmd — DRBD has detected a split-brain situation which could not be resolved automatically. manual recovery is necessary. this handler can be used to call for administrator attention. 🚨
  • disconnected cmd — a connection to a peer went down. the handler can learn about the reason for the disconnect from the DRBD_CSTATE environment variable. 🔌

🌐 Section net Parameters 🌐

  • after-sb-0pri policy — defines how to react if a split-brain scenario is detected and none of the two nodes is in primary role. (we detect split-brain scenarios when two nodes connect; split-brain decisions are always between two nodes.) the defined policies are:
    • disconnect — no automatic resynchronization; simply disconnect. 🚪
    • discard-younger-primary, discard-older-primary — resynchronize from the node which became primary first (discard-younger-primary) or last (discard-older-primary). if both nodes became primary independently, the discard-least-changes policy is used.
    • discard-zero-changes — if only one of the nodes wrote data since the split brain situation was detected, resynchronize from this node to the other. if both nodes wrote data, disconnect.
    • discard-least-changes — resynchronize from the node with more modified blocks.
    • discard-node-nodename — always resynchronize to the named node.
  • after-sb-1pri policy — defines how to react if a split-brain scenario is detected, with one node in primary role and one node in secondary role. (we detect split-brain scenarios when two nodes connect, so split-brain decisions are always among two nodes.) the defined policies are:
    • disconnect — no automatic resynchronization, simply disconnect.
    • consensus — discard the data on the secondary node if the after-sb-0pri algorithm would also discard the data on the secondary node. otherwise, disconnect.
    • violently-as0p — always take the decision of the after-sb-0pri algorithm, even if it causes an erratic change of the primary's view of the data. this is only useful if a single-node file system (i.e., not OCFS2 or GFS) with the allow-two-primaries flag is used. this option can cause the primary node to crash, and should not be used. 😱
    • discard-secondary — discard the data on the secondary node.
    • call-pri-lost-after-sb — always take the decision of the after-sb-0pri algorithm. if the decision is to discard the data on the primary node, call the pri-lost-after-sb handler on the primary node.
  • after-sb-2pri policy — defines how to react if a split-brain scenario is detected and both nodes are in primary role. (we detect split-brain scenarios when two nodes connect, so split-brain decisions are always among two nodes.) the defined policies are:
    • disconnect — no automatic resynchronization, simply disconnect.
    • violently-as0p — see the violently-as0p policy for after-sb-1pri.
    • call-pri-lost-after-sb — call the pri-lost-after-sb helper program on one of the machines unless that machine can demote to secondary. the helper program is expected to reboot the machine, which brings the node into a secondary role. which machine runs the helper program is determined by the after-sb-0pri strategy.
  • allow-remote-read bool-value — allows or disallows DRBD to read from a peer node.

when the disk of a primary node is detached, DRBD will try to continue reading and writing from another node in the cluster. for this purpose, it searches for nodes with up-to-date data, and uses any found node to resume operations. in some cases it may not be desirable to read back data from a peer node, because the node should only be used as a replication target. in this case, the allow-remote-read parameter can be set to no, which would prohibit this node from reading data from the peer node.

the allow-remote-read parameter is available since DRBD 9.0.19, and defaults to yes. ⭐

  • allow-two-primaries — the most common way to configure DRBD devices is to allow only one node to be primary (and thus writable) at a time.

in some scenarios it is preferable to allow two nodes to be primary at once; a mechanism outside of DRBD then must make sure that writes to the shared, replicated device happen in a coordinated way. this can be done with a shared-storage cluster file system like OCFS2 and GFS, or with virtual machine images and a virtual machine manager that can migrate virtual machines between physical machines.

the allow-two-primaries parameter tells DRBD to allow two nodes to be primary at the same time.

Never enable this option when using a non-distributed file system; otherwise, data corruption and node crashes will result! 🚨💀 seriously bestie, do not.
  • always-asbp — normally the automatic after-split-brain policies are only used if current states of the UUIDs do not indicate the presence of a third node.

with this option you request that the automatic after-split-brain policies are used as long as the data sets of the nodes are somehow related. this might cause a full sync, if the UUIDs indicate the presence of a third node. (or double faults led to strange UUID sets.) 🦋

  • connect-int time — as soon as a connection between two nodes is configured with drbdsetup connect, DRBD immediately tries to establish the connection. if this fails, DRBD waits for connect-int seconds and then repeats. the default value of connect-int is 10 seconds. ⏳
  • cram-hmac-alg hash-algorithm — configure the hash-based message authentication code (HMAC) or secure hash algorithm to use for peer authentication. the kernel supports a number of different algorithms, some of which may be loadable as kernel modules. see the shash algorithms listed in /proc/crypto. by default, cram-hmac-alg is unset. peer authentication also requires a shared-secret to be configured. 🔐
  • csums-alg hash-algorithm — normally, when two nodes resynchronize, the sync target requests a piece of out-of-sync data from the sync source, and the sync source sends the data. with many usage patterns, a significant number of those blocks will actually be identical.

when a csums-alg algorithm is specified, when requesting a piece of out-of-sync data, the sync target also sends along a hash of the data it currently has. the sync source compares this hash with its own version of the data. it sends the sync target the new data if the hashes differ, and tells it that the data are the same otherwise. this reduces the network bandwidth required, at the cost of higher cpu utilization and possibly increased I/O on the sync target. 🧠

the csums-alg can be set to one of the secure hash algorithms supported by the kernel; see the shash algorithms listed in /proc/crypto. by default, csums-alg is unset.

  • csums-after-crash-only — enabling this option (and csums-alg, above) makes it possible to use the checksum based resync only for the first resync after primary crash, but not for later "network hickups".

in most cases, block that are marked as need-to-be-resynced are in fact changed, so calculating checksums, and both reading and writing the blocks on the resync target is all effective overhead.

The advantage of checksum based resync is mostly after primary crash recovery, where the recovery marked larger areas (those covered by the activity log) as need-to-be-resynced, just in case. Introduced in 8.4.5.

  • data-integrity-alg alg — okay so normally DRBD just vibes with the data integrity checks that are already baked into the TCP/IP protocol, right? But if you configure a data integrity algorithm here, it'll also use that algorithm to double-check that the data arriving over the network actually matches what the sender sent. ✨ And if it catches a data integrity error? DRBD slams the network connection shut, reconnects, and that reconnect triggers a resync. No cap.

You can set data-integrity-alg to any of the secure hash algorithms your kernel supports — go peek at the shash algorithms listed in /proc/crypto. By default this whole mechanism is turned off.

Because of the CPU overhead involved, we recommend not to use this option in production environments. Also see the notes on data integrity below. 💀 (your CPU will be sweating, bestie)
  • fencing fencing_policy — fencing is a preventive measure to avoid situations where both nodes are primary and disconnected. You know, the classic split-brain situation. 🧠💥 DRBD supports the following fencing policies:
    • dont-care — no fencing actions are taken. This is the default policy. (very "not my problem" of it 🎀)
    • resource-only — if a node becomes a disconnected primary, it tries to fence the peer. How? By calling the fence-peer handler. That handler is supposed to reach the peer over an alternative communication path and call drbdadm outdate minor there.
    • resource-and-stonith — if a node becomes a disconnected primary, it freezes all its IO operations and calls its fence-peer handler. Same deal: the handler is supposed to reach the peer over an alternative communication path and call drbdadm outdate minor there. And in case it cannot do that, it should stonith the peer. IO is resumed as soon as the situation is resolved. But if the fence-peer handler fails, I/O can be resumed manually with drbdadm resume-io.
  • ko-count number — if a secondary node fails to complete a write request in ko-count times the timeout parameter, it gets excluded from the cluster. 🚫 The primary node then sets the connection to this secondary node to Standalone. To disable this feature you should explicitly set it to 0 — because defaults may change between versions. iykyk, don't rely on the default.
  • load-balance-paths {yes | no} — by default, the TCP transport establishes only one configured path at a time. It switches to another path only in case the established one fails. But when you set load-balance-paths to yes, the TCP transport establishes all paths in parallel, and it will transmit data packets over the paths with the least data in its socket send queue. 🦋✨
Please note: enabling load-balancing introduces additional chunking headers into the network protocol. In other words, you must enable it on both sides of a connection. 👀

As of drbd-9.2.6 the RDMA transport does not obey this setting — it always uses all paths in parallel. This option became available with drbd-9.2.6.

  • max-buffers number — this one limits the memory usage per DRBD minor device on the receiving side, or for internal buffers during resync or online-verify. Unit is PAGE_SIZE, which is 4 KiB on most systems. The minimum possible setting is hard coded to 32 (= 128 KiB). These buffers are used to hold data blocks while they are written to/read from disk. To avoid possible distributed deadlocks on congestion, this setting is used as a throttle threshold rather than a hard limit — once more than max-buffers pages are in use, further allocation from this pool is throttled. You want to increase max-buffers if you cannot saturate the IO backend on the receiving side. 💪
  • max-epoch-size number — defines the maximum number of write requests DRBD may issue before issuing a write barrier. The default value is 2048, with a minimum of 1 and a maximum of 20000. Setting this parameter to a value below 10 is likely to decrease performance. 😭
  • on-congestion policy, congestion-fill threshold, congestion-extents threshold — okay this trio is a whole thing, so let's break it down. By default, DRBD blocks when the TCP send queue is full. This prevents applications from generating further write requests until more buffer space becomes available again.

When DRBD is used together with DRBD-proxy, it can be better to use the pull-ahead on-congestion policy, which can switch DRBD into ahead/behind mode before the send queue is full. DRBD then records the differences between itself and the peer in its bitmap, but it no longer replicates them to the peer. When enough buffer space becomes available again, the node resynchronizes with the peer and switches back to normal replication. 🌈

The advantage: it doesn't block application I/O even when the queues fill up. The disadvantage: peer nodes can fall behind much further, and also, while resynchronizing, peer nodes will become inconsistent. 🙈

The available congestion policies are block (the default) and pull-ahead. The congestion-fill parameter defines how much data is allowed to be "in flight" in this connection. The default value is 0, which disables this mechanism of congestion control, with a maximum of 10 GiBytes. The congestion-extents parameter defines how many bitmap extents may be active before switching into ahead/behind mode, with the same default and limits as the al-extents parameter. The congestion-extents parameter is effective only when set to a value smaller than al-extents.

Ahead/behind mode is available since DRBD 8.3.10.

  • ping-int interval — when the TCP/IP connection to a peer is idle for more than ping-int seconds, DRBD will send a keep-alive packet to make sure that a failed peer or network connection is detected reasonably soon. ⏰ The default value is 10 seconds, with a minimum of 1 and a maximum of 120 seconds. The unit is seconds.
  • ping-timeout timeout — defines the timeout for replies to keep-alive packets. If the peer does not reply within ping-timeout, DRBD will close and try to reestablish the connection. The default value is 0.5 seconds, with a minimum of 0.1 seconds and a maximum of 30 seconds. The unit is tenths of a second. ⚠️ (yes, tenths — read it twice, bestie)
  • socket-check-timeout timeout — in setups involving a DRBD-proxy and connections that experience a lot of buffer-bloat, it might be necessary to set ping-timeout to an unusually high value. By default DRBD uses the same value to wait if a newly established TCP-connection is stable. Since the DRBD-proxy is usually located in the same data center, such a long wait time may hinder DRBD's connect process. 😵

In such setups, socket-check-timeout should be set to at least the round trip time between DRBD and DRBD-proxy. I.e. in most cases to 1.

The default unit is tenths of a second, the default value is 0 (which causes DRBD to use the value of ping-timeout instead). Introduced in 8.4.5.

  • protocol name — use the specified protocol on this connection. The supported protocols are:
    • A — writes to the DRBD device complete as soon as they have reached the local disk and the TCP/IP send buffer. (speedrun energy 🏃♀️)
    • B — writes to the DRBD device complete as soon as they have reached the local disk, and all peers have acknowledged the receipt of the write requests.
    • C — writes to the DRBD device complete as soon as they have reached the local and all remote disks. This is the default protocol. 👑
  • rcvbuf-size size — configure the size of the TCP/IP receive buffer. A value of 0 (the default) causes the buffer size to adjust dynamically. This parameter usually does not need to be set, but it can be set to a value up to 128 MiB. The default unit is bytes.
  • rdma-ctrl-rcvbuf-size value — by default, the RDMA transport divides the rcvbuf-size by 64 and uses the result for the number of buffers on the control stream. This result might be too low depending on the timing characteristics of the backing storage devices and the network link. 🤔

The option rdma-ctrl-rcvbuf-size allows you to explicitly set the number of buffers for the control stream, overruling the divide by 64 heuristics. The default unit of this setting is bytes.

  • rdma-ctrl-sndbuf-size value — by default, the RDMA transport divides the sndbuf-size by 64 and uses the result for the number of buffers on the control stream. This result might be too low depending on the timing characteristics of the backing storage devices and the network link.

The option rdma-ctrl-sndbuf-size allows you to explicitly set the number of buffers for the control stream, overruling the divide by 64 heuristics. The default unit of this setting is bytes.

  • rr-conflict policy — this option helps to solve the cases when the outcome of the resync decision is incompatible with the current role assignment in the cluster. The defined policies are:
    • disconnect — no automatic resynchronization, simply disconnect. ✌️
    • retry-connect — disconnect now, and retry to connect immediatly afterwards.
    • violently — resync to the primary node is allowed, violating the assumption that data on a block device are stable for one of the nodes. Do not use this option, it is dangerous. 🚨💀 (like, actually do not, bestie)
    • call-pri-lost — call the pri-lost handler on one of the machines. The handler is expected to reboot the machine, which puts it into secondary role.
    • auto-discard — auto-discard reverses the resync direction, so that DRBD resyncs the current primary to the current secondary. Auto-discard only applies when protocol A is in use and the resync decision is based on the principle that a crashed primary should be the source of a resync. When a primary node crashes, it might have written some last updates to its disk, which were not received by a protocol A secondary. By promoting the secondary in the meantime, the user accepted that those last updates have been lost. By using auto-discard you consent that the last updates (before the crash of the primary) should be rolled back automatically. 🎀
  • shared-secret secret — okay bestie this one's about to spill the tea but we're out of room… 😭 carry on to the next part for the rest of this one!
  • Configures the shared secret used for peer authentication 🔐 — the secret is a string of up to 64 characters, no more. And heads up bestie, peer authentication also needs the cram-hmac-alg parameter to be set, so don't come crying to me if you set one and not the other 💅
  • sndbuf-size size — sets the size of the TCP/IP send buffer. Since DRBD 8.0.13 / 8.2.7, a value of 0 (which is the default ✨) makes the buffer size adjust dynamically all on its own. Values below 32 KiB are genuinely harmful to the throughput on this connection (not me being dramatic, it's just true 😭). Big buffer sizes can be super useful, especially when protocol A is used over high-latency networks. The maximum value supported is 128 MiB.
  • tcp-cork — okay so by default DRBD uses the TCP_CORK socket option to stop the kernel from sending partial messages, which means fewer and bigger packets on the network 🎀. The thing is, some network stacks actually perform worse with this optimization. On those, you use the tcp-cork parameter to turn the optimization off. Vibe check your network stack first, fr fr.
  • timeout time — defines the timeout for replies over the network: if a peer node doesn't send an expected reply within the specified timeout, it's considered dead 💀 and the TCP/IP connection gets closed. The timeout value must be lower than connect-int and lower than ping-int. The default is 6 seconds, and the value is specified in tenths of a second.
  • tls bool-value — Enable TLS. That's it, that's the whole thing ✨
  • tls-keyring key-description — the key description (name) of the keyring where the TLS key material is stored. The keyring will be shared with the handshake daemon 🤝
  • tls-privkey key-description — key description (name) of the DER encoded private key for TLS encryption.
  • tls-certificate key-description — key description (name) of the DER encoded certificate for TLS encryption.
  • transport type — with DRBD9 the network transport used by DRBD is loaded as a separate module. With this option you get to specify which transport and module to load 👑. At present only two options exist: tcp and rdma. Default is tcp.
  • use-rle — okay this one needs a sec, it's giving lore 📖. Each replicated device on a cluster node has a separate bitmap for each of its peer devices. The bitmaps are used for tracking the differences between the local and peer device: depending on the cluster state, a disk range can be marked as different from the peer in the device's bitmap, in the peer device's bitmap, or in both bitmaps. When two cluster nodes connect, they exchange each other's bitmaps, and they each compute the union of the local and peer bitmap to determine the overall differences.

Bitmaps of very large devices are also relatively large, but they usually compress very well using run-length encoding 🦋. This can save time and bandwidth for the bitmap transfers.

The use-rle parameter determines if run-length encoding should be used. It's on by default since DRBD 8.4.0 ✨

  • verify-alg hash-algorithm — online verification (drbdadm verify) computes and compares checksums of disk blocks (i.e., hash values) to detect if they differ. The verify-alg parameter determines which algorithm gets used for these checksums. It must be set to one of the secure hash algorithms supported by the kernel before online verify can be used; see the shash algorithms listed in /proc/crypto 👀

We recommend scheduling online verifications regularly during low-load periods, for example once a month. Also see the notes on data integrity below 🌸

✨ Section on Parameters ✨

  • address [address-family] address:port — defines the address family, address, and port of a connection endpoint 🫶

The address families ipv4, ipv6, ssocks (Dolphin Interconnect Solutions' "super sockets"), sdp (Infiniband Sockets Direct Protocol), and sci are supported (sci is an alias for ssocks). If no address family is specified, ipv4 is assumed. For all address families except ipv6, the address is specified in IPV4 address notation (for example, 1.2.3.4). For ipv6, the address is enclosed in brackets and uses IPv6 address notation (for example, [fd01:2345:6789:abcd::1]). The port is always specified as a decimal number from 1 to 65535.

On each host, the port numbers must be unique for each address; ports cannot be shared. No borrowing, no sharing, it's not that kind of party 💅

  • node-id value — defines the unique node identifier for a node in the cluster 👑. Node identifiers are used to identify individual nodes in the network protocol, and to assign bitmap slots to nodes in the metadata.
Node identifiers can only be reassigned in a cluster when the cluster is down 😭. It is essential that the node identifiers in the configuration and in the device metadata are changed consistently on all hosts. To change the metadata, dump the current state with drbdmeta dump-md, adjust the bitmap slot assignment, and update the metadata with drbdmeta restore-md.

The node-id parameter exists since DRBD 9. Its value ranges from 0 to 16; there is no default.

💅 Section options Parameters (Resource Options) 💅

  • auto-promote bool-value — a resource must be promoted to primary role before any of its devices can be mounted or opened for writing, periodt.

Before DRBD 9, this could only be done explicitly (drbdadm primary). Since DRBD 9, the auto-promote parameter allows you to automatically promote a resource to primary role when one of its devices is mounted or opened for writing ✨. As soon as all devices are unmounted or closed with no more remaining users, the role of the resource changes back to secondary.

Automatic promotion only succeeds if the cluster state allows it (that is, if an explicit drbdadm primary command would succeed). Otherwise, mounting or opening the device fails exactly like it did before DRBD 9: the mount(2) system call fails with errno set to EROFS (Read-only file system); the open(2) system call fails with errno set to EMEDIUMTYPE (wrong medium type) 💀

Irrespective of the auto-promote parameter, if a device is promoted explicitly (drbdadm primary), it also needs to be demoted explicitly (drbdadm secondary). No take-backsies.

The auto-promote parameter is available since DRBD 9.0.0, and defaults to yes.

  • auto-promote-timeout 1/10-of-seconds — when a user process promotes a drbd resource by opening one of its devices, DRBD waits up to auto-promote-timeout for the device to become promotable if it isn't in the first place ⏳

auto-promote-timeout is specified in units of 0.1 seconds. Its default value is 20 (2 seconds), its minimum value is 0, and its maximum value is 600 (=one minute).

  • cpu-mask cpu-mask — sets the cpu affinity mask for DRBD kernel threads 🧵. The cpu mask is specified as a hexadecimal number. The default value is 0, which lets the scheduler decide which kernel threads run on which CPUs. CPU numbers in cpu-mask which don't exist in the system are just ignored, nbd.
  • max-io-depth value — this limits the number of outstanding requests on a DRBD device. Any process that tries to issue more I/O requests will sleep in "D state" (uninterruptible by signals 😴) until some previously issued requests finish.

max-io-depth has a default value of 8000, its minimum value is 4, and its maximum value is 2^32.

  • on-no-data-accessible policy — determines how to deal with I/O requests when the requested data is not available locally or remotely (for example, when all disks have failed) 😭. When quorum is enabled, on-no-data-accessible should be set to the same value as on-no-quorum. The defined policies are:
    • io-error — system calls fail with errno set to EIO.
    • suspend-io — the resource suspends I/O. I/O can be resumed by (re)attaching the lower-level device, by connecting to a peer which has access to the data, or by forcing DRBD to resume I/O with drbdadm resume-io res. When no data is available, forcing I/O to resume will result in the same behavior as the io-error policy.

This setting is available since DRBD 8.3.9; the default policy is io-error.

  • on-no-quorum {io-error | suspend-io} — by default DRBD freezes IO on a device that lost quorum 🧊. By setting on-no-quorum to io-error it completes all IO operations with an error if quorum is lost.

Usually, the on-no-data-accessible should be set to the same value as on-no-quorum, as it has precedence 👀

The on-no-quorum options is available starting with the DRBD kernel driver version 9.0.8.

  • on-suspended-primary-outdated {disconnect | force-secondary} — this setting is only relevant when on-no-quorum is set to suspend-io. It's relevant in the following scenario 📖: a primary node loses quorum hence has all IO requests frozen. This primary node then connects to another, quorate partition. It detects that a node in this quorate partition was promoted to primary, and started a newer data-generation there. As a result, the first primary learns that it has to consider itself outdated 😭
    • When it's set to force-secondary then it will demote to secondary immediately, and fail all pending (and new) IO requests with IO errors. It will refuse to allow any process to open the DRBD devices until all openers closed the device. This state is visible in status and events2 under the name force-io-failures.
    • The disconnect setting simply causes that node to reject connect attempts and stay isolated 🫥

The on-suspended-primary-outdated option is available starting with the DRBD kernel driver version 9.1.7. It has a default value of disconnect.

  • peer-ack-delay expiry-time — if after the last finished write request no new write request gets issued for expiry-time, then a peer-ack packet is sent 📨. If a new write request is issued before the timer expires, the timer gets reset to expiry-time. (Note: peer-ack packets may be sent due to other reasons as well, e.g. membership changes or the peer-ack-window option.)

This parameter may influence resync behavior on remote nodes. Peer nodes need to wait until they receive a peer-ack for releasing a lock on an AL-extent. Resync operations between peers may need to wait for these locks ⏳

The default value for peer-ack-delay is 100 milliseconds, the default unit is milliseconds. This option is available since 9.0.0.

  • peer-ack-window value

okay so picture this: on every single node, and for every single device, DRBD keeps a whole lil' bitmap of the differences between the local and the remote data — one bitmap for each peer device. So in a three-node setup (nodes A, B, C), each with a single device, every node is out here maintaining one bitmap for each of its peers. 💅

When nodes receive write requests, they know exactly how to update the bitmaps for the writing node, but they have zero clue how to update the bitmaps between themselves. In this example, when a write request propagates from node A to B and C, nodes B and C know that they have the same data as node A, but they do not know whether or not they both have the same data. 😭

As a remedy, the writing node occasionally sends peer-ack packets to its peers, which tell them which state they are in relative to each other.

The peer-ack-window parameter specifies how much data a primary node may send before sending a peer-ack packet. A low value causes increased network traffic; a high value causes less network traffic but higher memory consumption on secondary nodes and higher resync times between the secondary nodes after primary node failures. (Note: peer-ack packets may be sent due to other reasons as well, e.g. membership changes or expiry of the peer-ack-delay timer.)

The default value for peer-ack-window is 2 MiB, the default unit is sectors. This option is available since 9.0.0. 🎀

quorum value

When activated, a cluster partition requires quorum in order to modify the replicated data set. That means a node in the cluster partition can only be promoted to primary if the cluster partition has quorum. Every node with a disk directly connected to the node that should be promoted counts. If a primary node should execute a write request, but the cluster partition has lost quorum, it will freeze IO or reject the write request with an error (depending on the on-no-quorum setting). Upon losing quorum, a primary always invokes the quorum-lost handler. The handler is intended for notification purposes — its return code is ignored. 🙈

The option's value might be set to off, majority, all, or a numeric value. If you set it to a numeric value, make sure that the value is greater than half of your number of nodes. Quorum is a mechanism to avoid data divergence; it might be used instead of fencing when there are more than two replicas. It defaults to off.

If all missing nodes are marked as outdated, a partition always has quorum, no matter how small it is. I.e. if you disconnect all secondary nodes gracefully, a single primary continues to operate. In the moment a single secondary is lost, it has to be assumed that it forms a partition with all the missing outdated nodes. In case my partition might be smaller than the other, quorum is lost in this moment. 💀

In case you want to allow permanently diskless nodes to gain quorum, it is recommended to not use majority or all. It is recommended to specify an absolute number, since DBRD's heuristic to determine the complete number of diskfull nodes in the cluster is unreliable.

The quorum implementation is available starting with the DRBD kernel driver version 9.0.7.

quorum-minimum-redundancy value

This option sets the minimal required number of nodes with an UpToDate disk to allow the partition to gain quorum. This is a different requirement than the plain quorum option expresses. 👀

The option's value might be set to off, majority, all, or a numeric value. If you set it to a numeric value, make sure that the value is greater than half of your number of nodes.

In case you want to allow permanently diskless nodes to gain quorum, it is recommended to not use majority or all. It is recommended to specify an absolute number, since DBRD's heuristic to determine the complete number of diskfull nodes in the cluster is unreliable.

This option is available starting with the DRBD kernel driver version 9.0.10.

twopc-retry-timeout 1/10-of-seconds

Due to conflicting two-phase-commit, sometimes DRBD needs to retry them. But if two nodes retry their intended two-phase-commits after the same time, they would end up in an endless retry loop. To avoid that, DRBD selects a random wait time within an upper bound, an exponential backoff, and a function of the retry number. The twopc-retry-timeout is a base multiplier for that function. 🌀

twopc-retry-timeout has a default value of 10 (0.1 seconds), its minimum value is 1 (0.1 seconds), and its maximum value is 50 (5 seconds).

twopc-timeout 1/10-of-seconds

In some situations, a DRBD cluster requires a cluster-wide coordinated state transition. A perfect example of this is the 'promote-to-primary' action. Even if two not directly connected nodes in a cluster try this action concurrently, it may only succeed for one of the two. ✨

For these cluster-wide coordinated state transitions, DRBD implements a two-phase commit protocol. If a connection breaks in phase one (prepare packet sent), the coordinator of the two-phase commit might never get the expected reply packet. 😬

A cluster in this state can not start any new cluster-wide coordinated state transition, as the already prepared one blocks all such attempts. After twopc-timeout, all nodes abort the prepared transaction and unlock the cluster again.

twopc-timeout has a default value of 300 (30 seconds), its minimum value is 50 (5 seconds), and its maximum value is 600 (one minute).

✨ Section startup Parameters ✨

The parameters in this section define the behavior of DRBD at system startup time, in the DRBD init script. They have no effect once the system is up and running.

degr-wfc-timeout timeout

Define how long to wait until all peers are connected in case the cluster consisted of a single node only when the system went down. This parameter is usually set to a value smaller than wfc-timeout. The assumption here is that peers which were unreachable before a reboot are less likely to be reachable after the reboot, so waiting is less likely to help. 🫶

The timeout is specified in seconds. The default value is 0, which stands for an infinite timeout. Also see the wfc-timeout parameter.

outdated-wfc-timeout timeout

Define how long to wait until all peers are connected if all peers were outdated when the system went down. This parameter is usually set to a value smaller than wfc-timeout. The assumption here is that an outdated peer cannot have become primary in the meantime, so we don't need to wait for it as long as for a node which was alive before.

The timeout is specified in seconds. The default value is 0, which stands for an infinite timeout. Also see the wfc-timeout parameter.

stacked-timeouts

On stacked devices, the wfc-timeout and degr-wfc-timeout parameters in the configuration are usually ignored, and both timeouts are set to twice the connect-int timeout. The stacked-timeouts parameter tells DRBD to use the wfc-timeout and degr-wfc-timeout parameters as defined in the configuration, even on stacked devices. Only use this parameter if the peer of the stacked resource is usually not available, or will not become primary.

Incorrect use of this parameter can lead to unexpected split-brain scenarios. 💀

wait-after-sb

This parameter causes DRBD to continue waiting in the init script even when a split-brain situation has been detected, and the nodes therefore refuse to connect to each other.

wfc-timeout timeout

Define how long the init script waits until all peers are connected. This can be useful in combination with a cluster manager which cannot manage DRBD resources: when the cluster manager starts, the DRBD resources will already be up and running. With a more capable cluster manager such as Pacemaker, it makes more sense to let the cluster manager control DRBD resources. The timeout is specified in seconds. The default value is 0, which stands for an infinite timeout. Also see the degr-wfc-timeout parameter. 🌸

🎀 Section volume Parameters 🎀

device /dev/drbdminor-number

Define the device name and minor number of a replicated block device. This is the device that applications are supposed to access; in most cases, the device is not used directly, but as a file system. This parameter is required and the standard device naming convention is assumed. 💅

In addition to this device, udev will create /dev/drbd/by-res/resource/volume and /dev/drbd/by-disk/lower-level-device symlinks to the device.

disk {[disk] | none}

Define the lower-level block device that DRBD will use for storing the actual data.

While the replicated drbd device is configured, the lower-level device must not be used directly. Even read-only access with tools like dumpe2fs(8) and similar is not allowed. 🙅

The keyword none specifies that no lower-level block device is configured; this also overrides inheritance of the lower-level device.

meta-disk internal,

meta-disk device,

meta-disk device [index]

Define where the metadata of a replicated block device resides: it can be internal, meaning that the lower-level device contains both the data and the metadata, or on a separate device.

When the index form of this parameter is used, multiple replicated devices can share the same metadata device, each using a separate index. Each index occupies 128 MiB of data, which corresponds to a replicated device size of at most 4 TiB with two cluster nodes. We recommend not to share metadata devices anymore, and to instead use the lvm volume manager for creating metadata devices as needed. ✨

When the index form of this parameter is not used, the size of the lower-level device determines the size of the metadata. The size needed is 36 KiB + (size of lower-level device) / 32K * (number of nodes - 1). If the metadata device is bigger than that, the extra space is not used.

This parameter is required if a disk other than none is specified, and ignored if disk is set to none. A meta-disk parameter without a disk parameter is not allowed.

tiebreaker {yes | no}

Declare whether this volume participates as a tiebreaker in quorum decisions. The default is yes. Internally, drbdadm maps this to setting peer-tiebreaker on this device's peers. 🧚

NOTES ON DATA INTEGRITY 🛡️

okay bestie, this is the part where DRBD is literally checking if your data is lying to you. DRBD supports two different mechanisms for data integrity checking, and they are not the same vibe at all.

First up: the data-integrity-alg network parameter. That one lets you add a checksum to the data sent over the network. Second: the online verification mechanism (drbdadm verify and the verify-alg parameter), which lets you check for differences in the on-disk data. Two different lanes, same goal: catch corruption before it ruins your whole day.

Both mechanisms can produce false positives if the data is modified during I/O (i.e., while it is being sent over the network or written to disk). This does not always indicate a problem: for example, some file systems and applications do modify data under I/O for certain operations. Swap space can also undergo changes while under I/O. So don't panic immediately, okay? 😭

Network data integrity checking tries to identify data modification during I/O by verifying the checksums on the sender side after sending the data. If it detects a mismatch, it logs an error. The receiver also logs an error when it detects a mismatch. So: an error logged only on the receiver side indicates an error on the network, and an error logged on both sides indicates data modification under I/O. It's giving detective work 🔍.

The most recent example of systematic data corruption was identified as a bug in the TCP offloading engine and driver of a certain type of GBit NIC in 2007: the data corruption happened on the DMA transfer from core memory to the card. Because the TCP checksum were calculated on the card, the TCP/IP protocol checksums did not reveal this problem. Iconic in the worst way possible 💀.

VERSION 📌

This document was revised for version 9.0.0 of the DRBD distribution.

AUTHOR ✍️

Written by Philipp Reisner <philipp.reisner@linbit.com> and Lars Ellenberg <lars.ellenberg@linbit.com>.

REPORTING BUGS 🐛

Report bugs to <drbd-user@lists.linbit.com>.

Copyright 2001-2018 LINBIT Information Technologies, Philipp Reisner, Lars Ellenberg. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

SEE ALSO 👀

drbd(8), drbdsetup(8), drbdadm(8), DRBD User's Guide[1], DRBD Web Site[3]

NOTES 📝

  1. DRBD User's Guide http://www.drbd.org/users-guide/
  2. Online Usage Counter http://usage.drbd.org
  3. DRBD Web Site http://www.drbd.org/

And that's the whole page, bestie! The footer said DRBD 9.0.x, 17 January 2018, and DRBD.CONF(5) — so that's the source/version and date, periodt. Go forth and keep your data integrity cute 💖✨🦋

📦 where this page comes from