DRBDSETUP(8)

📦
version 9.34.0 · 7 versions
, in alpine 3.24, fedora 43, fedora 44, nixpkgs unstable

drbdsetup 💖 configure the DRBD kernel module

okay so quick vibe check first: this is the manual page for drbdsetup, and it lives in the System Administration section of the manual (section 8, bestie, that's the admin one ✨). it's giving "i talk directly to the kernel module" energy and honestly? iconic. 🫶

✨ NAME ✨

drbdsetup — configure the DRBD kernel module 💅

🌸 SYNOPSIS 🌸

drbdsetup command {argument...} [option...]

🦋 DESCRIPTION 🦋

okay so the drbdsetup utility is the one you use to configure the DRBD kernel module and to peek at its current configuration. that's its whole job. it's doing it slayfully. 💖

now here's the tea though: most users don't actually run drbdsetup by hand. they run drbdadm instead, because drbdadm gives you a much more high-level interface to DRBD than drbdsetup does. so drbdadm is the main character and drbdsetup is the cool older sibling behind the scenes. (pro tip: go look at drbdadm's --dry-run option to see exactly how drbdadm uses drbdsetup — it literally shows you the receipts. 🔍)

and one thing about numbers, bestie, because this trips people up: some option arguments have a default scale that kicks in when you just type a plain number. like, the default scale might be Kilo, aka 1024 times the numeric value. but you can override that default scale any time by slapping a suffix on there — for example, M for Mega. the common suffixes that are supported are:

  • K = 2^10 = 1024
  • M = 1024 K
  • G = 1024 M

so K is 1024, M is 1024 K, G is 1024 M. simple math, gorgeous math, we love to see it. 🧮✨

🛠️ COMMANDS 🛠️

🎀 attach and disk-options

drbdsetup attach minor lower_dev meta_data_dev meta_data_index
drbdsetup disk-options minor

so the attach command attaches a lower-level device to an existing replicated device. and the disk-options command changes the disk options of an attached lower-level device. either way, the replicated device must have already been created with drbdsetup new-minor — no skipping steps, bestie, it doesn't work like that. 🙈

both of these commands refer to the replicated device by its minor number. let's break down the args:

  • lower_dev is the name of the lower-level device. 🧱
  • meta_data_dev is the name of the device containing the metadata, and btw it's allowed to be the exact same device as lower_dev. no drama.
  • meta_data_index is one of three things: either a numeric metadata index, or the keyword internal for internal metadata, or the keyword flexible for variable-size external metadata. pick your fighter. 💅

available options, let's go:

  • --al-extents extents — okay so DRBD automatically maintains a "hot" or "active" disk area — one that's likely to get written to again soon — based on recent write activity. the active disk area can be written to immediately, while inactive disk areas have to be "activated" first, and that activation requires a meta-data write. we also call this active disk area the "activity log".

the activity log saves you meta-data writes, yes, but the whole log has to be resynced when a failed node recovers. so the size of the activity log is a major factor in how long a resync takes, and how fast a replicated disk becomes consistent after a crash. it's giving "big log = long resync".

the activity log is made up of a bunch of 4-Megabyte segments, and the al-extents parameter decides how many of those segments can be active at the same time. the default value for al-extents is 1237, with a minimum of 7 and a maximum of 65536.

heads up though: the effective maximum might be smaller, depending on how you created the device meta data. see also drbdmeta(8). the effective maximum is 919 * (available on-disk activity-log ring-buffer area/4kB -1), and the default 32kB ring-buffer effects a maximum of 6433 (which covers more than 25 GiB of data). we recommend keeping this well within the amount your backend storage and replication link are able to resync inside of about 5 minutes. 💀
  • --al-updates {yes | no} — with this parameter, the activity log can be turned off entirely (see the al-extents parameter above). this will speed up writes, because fewer meta-data writes will be necessary — but the entire device needs to be resynchronized upon recovery of a failed primary node. 😭 so it's a tradeoff, not free candy. the default value for al-updates is yes.
  • --disk-barrier, --disk-flushes, --disk-drain — DRBD has three methods of handling the ordering of dependent write requests. here's the rundown:
    • disk-barrier — use disk barriers to make sure that requests are written to disk in the right order. barriers ensure that all requests submitted before a barrier make it to the disk before any requests submitted after the barrier. this is implemented using 'tagged command queuing' on SCSI devices and 'native command queuing' on SATA devices. only some devices and device stacks support this method, and the device mapper (LVM) only supports barriers in some configurations.
⚠️ note that on systems which do not support disk barriers, enabling this option can lead to data loss or corruption. until DRBD 8.4.1, disk-barrier was turned on if the I/O stack below DRBD did support barriers. kernels since linux-2.6.36 (or 2.6.32 RHEL6) no longer allow you to detect if barriers are supported. since drbd-8.4.2, this option is off by default and needs to be enabled explicitly. 💅
  • disk-flushes — use disk flushes between dependent write requests, also referred to as 'force unit access' by drive vendors. this forces all data to disk. this option is enabled by default. ✨
  • disk-drain — wait for the request queue to "drain" (that is, wait for the requests to finish) before submitting a dependent write request. this method requires that requests are stable on disk when they finish. before DRBD 8.0.9, this was the only method implemented. this option is enabled by default. do not disable in production environments. 🚨

out of these three methods, drbd will use the first one that is enabled and supported by the backing storage device. and if all three of these options are turned off? DRBD will submit write requests without bothering about dependencies at all. depending on the I/O stack, write requests can be reordered, and they can be submitted in a different order on different cluster nodes. this can result in data loss or corruption. so turning off all three methods of controlling write ordering is strongly discouraged, bestie, i am begging you. 😭

a general guideline for configuring write ordering: use disk barriers or disk flushes when you're using ordinary disks (or an ordinary disk array) with a volatile write cache. on storage without cache, or with a battery backed write cache, disk draining can be a reasonable choice. ✨

  • --disk-timeout — if the lower-level device on which a DRBD device stores its data doesn't finish an I/O request within the defined disk-timeout, DRBD treats this as a failure. the lower-level device gets detached, and the device's disk state advances to Diskless. if DRBD is connected to one or more peers, the failed request is passed on to one of them.
🚨 this option is dangerous and may lead to kernel panic! 🚨

okay so let's talk about what "aborting" requests, or force-detaching the disk, is actually for: it's intended for completely blocked/hung local backing devices which do no longer complete requests at all, not even error completions. in that situation, usually a hard-reset and failover is the only way out. 🫠

by "aborting" — basically faking a local error-completion — we allow for a more graceful switchover by cleanly migrating services. still, the affected node has to be rebooted "soon". by completing these requests, we allow the upper layers to re-use the associated data pages.

and here's where it gets scary: if later the local backing device "recovers", and now DMAs some data from disk into the original request pages, in the best case it will just put random data into unused pages — but typically it will corrupt meanwhile completely unrelated data, causing all sorts of damage. 💀

which means delayed successful completion, especially for READ requests, is a reason to panic(). we assume that a delayed error completion is OK, though we still will complain noisily about it. 😤

the default value of disk-timeout is 0, which stands for an infinite timeout. timeouts are specified in units of 0.1 seconds. this option is available since DRBD 8.3.12.

  • --md-flushes — enable disk flushes and disk barriers on the meta-data device. this option is enabled by default. see the disk-flushes parameter above. 🎀
  • --on-io-error handler — configure how DRBD reacts to I/O errors on a lower-level device. the following policies are defined:
    • pass_on — change the disk status to Inconsistent, mark the failed block as inconsistent in the bitmap, and retry the I/O operation on a remote cluster node. 🔁
    • call-local-io-error — call the local-io-error handler (see the handlers section). 📞
    • detach — detach the lower-level device and continue in diskless mode. 🫥
  • --read-balancing policy — distribute read requests among cluster nodes as defined by policy. the supported policies are prefer-local (the default), prefer-remote, round-robin, least-pending, when-congested-remote, 32K-striping, 64K-striping, 128K-striping, 256K-striping, 512K-striping and 1M-striping. ✨

this option is available since DRBD 8.4.1.

note: the when-congested-remote option has no effect on Linux kernel 5.18 or above. it is deprecated starting from DRBD 9.1.12. 🙈
  • resync-after minor — define that a device should only resynchronize after the specified other device. by default, no order between devices is defined, and all devices will resynchronize in parallel. depending on the configuration of the lower-level devices, and the available network and disk bandwidth, this can slow down the overall resync process. this option can be used to form a chain or tree of dependencies among devices. 🌳
  • --size size — specify the size of the lower-level device explicitly instead of determining it automatically. the device size must be determined once and is remembered for the lifetime of the device. in order to determine it automatically, all the lower-level devices on all nodes must be attached, and all nodes must be connected. if the size is specified explicitly, this is not necessary. 🎉 the size value is assumed to be in units of sectors (512 bytes) by default.
  • --discard-zeroes-if-aligned {yes | no} — there are several aspects to discard/trim/unmap support on linux block devices. even if discard is supported in general, it may fail silently, or may partially ignore discard requests. devices also announce whether reading from unmapped blocks returns defined data (usually zeroes), or undefined data (possibly old data, possibly garbage). 💀

okay so picture this 💭 — your two nodes are backed by devices with differing discard characteristics, and suddenly discards become a whole drama. Old data or garbage gets left over on one backend, zeroes show up because of unmapped areas on the other backend, and boom 💥: data divergence. Online verify would then potentially report tons of spurious differences. For most use cases (think fstrim on a file system) that's probably harmless, but DRBD simply cannot have that. Not on our watch. 🙅‍♀️

To play it safe, we have to disable discard support if our local backend (on a Primary) does not support "discard_zeroes_data=true". And we also have to translate discards to an explicit zero-out on the receiving side, unless the receiving side (Secondary) supports "discard_zeroes_data=true" — because otherwise we'd be allocating areas that were supposed to be unmapped. iykyk.

Now, there are some devices (notably the LVM/DM thin provisioning ones 👀) that are capable of discard but announce discard_zeroes_data=false. In the case of DM-thin, discards aligned to the chunk size will be unmapped, and reading from unmapped sectors returns zeroes. BUT — unaligned partial head or tail areas of discard requests get silently ignored. Rude, honestly. 🙈

So if we add a helper to explicitly zero-out those unaligned partial areas, while passing on the discard of the aligned full chunks, we effectively achieve discard_zeroes_data=true on such devices. Iconic workaround, no cap. ✨

  • Setting discard-zeroes-if-aligned to yes will allow DRBD to use discards, and to announce discard_zeroes_data=true, even on backends that announce discard_zeroes_data=false.
  • Setting discard-zeroes-if-aligned to no will cause DRBD to always fall-back to zero-out on the receiving side, and to not even announce discard capabilities on the Primary, if the respective backend announces discard_zeroes_data=false.

We used to ignore the discard_zeroes_data setting completely. But to not break established and expected behaviour — and to not suddenly cause fstrim on thin-provisioned LVs to run out-of-space instead of freeing up space 😭 — the default value is yes.

This option is available since 8.4.7.

  • --disable-write-same {yes | no} — some disks announce WRITE_SAME support to the kernel but then fail with an I/O error the moment they actually receive such a request 💀. This mostly happens with virtualized disks — notably, this behavior has been observed with VMware's virtual disks. When disable-write-same is set to yes, WRITE_SAME detection is manually overriden and support is disabled. The default value of disable-write-same is no. This option is available since 8.4.7.
  • --rs-discard-granularity byte — when rs-discard-granularity is set to a non zero, positive value, DRBD tries to do a resync operation in requests of this size. If such a block contains only zero bytes on the sync source node, the sync target node will issue a discard/trim/unmap command for that area. 🧹

The value is constrained by the discard granularity of the backing block device. If rs-discard-granularity is not a multiplier of the discard granularity of the backing block device, DRBD rounds it up. The feature only gets active if the backing block device reads back zeroes after a discard command.

Heads up bestie: the usage of rs-discard-granularity may cause c-max-rate to be exceeded. In particular, the resync rate may reach 10x the value of rs-discard-granularity per second. 😳

The default value of rs-discard-granularity is 0. This option is available since 8.4.7.

  • --bitmap {yes | no} — by default, the drbd kernel driver allocates the change-tracking bitmap at the moment the virtual drbd block device gets attached to a local backing block device. Setting the bitmap option to no suppresses bitmap allocation — the benefit is the reduced memory footprint 🌸. The consequence? It is not possible to add a peer to the resource until the bitmap is allocated by changing this value to yes. The default value of bitmap is yes. This option is available since drbd-utils 9.33 and the drbd kernel driver 9.3.0.

💖 drbdsetup peer-device-options resource peer_node_id volume 💖

These are options that affect the peer's device. 🫶

  • --c-delay-target delay_target, --c-fill-target fill_target, --c-max-rate max_rate, --c-plan-ahead plan_time — dynamically control the resync speed. The following modes are available:
    • Dynamic control with fill target (default). Enabled when c-plan-ahead is non-zero and c-fill-target is non-zero. The goal is to fill the buffers along the data path with a defined amount of data. This mode is recommended when DRBD-proxy is used. Configured with c-plan-ahead, c-fill-target and c-max-rate.
    • Dynamic control with delay target. Enabled when c-plan-ahead is non-zero (default) and c-fill-target is zero. The goal is to have a defined delay along the path. Configured with c-plan-ahead, c-delay-target and c-max-rate.
    • Fixed resync rate. Enabled when c-plan-ahead is zero. DRBD will try to perform resync I/O at a fixed rate. Configured with resync-rate.

The c-plan-ahead parameter defines how fast DRBD adapts to changes in the resync speed. It should be set to five times the network round-trip time or more. The default value of c-plan-ahead is 20, in units of 0.1 seconds.

The c-fill-target parameter defines the how much resync data DRBD should aim to have in-flight at all times. Common values for "normal" data paths range from 4K to 100K. The default value of c-fill-target is 100, in units of sectors.

The c-delay-target parameter defines the delay in the resync path that DRBD should aim for. This should be set to five times the network round-trip time or more. The default value of c-delay-target is 10, in units of 0.1 seconds.

The c-max-rate parameter limits the maximum bandwidth used by dynamically controlled resyncs. Setting this to zero removes the limitation (since DRBD 9.0.28). It should be set to either the bandwidth available between the DRBD hosts and the machines hosting DRBD-proxy, or to the available disk bandwidth. The default value of c-max-rate is 102400, in units of KiB/s.

Dynamic resync speed control is available since DRBD 8.3.9.

  • --c-min-rate min_rate — a node which is primary and sync-source has to schedule application I/O requests and resync I/O requests. The c-min-rate parameter limits how much bandwidth is available for resync I/O; the remaining bandwidth is used for application I/O. A c-min-rate value of 0 means that there is no limit on the resync I/O bandwidth — and that can slow down application I/O significantly, so be careful 😬. Use a value of 1 (1 KiB/s) for the lowest possible resync rate. The default value of c-min-rate is 250, in units of KiB/s.
  • --resync-rate rate — define how much bandwidth DRBD may use for resynchronizing. DRBD allows "normal" application I/O even during a resync, but if the resync takes up too much bandwidth, application I/O can become very slow. This parameter allows to avoid that. Please note this option only works when the dynamic resync controller is disabled. 🎀
  • --resync-without-replication {yes | no} — control the replication of application writes during resyncs. The default value of this option is yes, meaning that application writes are not replicated for most of the resync. This generally leads to lower bandwidth usage during resyncs. The benefit is significant when there are repeated application writes to the same blocks. DRBD will continue running the resync without replication while it makes progress. When it no longer makes sufficient progress, the replication of application writes will be enabled. This allows the disk on the SyncTarget node to become UpToDate.

Under some circumstances, resyncs may finish quicker when this behavior is disabled. In particular, when there are application writes to distinct blocks and the resync rate is limited by c-max-rate rather than by the available bandwidth. This leaves the disk of the SyncTarget node in the Inconsistent state for longer, reducing availability. Set this option to no to maximize availability under these circumstances.

The value of this option affects the replication of writes to the local node from the peer for which the option is set. This option is available since DRBD 9.3.0.

  • --peer-tiebreaker {yes | no} — declare whether the peer volume participates as a tiebreaker in quorum decisions. The default is yes. Setting this to no excludes the peer from quorum tiebreaking for this volume.
Real talk: rather than setting this option directly in peer-device-options on each connection, consider using the tiebreaker volume-level shorthand in drbd.conf(5), which automatically propagates to all connections pointing at that host. 💅

🌈 drbdsetup check-resize minor 🌈

Remember the current size of the lower-level device of the specified replicated device. Used by drbdadm. The size information is stored in file /var/lib/drbd/drbd-minor-minor.lkbd. 📝

✨ drbdsetup new-peer resource peer_node_id ✨

✨ drbdsetup net-options resource peer_node_id ✨

The new-peer command creates a connection within a resource. The resource must have been created with drbdsetup new-resource. The net-options command changes the network options of an existing connection. Before a connection can be activated with the connect command, at least one path need to added with the new-path command. 🦋

Available options:

  • --after-sb-0pri policy — define how to react if a split-brain scenario is detected and none of the two nodes is in primary role. (We detect split-brain scenarios when two nodes connect; split-brain decisions are always between two nodes.) The defined policies are:
    • disconnect — no automatic resynchronization; simply disconnect.
  • discard-younger-primary, discard-older-primary — resync from whichever node went primary first (discard-younger-primary) or last (discard-older-primary). 💫 And if both nodes became primary independently? Then the discard-least-changes policy takes over. teamwork, kinda.
    • discard-zero-changes — if literally only one of the nodes wrote data since the split brain situation was detected, resync from that node to the other one. 🫶 But if both nodes wrote data? just disconnect. bye.
    • discard-least-changes — resync from the node with more modified blocks. the one who did the most gets to be the source, slay.
    • discard-node-nodename — always resync to the named node. no questions asked, that node is the main character. 👑
  • --after-sb-1pri policy — sets how to react when a split-brain scenario shows up with one node in primary role and one in secondary role. (Quick reminder bestie: we detect split-brain scenarios when two nodes connect, so split-brain decisions are always among two nodes. always. two.) The policies you can pick are:
    • disconnect — no automatic resynchronization, just disconnect. 🚪
    • consensus — throw away the data on the secondary node, but only if the after-sb-0pri algorithm would also discard the data on the secondary node. Otherwise? disconnect.
    • violently-as0p — always take the decision of the after-sb-0pri algorithm, even if it causes an erratic change of the primary's view of the data. This is only useful if a single-node file system (i.e., not OCFS2 or GFS) with the allow-two-primaries flag is used.
⚠️ This option can cause the primary node to crash, and should not be used. like, fr fr, do not. 💀
  • discard-secondary — discard the data on the secondary node. that's it, that's the policy.
  • call-pri-lost-after-sb — always take the decision of the after-sb-0pri algorithm. And if that decision is to discard the data on the primary node, call the pri-lost-after-sb handler on the primary node. 😭
  • --after-sb-2pri policy — sets how to react when a split-brain scenario is detected and both nodes are in primary role. (Same reminder as before: we detect split-brain scenarios when two nodes connect, so split-brain decisions are always among two nodes.) The policies:
    • disconnect — no automatic resynchronization, simply disconnect.
    • violently-as0p — see the violently-as0p policy for after-sb-1pri. 👀
    • call-pri-lost-after-sb — call the pri-lost-after-sb helper program on one of the machines, unless that machine can demote to secondary. That helper program is expected to reboot the machine, which brings the node into a secondary role. Which machine runs the helper program is determined by the after-sb-0pri strategy. 🔄
  • --allow-two-primaries — so the most common way to configure DRBD devices is to allow only one node to be primary (and therefore writable) at a time.

But in some scenarios it's preferable to allow two nodes to be primary at once; and then a mechanism outside of DRBD has to make sure that writes to the shared, replicated device happen in a coordinated way. This can be done with a shared-storage cluster file system like OCFS2 and GFS, or with virtual machine images and a virtual machine manager that can migrate virtual machines between physical machines.

The allow-two-primaries parameter tells DRBD to allow two nodes to be primary at the same time.

⚠️ Never enable this option when using a non-distributed file system; otherwise, data corruption and node crashes will result! 🚨 not a vibe. not even a little.
  • --always-asbp — normally the automatic after-split-brain policies are only used if current states of the UUIDs do not indicate the presence of a third node.

With this option you request that the automatic after-split-brain policies are used as long as the data sets of the nodes are somehow related. This might cause a full sync, if the UUIDs indicate the presence of a third node. (Or double faults led to strange UUID sets.) 🌀

  • --connect-int time — the second a connection between two nodes is configured with drbdsetup connect, DRBD immediately tries to establish the connection. If that fails, DRBD waits for connect-int seconds and then repeats. The default value of connect-int is 10 seconds. ⏳
  • --cram-hmac-alg hash-algorithm — configure the hash-based message authentication code (HMAC) or secure hash algorithm to use for peer authentication. The kernel supports a number of different algorithms, some of which may be loadable as kernel modules. See the shash algorithms listed in /proc/crypto. By default, cram-hmac-alg is unset. Peer authentication also requires a shared-secret to be configured. 🔐 (both of them, bestie, it's giving buddy system)
  • --csums-alg hash-algorithm — okay so normally, when two nodes resynchronize, the sync target requests a piece of out-of-sync data from the sync source, and the sync source sends the data. And with many usage patterns, a significant number of those blocks will actually be identical. 🙃

When a csums-alg algorithm is specified, when requesting a piece of out-of-sync data, the sync target also sends along a hash of the data it currently has. The sync source compares this hash with its own version of the data. It sends the sync target the new data if the hashes differ, and tells it that the data are the same otherwise. This reduces the network bandwidth required, at the cost of higher cpu utilization and possibly increased I/O on the sync target.

The csums-alg can be set to one of the secure hash algorithms supported by the kernel; see the shash algorithms listed in /proc/crypto. By default, csums-alg is unset.

  • --csums-after-crash-only — enabling this option (and csums-alg, above) makes it possible to use the checksum based resync only for the first resync after primary crash, but not for later "network hickups". 🌸

In most cases, block that are marked as need-to-be-resynced are in fact changed, so calculating checksums, and both reading and writing the blocks on the resync target is all effective overhead.

The advantage of checksum based resync is mostly after primary crash recovery, where the recovery marked larger areas (those covered by the activity log) as need-to-be-resynced, just in case. Introduced in 8.4.5.

  • --data-integrity-alg alg — DRBD normally relies on the data integrity checks built into the TCP/IP protocol, but if a data integrity algorithm is configured, it will additionally use this algorithm to make sure that the data received over the network match what the sender has sent. If a data integrity error is detected, DRBD will close the network connection and reconnect, which will trigger a resync. 🔁

The data-integrity-alg can be set to one of the secure hash algorithms supported by the kernel; see the shash algorithms listed in /proc/crypto. By default, this mechanism is turned off.

Because of the CPU overhead involved, we recommend not to use this option in production environments. Also see the notes on data integrity below. 👀

  • --fencing fencing_policy — fencing is a preventive measure to avoid situations where both nodes are primary and disconnected. This is also known as a split-brain situation. DRBD supports the following fencing policies:
    • dont-care — no fencing actions are taken. This is the default policy. 😌
    • resource-only — if a node becomes a disconnected primary, it tries to fence the peer. This is done by calling the fence-peer handler. The handler is supposed to reach the peer over an alternative communication path and call 'drbdadm outdate minor' there. 📞
    • resource-and-stonith — if a node becomes a disconnected primary, it freezes all its IO operations and calls its fence-peer handler. The fence-peer handler is supposed to reach the peer over an alternative communication path and call 'drbdadm outdate minor' there. In case it cannot do that, it should stonith the peer. IO is resumed as soon as the situation is resolved. In case the fence-peer handler fails, I/O can be resumed manually with 'drbdadm resume-io'. 🧊
  • --ko-count number — if a secondary node fails to complete a write request in ko-count times the timeout parameter, it is excluded from the cluster. The primary node then sets the connection to this secondary node to Standalone. To disable this feature, you should explicitly set it to 0; defaults may change between versions. 🚫
  • --load-balance-paths {yes | no} — by default, the TCP transport establishes only one configured path at a time. It switches to another path only in case the established one fails. When you set load-balance-paths to yes the TCP transport establishes all paths in parallel. It will transmit data packets over the paths with the least data in its socket send queue. 🌈

Please note enabling load-balancing introduces additional chunking headers into the network protocol. In other words, you must enable it on both sides of a connection.

As of drbd-9.2.6 the RDMA transport does not obey this setting. It always uses all paths in parallel. This option became available with drbd-9.2.6. ✨

  • --max-buffers number — caps how much memory each DRBD minor device is allowed to hog on the receiving side, and it also covers internal buffers while you're doing a resync or an online-verify. The unit is PAGE_SIZE, which is 4 KiB on most systems. The smallest you're allowed to go is hard coded to 32 (that's 128 KiB bestie, do the math 💅). These buffers are literally where data blocks chill while they get written to / read from disk. And here's the tea: to dodge possible distributed deadlocks when things get congested, this setting acts as a throttle threshold, not a hard limit. The moment more than max-buffers pages are in use, any further allocation from this pool gets throttled. You wanna bump max-buffers up if you can't saturate the IO backend on the receiving side. 📈
  • --max-epoch-size number — sets the maximum number of write requests DRBD may issue before it issues a write barrier. Default is 2048, minimum is 1, maximum is 20000. Setting this below 10 is likely to decrease performance, so like... don't. 🙈
  • --on-congestion policy,
  • --congestion-fill threshold,
  • --congestion-extents threshold — okay so by default DRBD just blocks when the TCP send queue is full. That stops applications from generating even more write requests until more buffer space frees up again.

When DRBD is used together with DRBD-proxy, it can actually be better to use the pull-ahead on-congestion policy, which can switch DRBD into ahead/behind mode before the send queue is full. DRBD then records the differences between itself and the peer in its bitmap, but it no longer replicates them to the peer. When enough buffer space becomes available again, the node resynchronizes with the peer and switches back to normal replication. 🦋

The glow-up here: it doesn't block application I/O even when the queues fill up. The trade-off: peer nodes can fall behind much further. Also, while resynchronizing, peer nodes will become inconsistent. Not me warning you twice 😭

The available congestion policies are block (the default) and pull-ahead. The congestion-fill parameter defines how much data is allowed to be "in flight" in this connection. The default value is 0, which disables this mechanism of congestion control, with a maximum of 10 GiBytes. The congestion-extents parameter defines how many bitmap extents may be active before switching into ahead/behind mode, with the same default and limits as the al-extents parameter. The congestion-extents parameter is effective only when set to a value smaller than al-extents.

Ahead/behind mode is available since DRBD 8.3.10. ✨
  • --ping-int interval — when the TCP/IP connection to a peer is idle for more than ping-int seconds, DRBD will send a keep-alive packet to make sure that a failed peer or network connection is detected reasonably soon. Default is 10 seconds, minimum 1, maximum 120. The unit is seconds. ⏰
  • --ping-timeout timeout — defines the timeout for replies to keep-alive packets. If the peer does not reply within ping-timeout, DRBD will close and try to reestablish the connection. Default is 0.5 seconds, minimum 0.1 seconds, maximum 30 seconds. The unit is tenths of a second. (yes, really — that's what the manpage says, don't come for me 🫶)
  • --socket-check-timeout timeout — in setups involving a DRBD-proxy and connections that experience a lot of buffer-bloat, it might be necessary to set ping-timeout to an unusually high value. By default DRBD uses the same value to wait if a newly established TCP-connection is stable. Since the DRBD-proxy is usually located in the same data center, such a long wait time may hinder DRBD's connect process. 🐌

In such setups socket-check-timeout should be set to at least the round trip time between DRBD and DRBD-proxy. I.e. in most cases to 1.

The default unit is tenths of a second, the default value is 0 (which causes DRBD to use the value of ping-timeout instead). Introduced in 8.4.5. ⭐

  • --protocol name — use the specified protocol on this connection. The supported protocols are, and I need you to pay attention bestie:
    • A — Writes to the DRBD device complete as soon as they have reached the local disk and the TCP/IP send buffer.
    • B — Writes to the DRBD device complete as soon as they have reached the local disk, and all peers have acknowledged the receipt of the write requests.
    • C — Writes to the DRBD device complete as soon as they have reached the local and all remote disks. This is the default protocol. 👑
  • --rcvbuf-size size — configure the size of the TCP/IP receive buffer. A value of 0 (the default) causes the buffer size to adjust dynamically. This parameter usually does not need to be set, but it can be set to a value up to 128 MiB. The default unit is bytes. 🌊
  • --rdma-ctrl-rcvbuf-size value — by default, the RDMA transport divides the rcvbuf-size by 64 and uses the result for the number of buffers on the control stream. This result might be too low depending on the timing characteristics of the backing storage devices and the network link.

The option rdma-ctrl-rcvbuf-size allows you to explicitly set the number of buffers for the control stream, overruling the divide by 64 heuristics. The default unit of this setting is bytes.

  • --rdma-ctrl-sndbuf-size value — by default, the RDMA transport divides the sndbuf-size by 64 and uses the result for the number of buffers on the control stream. This result might be too low depending on the timing characteristics of the backing storage devices and the network link.

The option rdma-ctrl-sndbuf-size allows you to explicitly set the number of buffers for the control stream, overruling the divide by 64 heuristics. The default unit of this setting is bytes.

  • --rr-conflict policy — this option helps to solve the cases when the outcome of the resync decision is incompatible with the current role assignment in the cluster. The defined policies are:
    • disconnect — No automatic resynchronization, simply disconnect. ✌️
    • retry-connect — Disconnect now, and retry to connect immediatly afterwards.
    • violently — Resync to the primary node is allowed, violating the assumption that data on a block device are stable for one of the nodes. Do not use this option, it is dangerous. 💀
    • call-pri-lost — Call the pri-lost handler on one of the machines. The handler is expected to reboot the machine, which puts it into secondary role.
    • auto-discard — Auto-discard reverses the resync direction, so that DRBD resyncs the current primary to the current secondary. Auto-discard only applies when protocol A is in use and the resync decision is based on the principle that a crashed primary should be the source of a resync. When a primary node crashes, it might have written some last updates to its disk, which were not received by a protocol A secondary. By promoting the secondary in the meantime the user accepted that those last updates have been lost. By using auto-discard you consent that the last updates (before the crash of the primary) should be rolled back automatically.
  • --shared-secret secret — configure the shared secret used for peer authentication. The secret is a string of up to 64 characters. Peer authentication also requires the cram-hmac-alg parameter to be set. 🔐
  • --sndbuf-size size — configure the size of the TCP/IP send buffer. Since DRBD 8.0.13 / 8.2.7, a value of 0 (the default) causes the buffer size to adjust dynamically. Values below 32 KiB are harmful to the throughput on this connection. Large buffer sizes can be useful especially when protocol A is used over high-latency networks; the maximum value supported is 128 MiB. 📦
  • --tcp-cork — by default, DRBD uses the TCP_CORK socket option to prevent the kernel from sending partial messages; this results in fewer and bigger packets on the network. Some network stacks can perform worse with this optimization. On these, the tcp-cork parameter can be used to turn this optimization off. 🍾
  • --timeout time — defines the timeout for replies over the network: if a peer node does not send an expected reply within the specified timeout, it is considered dead and the TCP/IP connection is closed. The timeout value must be lower than connect-int and lower than ping-int. The default is 6 seconds; the value is specified in tenths of a second.
  • --tls bool-value — Enable TLS. 🔒
  • --tls-keyring key-description — Key description (name) of the keyring where the TLS key material is stored. The keyring will be shared with the handshake daemon.
  • --tls-privkey key-description — Key description (name) of the DER encoded private key for TLS encryption.
  • --tls-certificate key-description — Key description (name) of the DER encoded certificate for TLS encryption.
  • --transport type — With DRBD9 the network transport used by DRBD is loaded as a seperate module. With this option you can specify which transport and module to load. At present only two options exist, tcp and rdma. Default is tcp. 🚚
  • --use-rle — Each replicated device on a cluster node has a separate bitmap for each of its peer devices. The bitmaps are used for tracking the differences between the local and peer device: depending on the cluster state, a disk range can be marked as different from the peer in the device's bitmap, in the peer device's bitmap, or in both bitmaps. When two cluster nodes connect, they exchange each other's bitmaps, and they each compute the union of the local and peer bitmap to determine the overall differences. 🗺️

Bitmaps of very large devices are also relatively large, but bestie, they usually compress so well with run-length encoding 💖 That can save you time and bandwidth on the bitmap transfers, which is lowkey a whole mood.

The use-rle parameter decides whether run-length encoding gets used at all. It's on by default since DRBD 8.4.0. Periodt.

  • --verify-alg hash-algorithm — Online verification (drbdadm verify) computes and compares checksums of disk blocks (aka hash values) to figure out if they differ 👀 The verify-alg parameter picks which algorithm gets used for those checksums. It must be set to one of the secure hash algorithms the kernel supports before online verify can even be used — go peek at the shash algorithms listed in /proc/crypto. We recommend scheduling online verifications regularly during low-load periods, like once a month, fr fr. Also check the notes on data integrity below 🫶

drbdsetup new-path resource peer_node_id local-addr remote-addr

The new-path command creates a path within a connection. The connection must have already been created with drbdsetup new-peer — no skipping steps, bestie 💅

Local_addr and remote_addr refer to the local and remote protocol, network address, and port, in the format [address-family:]address[:port].

The address families supported are ipv4, ipv6, ssocks (Dolphin Interconnect Solutions' "super sockets"), sdp (Infiniband Sockets Direct Protocol), and sci (sci is just an alias for ssocks, iykyk).

If no address family is specified, ipv4 is assumed. For every address family except ipv6, the address uses IPv4 address notation (for example, 1.2.3.4). For ipv6, the address gets wrapped in brackets and uses IPv6 address notation (for example, [fd01:2345:6789:abcd::1]). The port defaults to 7788 ✨

drbdsetup connect resource peer_node_id

The connect command activates a connection. What that means: the DRBD driver will bind and listen on all local addresses of the connection's paths, and it will begin trying to establish one or more paths of the connection 🌈

Available options:

  • --tentative — Only determine if a connection to the peer can be established, and if a resync is necessary (and in which direction), without actually establishing the connection or starting the resync. Check the system log to see what DRBD would do without the --tentative option 👀
  • --discard-my-data — Discard the local data and resynchronize with the peer that has the most up-to-date data. Use this to manually recover from a split-brain situation 💀

drbdsetup del-peer resource peer_node_id

The del-peer command removes a connection from a resource. Bye bestie 🦋

drbdsetup del-path resource peer_node_id local-addr remote-addr

The del-path command removes a path from a connection. Heads up though — it fails if the path is necessary to keep a connected connection intact. So if you wanna remove all the paths, disconnect the connection first 🙈

drbdsetup cstate resource peer_node_id

Shows the current state of a connection. The connection is identified by the node-id of the peer — see the drbdsetup connect command ✨

drbdsetup del-minor minor

Remove a replicated device. No lower-level device may be attached — see drbdsetup detach 🎀

drbdsetup del-resource resource

Remove a resource. All volumes and connections must be removed first (drbdsetup del-minor, drbdsetup disconnect). Alternatively, drbdsetup down can be used to remove a resource together with all its volumes and connections 💜

drbdsetup detach minor

Detach the lower-level device of a replicated device. Available options:

  • --force, --diskless — Force the detach and return immediately. This puts the lower-level device into failed state until all pending I/O has completed, and then detaches the device. Any I/O not yet submitted to the lower-level device (for example, because I/O on the device was suspended) is assumed to have failed 💀 Mark the detached volume as "permanently diskless", converting it to an "intentionally diskless client" — as opposed to temporarily diskless, which is meant to be re-attached later 👑

drbdsetup disconnect resource peer_node_id

Remove a connection to a peer host. The connection is identified by the node-id of the peer — see the drbdsetup connect command 🌸

drbdsetup down {resource | all}

Take a resource down by removing all volumes, connections, and the resource itself. Full clean slate energy ✨

drbdsetup dstate minor

Show the current disk state of a lower-level device 🔍

drbdsetup events2 {resource | all}

Show the current state of all configured DRBD objects, followed by all the changes to that state 👑

The output format is meant to be human readable and machine readable, which is giving versatile icon behavior. The line starts with a word indicating the kind of event:

  • exists — for an existing object
  • create, destroy, and change — if an object is created, destroyed, or changed
  • call or response — if an event handler is called or it returns
  • rename — when the name of an object is changed

The second word indicates the object the event applies to: resource, device, connection, peer-device, path, helper, or a dash (-) to indicate that the current state has been dumped completely.

The remaining words identify the object and describe the state the object is in. Some special keys are worth mentioning, bestie:

  • resource may_promote:{yes|no} — Whether promoting to primary is expected to succeed. When quorum is enabled, this can be used to trigger failover. When may_promote:yes is reported on this node, then no writes are possible on any other node, which generally means the application can be started on this node, even when it has been running on another 🌈
  • resource promotion_score:score — An integer heuristic indicating the relative preference for promoting this resource. A higher score is better, in terms of having local disks and having access to up-to-date data. The score may be positive even when some node is primary. It will be zero when promotion is impossible due to quorum or lack of any access to up-to-date data.

Available options:

  • --now — Terminate after reporting the current state. The default is to continuously listen and report state changes 👀
  • --poll — Read from stdin and update when n is read. Newlines are ignored. Every other input terminates the command.

Without --now, changes are printed as usual. On each n the current state is fetched, but only changed objects are printed. This is useful with --statistics or --full, because DRBD does not otherwise send updates when only the statistics change.

In combination with --now, the full state is printed on each n. No other changes are printed ✨

  • --statistics — Include statistics in the output 📊
  • --diff — Write information in the form of a diff between old and new state. This helps simple tools avoid (old) state tracking on their own, so they can just vibe 💅
  • --full — Write complete state information, especially on change events. This enables --statistics 🔥
  • --timestamps — Prefix event lines with a timestamp in ISO 8601 format (YYYY-MM-DDThh:mm:ss.ssssss±hh:mm). The timestamp is generated by drbdsetup just before it prints the event line(s). If a single event causes multiple lines to be printed, they should all have the same timestamp prefix ⏰
  • --rcvbuf — On a "busy" system, you may need to increase the netlink receive buffer size, because "too much" activity causes many events to be multicast via netlink, which may overrun the receive buffer 😭

You can adjust the receive buffer size by setting the environment variable DRBD_GENL_RCVBUF_SZ or by using this command line option. Command line wins if both are used.

Value in bytes, accepts unit suffixes, defaults to 1MiB. Capped for sanity between 100k and 100M.

You may need to adjust net.core.wmem_max to be effective without CAP_NET_ADMIN.

  • --color={always | auto | never} — Colorize the output. With --color=auto, drbdsetup emits color codes only when standard output is connected to a terminal 🎀

drbdsetup get-gi resource peer_node_id volume

Show the data generation identifiers for a device on a particular connection. The device is identified by its volume number. The connection is identified by its endpoints — see the drbdsetup connect command 🔍

The output consists of the current UUID, bitmap UUID, and the first two history UUIDs, followed by a set of flags. The current UUID and history UUIDs are device specific; the bitmap UUID and flags are peer device specific. This command only shows the first two history UUIDs. Internally, DRBD maintains one history UUID for each possible peer device 👑

drbdsetup invalidate minor

Replace the local data of a device with that of a peer. All the local data will be marked out-of-sync, and a resync with the specified peer device will be initiated 😤

Available options:

  • --reset-bitmap=no — Usually an invalidate operation sets all bits in the bitmap to out-of-sync before beginning the resync from the peer. By giving --reset-bitmap=no, DRBD will use the bitmap as it is. Usually this is used after an online verify operation found differences in the backing devices 👀

The --reset-bitmap option is available since DRBD kernel driver 9.0.29 and drbd-utils 9.17.

  • --sync-from-peer-node-id — This option allows the caller to select the node to resync from. If it is not given, DRBD selects a suitable source node itself ✨

drbdsetup invalidate-remote resource peer_node_id volume

Replace a peer device's data of a resource with the local data. The peer device's data will be marked out-of-sync, and a resync from the local node to the specified peer will be initiated 💅

Available options:

  • --reset-bitmap=no — Usually an invalidate remote operation sets all bits in the bitmap to out-of-sync before beginning the resync to the peer. By giving --reset-bitmap=no, DRBD will use the bitmap as it is. Usually this is used after an online verify operation found differences in the backing devices 🫶

The --reset-bitmap option is available since DRBD kernel driver 9.0.29 and drbd-utils 9.17.

drbdsetup new-current-uuid minor

okay so this one is the glow-up command 💅 it generates a brand new current UUID and rotates all the other UUID values while it's at it. Three use cases, bestie, and they're all lowkey iconic:

  • start the initial resync
  • skip the initial resync
  • bootstrap a single node cluster

Available options:

  • --force-resync — starts an initial resync. The catch (and there's always a catch) is that a precondition is the volume being in disk state Inconsistent on all nodes. This command updates the disk state on the current node to UpToDate and makes it the source of the resync operations to the peers 👑
  • --clear-bitmap — clears the sync bitmap in addition to generating a new current UUID. This is the one that skips the initial resync entirely. As a consequence, this volume's disk state changes to UpToDate on all nodes in this resource ✨

Both operations require a "Just Created" meta data. Here is the complete sequence, step by step, for how to skip the initial resync:

  1. On both nodes, initialize meta data and configure the device.
drbdadm create-md --force res/volume-number
  1. They need to do the initial handshake, so they know their sizes.
drbdadm up res
  1. They are now Connected Secondary/Secondary Inconsistent/Inconsistent. Generate a new current-uuid and clear the dirty bitmap.
drbdadm --clear-bitmap new-current-uuid res
  1. They are now Connected Secondary/Secondary UpToDate/UpToDate. Make one side primary and create a file system.
drbdadm primary res

mkfs -t fs-type $(drbdadm sh-dev res/vol)

One obvious side-effect: the replica is full of old garbage 🗑️ (unless you made them identical using other means), so any online-verify is expected to find any number of out-of-sync blocks. Not a vibe, but there it is.

You must not use this on pre-existing data! Even though it may appear to work at first glance, once you switch to the other node, your data is toast 🔥, as it never got replicated. So do not leave out the mkfs (or equivalent).

Bootstraping a single node cluster

This can also be used to shorten the initial resync of a cluster where the second node is added after the first node is gone into production, by means of disk shipping. This use-case works on disconnected devices only, and the device may be in primary or secondary role.

The necessary steps on the current active server are:

  1. drbdsetup new-current-uuid --clear-bitmap minor
  2. Take the copy of the current active server. E.g. by pulling a disk out of the RAID1 controller, or by copying with dd. You need to copy the actual data, and the meta data.
  3. drbdsetup new-current-uuid minor

Now add the disk to the new secondary node, and join it to the cluster. You will get a resync of that parts that were changed since the first call to drbdsetup in step 1.

drbdsetup new-minor resource minor volume

Creates a new replicated device within a resource 🦋 The command creates a block device inode for the replicated device (by default, /dev/drbdminor). The volume number identifies the device within the resource.

--block-size size

Block storage devices have a particular sector size or block size, and this block size has so many different names — it's giving aliases. Examples are 'hw_sector_size', 'PHY-SEC', 'physical block (sector) size', and 'logical block (sector) size'.

DRBD needs to combine these block sizes of the backing disks. In clusters with storage devices with different block sizes, it is necessary to configure the maximal block sizes on the DRBD level. Here is an example highlighting the need 👇

Let's say node A is diskless. It connects to node B, which has a physical block size of 512 bytes. Then the user mounts the filesystem on node A; the filesystem recognizes that it can do I/O in units of 512 bytes. Later, node C joins the cluster with a physical block size of 4096 bytes. Now, suddenly DRBD starts to deliver I/O errors to the filesystem if it chooses to do I/O on, e.g., 512 or 1024 bytes. 💀

The default value of block-size 512 bytes. This option is available since drbd-utils 9.24 and the drbd kernel driver 9.1.14 and 9.2.3.

--discard-granularity size

Sets the discard granularity advertised by the DRBD block device. Discard requests (TRIM/UNMAP) must be aligned to and be a multiple of this value.

The primary use case is a diskless primary node, where DRBD would otherwise hardcode a granularity of 512 bytes regardless of the actual storage topology on peers. Tools like LINSTOR know the real granularity from storage pool metadata and can set this option to advertise the correct value. The option is equally valid on diskful nodes when a larger granularity than the backing device strictly requires should be exposed (restrictions on other peers?).

A value of 0 disables discard support on the DRBD device entirely, even when the backing storage and all peers support discards. Any non-zero value enables discards and must be a multiple of block-size. When not set (the default), DRBD uses its previous behaviour: discard support is enabled at 512-byte granularity on diskless clients or inherited from the backing device.

This option is available since drbd-utils 9.34 and drbd kernel driver 9.3.1

drbdsetup new-resource resource node_id, and drbdsetup resource-options resource

drbdsetup new-resource resource node_id,
drbdsetup resource-options resource

The new-resource command creates a new resource. The resource-options command changes the resource options of an existing resource. Available options:

--auto-promote bool-value

A resource must be promoted to primary role before any of its devices can be mounted or opened for writing.

Before DRBD 9, this could only be done explicitly (drbdadm primary). Since DRBD 9, the auto-promote parameter allows to automatically promote a resource to primary role when one of its devices is mounted or opened for writing. As soon as all devices are unmounted or closed with no more remaining users, the role of the resource changes back to secondary.

Automatic promotion only succeeds if the cluster state allows it (that is, if an explicit drbdadm primary command would succeed). Otherwise, mounting or opening the device fails as it already did before DRBD 9: the mount(2) system call fails with errno set to EROFS (Read-only file system); the open(2) system call fails with errno set to EMEDIUMTYPE (wrong medium type).

Irrespective of the auto-promote parameter, if a device is promoted explicitly (drbdadm primary), it also needs to be demoted explicitly (drbdadm secondary). No take-backsies 🙈

The auto-promote parameter is available since DRBD 9.0.0, and defaults to yes.

  • --auto-promote-timeout 1/10-of-seconds — when a user process promotes a drbd resource by opening one of its devices, DRBD waits up to auto-promote-timeout for the device to become promotable if it is not in the first place. It's specified in units of 0.1 seconds. Its default value is 20 (2 seconds), its minimum value is 0, and its maximum value is 600 (=one minute).
  • --cpu-mask cpu-mask — set the cpu affinity mask for DRBD kernel threads 🧚 The cpu mask is specified as a hexadecimal number. The default value is 0, which lets the scheduler decide which kernel threads run on which CPUs. CPU numbers in cpu-mask which do not exist in the system are ignored.
  • --max-io-depth value — this limits the number of outstanding requests on a DRBD device. Any process that tries to issue more I/O requests will sleep in "D state" (uninterruptible by signals) until some previously issued requests finish. 😴

max-io-depth has a default value of 8000, its minimum value is 4, and its maximum value is 2^32.

--on-no-data-accessible policy

Determine how to deal with I/O requests when the requested data is not available locally or remotely (for example, when all disks have failed). When quorum is enabled, on-no-data-accessible should be set to the same value as on-no-quorum. The defined policies are:

  • io-error — System calls fail with errno set to EIO.
  • suspend-io — The resource suspends I/O. I/O can be resumed by (re)attaching the lower-level device, by connecting to a peer which has access to the data, or by forcing DRBD to resume I/O with drbdadm resume-io res. When no data is available, forcing I/O to resume will result in the same behavior as the io-error policy.

This setting is available since DRBD 8.3.9; the default policy is io-error.

--on-no-quorum {io-error | suspend-io}

By default DRBD freezes IO on a device that lost quorum 🧊 By setting the on-no-quorum to io-error it completes all IO operations with an error if quorum is lost.

Usually, the on-no-data-accessible should be set to the same value as on-no-quorum, as it has precedence.

The on-no-quorum options is available starting with the DRBD kernel driver version 9.0.8.

--on-suspended-primary-outdated {disconnect | force-secondary}

This setting is only relevant when on-no-quorum is set to suspend-io. It is relevant in the following scenario. A primary node loses quorum hence has all IO requests frozen. This primary node then connects to another, quorate partition. It detects that a node in this quorate partition was promoted to primary, and started a newer data-generation there. As a result, the first primary learns that it has to consider itself outdated. 😭

When it is set to force-secondary then it will demote to secondary immediately, and fail all pending (and new) IO requests with IO errors. It will refuse to allow any process to open the DRBD devices until all openers closed the device. This state is visible in status and events2 under the name force-io-failures.

The disconnect setting simply causes that node to reject connect attempts and stay isolated. Like, it literally just ghosts everyone and vibes alone 💅

The on-suspended-primary-outdated option is available starting with the DRBD kernel driver version 9.1.7. It has a default value of disconnect.

  • --peer-ack-delay expiry-time — bestie, picture this: if after the last finished write request no new write request gets issued for expiry-time, then a peer-ack packet gets sent. But if a new write request gets issued before the timer expires, the timer gets reset to expiry-time. (Note: peer-ack packets may be sent due to other reasons as well, e.g. membership changes or the peer-ack-window option.) it's giving "timer resets, main character energy" ⏰✨

This parameter may influence resync behavior on remote nodes. Peer nodes need to wait until they receive an peer-ack for releasing a lock on an AL-extent. Resync operations between peers may need to wait for for these locks.

The default value for peer-ack-delay is 100 milliseconds, the default unit is milliseconds. This option is available since 9.0.0.

  • --peer-ack-window value — okay so on each node and for each device, DRBD maintains a bitmap of the differences between the local and remote data for each peer device. For example, in a three-node setup (nodes A, B, C) each with a single device, every node maintains one bitmap for each of its peers. 🦋

When nodes receive write requests, they know how to update the bitmaps for the writing node, but not how to update the bitmaps between themselves. In this example, when a write request propagates from node A to B and C, nodes B and C know that they have the same data as node A, but not whether or not they both have the same data. the drama of not knowing 😭

As a remedy, the writing node occasionally sends peer-ack packets to its peers which tell them which state they are in relative to each other.

The peer-ack-window parameter specifies how much data a primary node may send before sending a peer-ack packet. A low value causes increased network traffic; a high value causes less network traffic but higher memory consumption on secondary nodes and higher resync times between the secondary nodes after primary node failures. (Note: peer-ack packets may be sent due to other reasons as well, e.g. membership changes or expiry of the peer-ack-delay timer.) lowkey a trade-off, highkey a trade-off, fr fr

The default value for peer-ack-window is 2 MiB, the default unit is sectors. This option is available since 9.0.0.

  • --quorum value — when activated, a cluster partition requires quorum in order to modify the replicated data set. That means a node in the cluster partition can only be promoted to primary if the cluster partition has quorum. Every node with a disk directly connected to the node that should be promoted counts. If a primary node should execute a write request, but the cluster partition has lost quorum, it will freeze IO or reject the write request with an error (depending on the on-no-quorum setting). Upon loosing quorum a primary always invokes the quorum-lost handler. The handler is intended for notification purposes, its return code is ignored. the handler is literally just there to tell you the tea, it doesn't do anything 🫶

The option's value might be set to off, majority, all or a numeric value. If you set it to a numeric value, make sure that the value is greater than half of your number of nodes. Quorum is a mechanism to avoid data divergence, it might be used instead of fencing when there are more than two repicas. It defaults to off. repicas. iconic spelling, no notes ✨

If all missing nodes are marked as outdated, a partition always has quorum, no matter how small it is. I.e. If you disconnect all secondary nodes gracefully a single primary continues to operate. In the moment a single secondary is lost, it has to be assumed that it forms a partition with all the missing outdated nodes. In case my partition might be smaller than the other, quorum is lost in this moment. brutal but fair 💀

In case you want to allow permanently diskless nodes to gain quorum it is recommended to not use majority or all. It is recommended to specify an absolute number, since DBRD's heuristic to determine the complete number of diskfull nodes in the cluster is unreliable.

The quorum implementation is available starting with the DRBD kernel driver version 9.0.7.

  • --quorum-minimum-redundancy value — this option sets the minimal required number of nodes with an UpToDate disk to allow the partition to gain quorum. This is a different requirement than the plain quorum option expresses.

The option's value might be set to off, majority, all or a numeric value. If you set it to a numeric value, make sure that the value is greater than half of your number of nodes.

In case you want to allow permanently diskless nodes to gain quorum it is recommended to not use majority or all. It is recommended to specify an absolute number, since DBRD's heuristic to determine the complete number of diskfull nodes in the cluster is unreliable.

This option is available starting with the DRBD kernel driver version 9.0.10.

  • --twopc-retry-timeout 1/10-of-seconds — due to conflicting two-phase-commit sometimes DRBD needs to retry them. But if two nodes retry their intended two-phase-commits after the same time, they would end up in an endless retry loop. To avoid that, DRBD selects a random wait time within an upper bound, an exponential backoff, and a function of the retry number. The twopc-retry-timeout is a base multiplier for that function. randomness is the cure, periodt 🎲

twopc-retry-timeout has a default value of a (0.1 seconds), its minimum value is 1 (0.1 seconds), and its maximum value is 50 (5 seconds).

  • --twopc-timeout 1/10-of-seconds — in some situations, a DRBD cluster requires a cluster-wide coordinated state transition. A perfect example of this is the 'promote-to-primary' action. Even if two not directly connected nodes in a cluster try this action concurrently, it may only succeed for one of the two. only one winner, sorry bestie 👑

For these cluster-wide coordinated state transitions, DRBD implements a two-phase commit protocol. If a connection breaks in phase one (prepare packet sent), the coordinator of the two-phase commit might never get the expected reply packet.

A cluster in this state can not start any new cluster-wide coordinated state transition, as the already prepared one blocks all such attempts. After twopc-timeout all nodes abort the prepared transaction and unlock the cluster again. the unlock is the healing arc ✨

twopc-timeout has a default value of 300 (30 seconds), its minimum value is 50 (5 seconds), and its maximum value is 600 (one minute).

drbdsetup outdate minor

Mark the data on a lower-level device as outdated. This is used for fencing, and prevents the resource the device is part of from becoming primary in the future. See the --fencing disk option. 🎀

drbdsetup pause-sync resource peer_node_id volume

Stop resynchronizing between a local and a peer device by setting the local pause flag. The resync can only resume if the pause flags on both sides of a connection are cleared. it takes two, literally 💅

drbdsetup primary resource

Change the role of a node in a resource to primary. This allows the replicated devices in this resource to be mounted or opened for writing. Available options:

  • --overwrite-data-of-peer — this option is an alias for the --force option.
  • --force — force the resource to become primary even if some devices are not guaranteed to have up-to-date data. This option is used to turn one of the nodes in a newly created cluster into the primary node, or when manually recovering from a disaster.

Note that this can lead to split-brain scenarios. Also, when forcefully turning an inconsistent device into an up-to-date device, it is highly recommended to use any integrity checks available (such as a filesystem check) to make sure that the device can at least be used without crashing the system. no cap, do the integrity check 😭

Note that DRBD usually only allows one node in a cluster to be in primary role at any time; this allows DRBD to coordinate access to the devices in a resource across nodes. The --allow-two-primaries network option changes this; in that case, a mechanism outside of DRBD needs to coordinate device access.

drbdsetup resize minor

Reexamine the size of the lower-level devices of a replicated device on all nodes. This command is called after the lower-level devices on all nodes have been grown to adjust the size of the replicated device. Available options:

  • --assume-peer-has-space — resize the device even if some of the peer devices are not connected at the moment. DRBD will try to resize the peer devices when they next connect. It will refuse to connect to a peer device which is too small.
  • --assume-clean — do not resynchronize the added disk space; instead, assume that it is identical on all nodes. This option can be used when the disk space is uninitialized and differences do not matter, or when it is known to be identical on all nodes. See the drbdsetup verify command. assuming, but make it intentional ✨
  • --size val — this option can be used to online shrink the usable size of a drbd device. It's the users responsibility to make sure that a file system on the device is not truncated by that operation. your file system, your problem bestie 🍓
  • --al-stripes val --al-stripes val — these options may be used to change the layout of the activity log online. In case of internal meta data this may invovle shrinking the user visible size at the same time (unsing the --size) or increasing the avalable space on the backing devices. yes the flag is listed twice, yes we're keeping it exactly as-is 💜

drbdsetup resume-io minor

Resume I/O on a replicated device. See the --fencing net option.

drbdsetup resume-sync resource peer_node_id volume

Allow resynchronization to resume by clearing the local sync pause flag.

drbdsetup role resource

Show the current role of a resource. who am I? what's my role? the people need to know 👑

drbdsetup secondary resource

okay so you want this node to STOP being the main character and go back to being secondary. switching the role of a node in a resource to secondary. 💅

heads up bestie: this command fails if the replicated device is in use. like, actively being used. it will not do the thing.
  • --force — a forced demotion to secondary causes all pending and new IO requests to terminate with IO errors. 😭
please note that a forced demotion returns immediately. the user should unmount any filesystem that might be mounted on the DRBD device. the device can be used again when force-io-failures has a value of no. (See drbdsetup status and drbdsetup events2).

drbdsetup show {resource | all}

show the current configuration of a resource, or of all resources. it's giving "let me see the receipts" ✨ Available options:

  • --show-defaults — show all configuration parameters, even the ones with default values. Normally, parameters with default values are not shown.

drbdsetup show-gi resource peer_node_id volume

show the data generation identifiers for a device on a particular connection. In addition, explain the output. 🎀 The output otherwise is the same as in the drbdsetup get-gi command.

drbdsetup state

This is an alias for drbdsetup role. Deprecated. 💀

drbdsetup status {resource | all}

show the status of a resource, or of all resources. The output consists of one paragraph for each configured resource. Each paragraph contains one line for each resource, followed by one line for each device, and one line for each connection. The device and connection lines are indented. The connection lines are followed by one line for each peer device; these lines are indented against the connection line.

Long lines are wrapped around at terminal width, and indented to indicate how the lines belongs together. Available options:

  • --verbose — include more information in the output even when it is likely redundant or irrelevant.
  • --statistics — include data transfer statistics in the output.
  • --color={always | auto | never} — colorize the output. With --color=auto, drbdsetup emits color codes only when standard output is connected to a terminal. 🌈

For example, the non-verbose output for a resource with only one connection and only one volume could look like this:

drbd0 role:Primary
  disk:UpToDate
  host2.example.com role:Secondary
    disk:UpToDate

With the --verbose option, the same resource could be reported as:

drbd0 node-id:1 role:Primary suspended:no
  volume:0 minor:1 disk:UpToDate blocked:no
  host2.example.com local:ipv4:192.168.123.4:7788
      peer:ipv4:192.168.123.2:7788 node-id:0 connection:WFReportParams
      role:Secondary congested:no
    volume:0 replication:Connected disk:UpToDate resync-suspended:no

drbdsetup suspend-io minor

suspend I/O on a replicated device. 🫶 The primary purpose of this command is to enable the user to take snapshots of the backing device across multiple nodes consistently. like, freeze frame, everyone, on three 📸

  • --bdev-freeze={yes | no} — by default, DRBD tries to bring the filesystem mounted on the DRBD device into a consistent state using the kernel function bdev_freeze(). Passing --bdev-freeze=no suppresses that.

drbdsetup verify resource peer_node_id volume

start online verification, change which part of the device will be verified, or stop online verification. The command requires the specified peer to be connected.

Online verification compares each disk block on the local and peer node. Blocks which differ between the nodes are marked as out-of-sync, but they are not automatically brought back into sync. To bring them into sync, the drbdsetup invalidate or drbdsetup invalidate-remote with the --reset-bitmap=no option can be used. Progress can be monitored in the output of drbdsetup status --statistics. Available options:

  • --start position — define where online verification should start. This parameter is ignored if online verification is already in progress. If the start parameter is not specified, online verification will continue where it was interrupted (if the connection to the peer was lost while verifying), after the previous stop sector (if the previous online verification has finished), or at the beginning of the device (if the end of the device was reached, or online verify has not run before).

The position on disk is specified in disk sectors (512 bytes) by default.

  • --stop position — define where online verification should stop. If online verification is already in progress, the stop position of the active online verification process is changed. Use this to stop online verification. 🛑

The position on disk is specified in disk sectors (512 bytes) by default.

Also see the notes on data integrity in the drbd.conf(5) manual page.

drbdsetup wait-connect-volume resource peer_node_id volume, drbdsetup wait-connect-connection resource peer_node_id, drbdsetup wait-connect-resource resource, drbdsetup wait-sync-volume resource peer_node_id volume, drbdsetup wait-sync-connection resource peer_node_id, drbdsetup wait-sync-resource resource

The wait-connect-* commands waits until a device on a peer is visible. The wait-sync-* commands waits until a device on a peer is up to date. ⏳ Available options for both commands:

  • --degr-wfc-timeout timeout — define how long to wait until all peers are connected in case the cluster consisted of a single node only when the system went down. This parameter is usually set to a value smaller than wfc-timeout. The assumption here is that peers which were unreachable before a reboot are less likely to be reachable after the reboot, so waiting is less likely to help.

The timeout is specified in seconds. The default value is 0, which stands for an infinite timeout. Also see the wfc-timeout parameter.

  • --outdated-wfc-timeout timeout — define how long to wait until all peers are connected if all peers were outdated when the system went down. This parameter is usually set to a value smaller than wfc-timeout. The assumption here is that an outdated peer cannot have become primary in the meantime, so we don't need to wait for it as long as for a node which was alive before.

The timeout is specified in seconds. The default value is 0, which stands for an infinite timeout. Also see the wfc-timeout parameter.

  • --wait-after-sb — this parameter causes DRBD to continue waiting in the init script even when a split-brain situation has been detected, and the nodes therefore refuse to connect to each other. 🙈
  • --wfc-timeout timeout — define how long the init script waits until all peers are connected. This can be useful in combination with a cluster manager which cannot manage DRBD resources: when the cluster manager starts, the DRBD resources will already be up and running. With a more capable cluster manager such as Pacemaker, it makes more sense to let the cluster manager control DRBD resources. The timeout is specified in seconds. The default value is 0, which stands for an infinite timeout. Also see the degr-wfc-timeout parameter.

drbdsetup forget-peer resource peer_node_id

The forget-peer command removes all traces of a peer node from the meta-data. It frees a bitmap slot in the meta-data and make it avalable for futher bitmap slot allocation in case a so-far never seen node connects. 🧹✨

The connection must be taken down before this command may be used. In case the peer re-connects at a later point a bit-map based resync will be turned into a full-sync.

fr fr, remember that last part. forgetting is not free. 💀

drbdsetup rename-resource resource new_name

change the name of resource to new_name on the local node. Note that, since there is no concept of resource names in DRBD's network protocol, it is technically possible to have different names for a resource on different nodes. However, it is strongly recommended to issue the same rename-resource command on all nodes to have consistent naming across the cluster. 👑

A rename event will be issued on the events2 stream to notify users of the new name.

✨ EXAMPLES ✨

Please see the DRBD User's Guide[1] for examples. (yes that's the whole section, she's brief and she ate 💅)

🎀 VERSION 🎀

This document was revised for version 9.0.0 of the DRBD distribution.

🫶 AUTHOR 🫶

Written by Philipp Reisner <philipp.reisner@linbit.com> and Lars Ellenberg <lars.ellenberg@linbit.com>.

🐛 REPORTING BUGS 🐛

Report bugs to <drbd-user@lists.linbit.com>.

Copyright 2001-2018 LINBIT Information Technologies, Philipp Reisner, Lars Ellenberg. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

👀 SEE ALSO 👀

drbd.conf(5), drbd(8), drbdadm(8), DRBD User's Guide[1], DRBD Web Site[2]

📝 NOTES 📝

  1. DRBD User's Guide http://www.drbd.org/users-guide/
  2. DRBD Web Site http://www.drbd.org/

okay bestie that's the whole page, we did it, every last option accounted for 🏁💖 this one lives in DRBD 9.0.x, dated 17 January 2018, and she goes by DRBDSETUP(8). go forth and demote responsibly, mwah 💅✨

📦 where this page comes from