LVMLOCKD(8)

📦
version 2.03.43 · 10 versions
, in centos-stream 10

🎀 LVMLOCKD(8) 🎀

okay so bestie, welcome to the lvmlockd page — the LVM locking daemon manual, straight from the System Manager's Manual. sit down, grab a snack, we're learning about shared VGs 💖

✨ NAME ✨

lvmlockd — the LVM locking daemon. that's it, that's the name, she's iconic 👑

✨ SYNOPSIS ✨

lvmlockd [options]

literally just run her with whatever options you need, we'll get to those 🫶

✨ DESCRIPTION ✨

so a shared Volume Group is a VG that lives on shared storage devices (think SAN vibes ✨) and can be used by all the hosts that can access those devices. a shared VG requires lvmlockd and an external lock manager. no exceptions, periodt.

different hosts can activate and use LVs in the same shared VG, and different hosts can also create, extend, or remove LVs in that VG. locking, through lvmlockd, is what makes all of that safe and consistent — she's the main character keeping everyone in line 💅. LVM commands request locks from lvmlockd, which then passes those lock requests along to an external lock manager. each shared VG has a locktype attribute that says which external lock manager to use for that VG.

the lock manager options are:

  • sanlock — places locks on storage space reserved within the VG 🍓
  • dlm — uses a network locking protocol, and comes with additional clustering requirements 🌈

there are three types of locking performed through lvmlockd:

  • per-VG locks — protect changes to the VG metadata 📝
  • LV locks — limit LV activation to one host at a time 🎀
  • A Global lock — protects unused PVs and the VG namespace 👑

✨ OPTIONS ✨

daemon command line options, let's gooo:

  • -h, --help — shows this help information 🫶
  • -V, --version — shows the version of lvmlockd ⭐
  • -T, --test — test mode, does not call the lock manager 🙈
  • -f, --foreground — don't fork.
  • -D, --daemon-debug — don't fork, and print debugging to stdout 🐛
  • -p, --pid-file path — set the path to the pid file
  • -s, --socket-path path — set the path to the socket to listen on
  • --adopt-file path — set the path to the adopt file
  • -S, --syslog-priority err|warning|debug — write log messages from this level up to syslog
  • -g, --gl-type sanlock|dlm — set the global lock type to be sanlock or dlm
  • -i, --host-id num — set the local sanlock host id
  • -F, --host-id-file path — a file containing the local sanlock host_id
  • -o, --sanlock-timeout seconds — override the default sanlock I/O timeout ⏱️
  • -A, --adopt 0|1 — enable (1) or disable (0) lock adoption

✨ SETUP ✨

the following steps are the quick overview of how to use shared VGs. more deets live down in SETUP DETAILS 👇

🦋 Devices

identify the shared device(s) that will be used in the shared VG, and add them to each host's devices file (local device names often differ, unless a WWN-based name from /dev/disk/by-id is used, iykyk).

$ lvmdevices --adddev device

🦋 Configuration

edit lvm.conf, setting use_lvmlockd=1 on each host using a shared VG 💖

edit lvmlocal.conf, setting host_id on each host (only required when using sanlock). the host_id is a unique integer for each host between 1 and 2000. slay, unique numbers only 🔢

🦋 Locking

start lvmlockd and the external lock manager (sanlock or dlm) on each host.

🦋 VG

create a shared VG from one host (uses the running lock manager):

$ vgcreate --shared VG devices

include vgcreate options to use Persistent Reservations (sanlock only) — --setpersist y --setlockargs persist,notimeout ✨

start Persistent Reservations for the VG (if they are used):

$ vgchange --persist start VG

start the lockspace for the shared VG on all hosts:

$ vgchange --lockstart VG

🦋 Usage

begin using the VG, e.g. creating LVs 🎀

regular shutdown steps:

$ vgchange -an VG
$ vgchange --lockstop [--persist stop] VG
$ stop lvmlockd and lock manager

regular startup steps:

$ start lvmlockd and lock manager
$ vgchange --lockstart [--persist start] VG

✨ SETUP DETAILS ✨

🧚 Identifying devices

devices often have different local names on each host — like, /dev/sda on one host is named /dev/sdb on another, which is genuinely so annoying 😭. one method to identify the same device on each host is to run the command lsblk -o+WWN on each host, looking for the same WWN, and its corresponding local device name. or, the WWN-based device symlinks under /dev/disk/by-id/ can be a useful way to consistently identify a device across multiple hosts 🦋

🧚 Device access

on each host, LVM needs to be given access to the shared devices by adding them to the local system.devices file (unless the devices file feature has been disabled, in which case, different story). use the local device name identified in the previous step, or a WWN-based device symlink, and run the command lvmdevices --adddev device_path to add each of the shared devices to the local system.devices file. alternatively, after the shared VG has been created from one host, other hosts can run the command vgimportdevices VG to add devices from VG to system.devices. see lvmdevices(8) for more information 🫶

🧚 Config files

after /etc/lvm/lvm.conf use_lvmlockd has been set to 1, lvm commands will begin attempting to acquire locks from lvmlockd. failures to acquire the global lock from lvmlockd may appear until the first shared VG is operational. these warnings won't interfere with lvm commands doing read operations, but lvm commands attempting to write may fail with a global lock error 👀 (not me warning you in advance)

the /etc/lvm/lvmlocal.conf host_id setting is only required when using VGs with lock type sanlock. each host using sanlock VGs must be configured with a unique host_id value. valid host_ids are integers in the range 1-2000. (the lvmlocal.conf descriptions of host_id and sanlock_align_size mention special cases with 4K disks in which the valid host_id range can be smaller, e.g. 1-250.) 💀 so read the fine print bestie

🧚 Lock managers

a specific build of LVM may exclude support for sanlock or dlm lock managers. to check this, run the command lvm version and look for --enable-lvmlockd-sanlock or --enable-lvmlockd-dlm 🔍

with the --shared option, vgcreate searches for a running lock manager (dlm or sanlock), and uses the result for the VG lock type. in place of --shared, vgcreate --locktype sanlock|dlm can be specified directly ✨

when using lvmlockd in a cluster which already uses corosync and dlm, then the dlm lock type should be used for shared VGs. otherwise, using sanlock is usually more straightforward 🍓

details on starting or setting up each lock manager are outside the scope of LVM. sanlock can usually be started with a simple systemctl start wdmd sanlock. steps for setting up and starting corosync and dlm can be more complicated 😮‍💨

🧚 Lock start

shared VGs must be started before they are used, via vgchange --lockstart VG. starting the VG performs lock manager initialization that is necessary to begin using locks (i.e. creating and joining a lockspace). starting the VG may take some time, and until the start completes the VG may not be modified or activated ⚠️. when shutting down, the lockspace is stopped with vgchange --lockstop VG 🎀

🧚 Persistent Reservations

a shared VG with locktype sanlock can take advantage of Persistent Reservations (PR) for faster and more reliable recovery 🔥. this requires all of the devices in the VG to support PR. test if PR is supported by a device with the command:

$ lvmpersist devtest --device device

the vgcreate command options when enabling PR-based recovery with sanlock:

$ vgcreate --shared --setpersist y --setlockargs persist,notimeout

when enabled, PR needs to be started for the VG before lockstart:

$ vgchange --persist start VG

or start PR and locking with one command, efficiency queen 👑:

$ vgchange --persist start --lockstart VG

🧚 Atomic leases

sanlock can use atomic leases when devices support the SCSI COMPARE AND WRITE (CAW) command (sometimes referred to as atomic test and set.) atomic/CAW leases are faster and more efficient than the lease algorithms used without CAW (disk-paxos for resource leases, and time-delay leases for lockspace leases.) ⚡

sanlock leases are configured to use CAW when they are initialized during vgcreate. CAW usage depends on the sanlock configuration, device COMPARE AND WRITE support, and vgcreate --setlockargs options:

  • --setlockargs caw — use CAW if supported by the device and allowed by the sanlock configuration 💖
  • --setlockargs nocaw — do not use CAW, even if supported by the device and the sanlock configuration 🙅‍♀️

if neither option is specified, CAW usage will depend on the sanlock configuration and device support 🌸

✨ TOPICS ✨

🌈 Displaying shared VGs

the vgs command shows the letter s in the last (sixth) attr position for a shared VG:

$ vgs
  VG    #PV #LV #SN Attr   VSize   VFree
  vgfoo   1   0   0 wz--ns 992.00m 736.00m

or, the shared attribute can be displayed as a vgs reporting field:

$ vgs -o+shared
  VG    #PV #LV #SN Attr   VSize   VFree   Shared
  vgfoo   1   0   0 wz--ns 992.00m 736.00m  shared

the vgs command can also display the lock type:

$ vgs -o+locktype
  VG    #PV #LV #SN Attr   VSize   VFree   LockType
  vgfoo   1   0   0 wz--ns 992.00m 736.00m sanlock

on hosts that are attached to the shared devices, but do not have lvmlockd enabled, the vgs command will not display shared VGs unless the --shared option is added 🫶:

$ vgs --shared
  VG    #PV #LV #SN Attr   VSize   VFree
  vgfoo   1   0   0 wz--ns 992.00m 736.00m

🌈 Persistent Reservations

PR-based lock recovery for a shared VG is possible when using sanlock 🎀. in the standard form of this feature, lvmlockd removes the PR key of a failed host, and then permits sanlock to grant locks that had been held by the failed host. LVM commands on other hosts can then acquire those locks, e.g. activate LVs that the failed host had active at the time it failed. removal of PR keys for failed hosts can be considered a form of "fencing" ⚔️. for more information on using PR with LVM in general, see lvmpersist(8).

💜 Enabling

the vgcreate options to enable PR-based lock recovery in a new VG (also shown above) are:

$ vgcreate --shared --setpersist y --setlockargs persist,notimeout ...

use vgchange to enable PR-based lock recovery in an existing VG. first enable PR for the VG (if not already done), then change the VG lock args. note that changing lock args can involve stopping/starting both PR and the lockspace at different stages 😭. changing the PR or lock args settings requires the VG to be stopped on all other nodes ⚠️

example of enabling using vgchange:

$ vgcreate --shared VG devices
$ vgchange --setpersist y VG
$ vgchange --persist start VG
$ vgchange --setlockargs persist,notimeout VG
$ vgchange --persist start --lockstart VG

display the VG attributes configured by setpersist and setlockargs:

$ vgs -o+persist VG
$ vgs -o+lockargs VG

✨ Starting ✨

Okay so bestie, listen — when PR is enabled for a shared VG (that's the --setpersist y option, iykyk), LVM is so particular about the order of things. There's a whole startup sequence that has to happen before the VG is even usable:

  1. PR goes first — this step registers the local PR key and acquires a reservation on the VG devices.
  2. The lock manager goes second — this is what enables distributed locks to be acquired for the VG.

Both of these startup steps have to complete successfully before VG modifications or LV activations are allowed. Like, no shortcuts, periodt. 💅

You can start PR and locking in separate commands, or be efficient and do it in a single command:

$ vgchange --persist start VG; vgchange --lockstart VG
$ vgchange --persist start --lockstart VG

🎀 Variations 🎀

The sanlock lock manager can be configured to handle recovery of failed hosts in different ways. The first variation (persist,notimeout) is considered the standard form of PR-based lock recovery. The second variation (nopersist,timeout) is the default behavior of lvmlockd/sanlock when no setlockargs value is specified — and heads up, the default settings "nopersist" and "timeout" do not appear in the lockargs reporting field. 🙈

  • --setlockargs persist,notimeout
    • persist: lvmlockd will remove the PR key of a failed host when a lock request fails due to a lock owned by a failed host. After PR key removal, sanlock is permitted to grant the lock.
    • notimeout: a failed host's locks will neither expire nor be granted to another host after a timeout (i.e. after the time when the failed host's watchdog has fired.)
    • Removal of the PR key is the only mechanism by which a failed host's locks are recovered.
  • --setlockargs nopersist,timeout
    • nopersist: lvmlockd will not remove a PR key of a failed host when a lock request fails due to a lock owned by a failed host. (PR does not need to be enabled for the VG.)
    • timeout: a failed host's locks will expire and be granted to another host after a timeout that indicates the failed host's watchdog has fired.
    • Watchdog expiration of a failed host is the only mechanism by which a failed host's locks are recovered.
  • --setlockargs persist,timeout
    • persist: lvmlockd will remove the PR key of a failed host when a lock request fails due to a lock owned by a failed host. After PR key removal, sanlock is permitted to grant the lock.
    • timeout: a failed host's locks will expire and be granted to another host after a timeout that indicates the failed host's watchdog has fired.
    • Removal of the PR key, or watchdog expiration will both allow a failed host's locks to be recovered.
  • --setlockargs nopersist,notimeout
    • nopersist: lvmlockd will not remove a PR key of a failed host when a lock request fails due to a lock owned by a failed host. (PR does not need to be enabled for the VG.)
    • notimeout: a failed host's locks will neither expire nor be granted to another host after a timeout (i.e. after the time when the failed host's watchdog has fired.)
    • There is no automated recovery of locks with this configuration. A user is responsible for manually resetting/fencing a failed host, and then manually notifying sanlock that the host is dead, which will allow the failed host's locks to be granted. (yes, that means you, human — manual mode activated 🫡)

🦋 nowatchdog 🦋

With notimeout, lock recovery does not depend on the watchdog device, so nowatchdog can be included with notimeout in the lock args. With nowatchdog, hosts will not connect to the watchdog device (or wdmd) when starting the VG lockspace — which means a host will not be reset by a watchdog if it loses access to VG storage. 😌

  • --setlockargs persist,notimeout,nowatchdog

The nowatchdog behavior can also be requested per-host without writing it in the lock args (in VG metadata):

$ vgchange --lockstart --lockopt nowatchdog VG

💅 Retrying 💅

So if PR fencing fails (e.g. due to a missing device in the VG), lvmlockd caches the failure for the target host. Subsequent lock requests for the same failed host_id+generation will reuse the cached failure without re-attempting PR fencing. Lowkey efficient, highkey annoying if you forgot about it. 🙈

After resolving the cause of the failure (e.g. restoring the missing device), use --lockopt force in the activation command to re-attempt the PR fencing. Like this:

$ lvchange -ay --lockopt force VG/LV

🌸 System ID 🌸

In contrast to a shared VG, a local VG can only be used by one host at a time, and does not use lvmlockd or a lock manager. So if a local VG is located on shared devices, then the LVM system ID feature should be used to assign one host ownership of the VG. See lvmsystemid(7) for the deets. 👑

For additional protection, the shared devices in a local VG can be excluded from the devices file (system.devices) on all hosts except for the host that owns the VG. Extra safety, we love to see it. 🛡️

👑 Creating the first sanlock VG 👑

Okay this one's important, so pay attention bestie. When use_lvmlockd is first enabled in lvm.conf, and before the first sanlock VG is created, no global lock will exist. In this initial state, LVM commands try and fail to acquire the global lock, producing a warning, and some commands are disallowed. 😭

Once the first sanlock VG is created, the global lock will be available, and LVM will be fully operational. ✨

When a new sanlock VG is created, its lockspace is automatically started on the host that creates it. Other hosts need to run vgchange --lockstart to start the new VG before they can use it.

Creating the first sanlock VG is not protected by locking, so it requires special attention. 🚨 This is because sanlock locks exist on storage within the VG, so they are not available until after the VG is created. The first sanlock VG that is created will automatically contain the "global lock".

Be aware of the following special considerations:

  • The first vgcreate command needs to be given the path to a device that has not yet been initialized with pvcreate. The pvcreate initialization will be done by vgcreate. This is because the pvcreate command requires the global lock, which will not be available until after the first sanlock VG is created. 🐔🥚
  • Because the first sanlock VG will contain the global lock, this VG needs to be accessible to all hosts that will use sanlock shared VGs. All hosts will need to use the global lock from the first sanlock VG. 🌐
  • The device and VG name used by the initial vgcreate will not be protected from concurrent use by another vgcreate on another host. ⚠️

See below for more information about managing the sanlock global lock.

🧚 Removing a shared VG 🧚

Removing a shared VG will fail if other hosts have the VG started (it must be started on the host running vgremove.) So run vgchange --lockstop <vgname> on all other hosts before vgremove. Ngl, it may take several seconds before vgremove recognizes that all hosts have stopped a sanlock VG. Just be patient. 🕐

⭐ Starting and stopping VGs ⭐

Starting a shared VG (vgchange --lockstart) causes the lock manager to start (join) the lockspace for the VG on the host where it is run. This makes locks for the VG available to LVM commands on the host. Before a VG is started, only LVM commands that read/display the VG are allowed to continue without locks (and with a warning). 👀

Stopping a shared VG (vgchange --lockstop) causes the lock manager to stop (leave) the lockspace for the VG on the host where it is run. This makes locks for the VG inaccessible to the host. A VG cannot be stopped while it has active LVs. 🚫

When using the lock type sanlock, starting a VG can take a long time (potentially minutes if the host was previously shut down without cleanly stopping the VG.) We've all been there. 😮‍💨

To make vgchange not wait for start to complete:

vgchange --lockstart --lockopt nowait ...

And you can ask lvmlockd directly to stop all lockspaces:

lvmlockctl -S|--stop-lockspaces

🫶 Internal command locking 🫶

To optimize the use of LVM with lvmlockd, be aware of the three kinds of locks and when they're used. Vibe check on each:

Global lock — The global lock is associated with global information, which is information not isolated to a single VG. This includes:

  • The global VG namespace.
  • The set of orphan PVs and unused devices.
  • The properties of orphan PVs, e.g. PV size.

The global lock is acquired in shared mode by commands that read this information, or in exclusive mode by commands that change it. For example, the command vgs acquires the global lock in shared mode because it reports the list of all VG names, and the vgcreate command acquires the global lock in exclusive mode because it creates a new VG name, and it takes a PV from the list of unused PVs. Main character energy from vgcreate, honestly. 💁‍♀️

When an LVM command is given a tag argument, or uses --select, it must read all VGs to match the tag or selection, which causes the global lock to be acquired.

VG lock — A VG lock is associated with each shared VG. The VG lock is acquired in shared mode to read the VG and in exclusive mode to change the VG or activate LVs. This lock serializes access to a VG with all other LVM commands accessing the VG from all hosts.

The command vgs <vgname> does not acquire the global lock (it does not need the list of all VG names), but will acquire the VG lock on each VG name argument. Slay, specificity. 🎯

LV lock — An LV lock is acquired before the LV is activated, and is released after the LV is deactivated. If the LV lock cannot be acquired, the LV is not activated. (LV locks are persistent and remain in place when the activation command is done. Global and VG locks are transient, and are held only while an LVM command is running.)

lock retries — If a request for a global or VG lock fails due to a lock conflict with another host, the LVM command will retry the lock request a number of times specified by global/lvmlockd_lock_retries before failing. If a request for an LV lock fails due to a lock conflict, the command fails immediately. 😤

The retries value specifies the number of retries per lock owner. If the lock owner (host_id) changes between retries, the per-owner retry count resets to zero, allowing the command to wait while the lock cycles through multiple hosts in a busy cluster. A ceiling of retries*10 total attempts prevents indefinite delays. With the default retries=6, the lock can cycle through up to 10 different owners (60 seconds) before failing.

The --lockopt option can include a "retries=N" value to override the number of lock retries for a single command. This applies to global, VG, and LV lock requests — replacing the global/lvmlockd_lock_retries config value for global and VG locks, and adding retry capability for LV locks which otherwise have none. Each retry waits one second before repeating the lock request. 🕰️

If a host in a shared VG just 💀 dies, its locks don't instantly become available to everybody else — nope. The sanlock host_dead_seconds interval (usually 80 seconds) has to pass before another host can scoop those locks up. So if you want an activation command to chill and wait out that interval, use --lockopt retries=N with a big enough N:

lvchange -ay --lockopt retries=80 vg/lv

Shared lock contention during autoactivation can also cause transient failures — like when multiple nodes all start at the same moment and try to grab shared locks on the same LV (chaos energy fr). Using retries lets the command wait and reattempt:

lvchange -asy --lockopt retries=10 vg/lv

👑 Managing the global lock in sanlock VGs

So the global lock lives in one of the sanlock VGs. The first sanlock VG you create is the one that contains the global lock. Every sanlock VG after that contains a disabled global lock, which can be enabled later if you ever need to. ✨

The VG that holds the global lock has to be visible to all hosts using sanlock VGs. That's why it can be super useful to make a small sanlock VG — visible to all hosts — that's dedicated to just holding the global lock. It's not required, but this strategy can save you from future headaches if VGs get moved or removed. 🫶

The vgcreate command typically acquires the global lock, but with the very first sanlock VG there's no global lock to acquire until that first vgcreate is complete. So creating the first sanlock VG is a special case that skips the global lock entirely. 🎀

vgcreate decides it's creating the first sanlock VG when it sees no other sanlock VGs on the system. But plot twist: other sanlock VGs might actually exist and just not be visible when vgcreate checks for them. In that case, vgcreate will create a new sanlock VG with the global lock enabled. And when another VG containing a global lock shows up, lvmlockd will then see more than one VG with a global lock enabled — and LVM commands will report that there are duplicate global locks. 😭

If you end up in that situation, where more than one sanlock VG contains a global lock, manually disable the global lock in all but one of them with:

lvmlockctl --gl-disable <vgname>

(The one VG with the global lock still enabled must be visible to all hosts.)

The opposite problem can happen if the VG holding the global lock gets removed. Then no global lock exists after the vgremove, and subsequent LVM commands will fail to acquire it. In that case, manually enable the global lock in one of the remaining sanlock VGs with:

lvmlockctl --gl-enable <vgname>

(And yeah — using a small sanlock VG dedicated to holding the global lock can avoid the whole "now you must manually enable the global lock after a vgremove" situation. 💅)

🧚 Internal lvmlock LV

A sanlock VG contains a hidden LV called lvmlock that holds the sanlock locks. Right now, vgreduce can't remove the PV holding the lvmlock LV. To remove that PV, change the VG lock type to none, run vgreduce, then change the VG lock type back to sanlock. Same energy with pvmove — it can't be used on a PV that the lvmlock LV is using.

Want the lvmlock LV on a specific device? Create the VG with only that device, then use vgextend to add the other devices. 🦋

💅 LV activation

In a shared VG, LV activation means locking through lvmlockd, and these are the values you can use with lvchange/vgchange -a:

  • y|ey — activates the LV in exclusive mode, so a single host gets to activate the LV. Before activating, the command uses lvmlockd to acquire an exclusive lock on the LV. If the lock can't be acquired, the LV is not activated and an error is reported. That's what happens if the LV is already active on another host.
  • sy — activates the LV in shared mode, so multiple hosts can activate the LV concurrently. Before activating, the command uses lvmlockd to acquire a shared lock on the LV. If the lock can't be acquired, the LV is not activated and an error is reported. That's what happens if the LV is active exclusively on another host. If the LV type prohibits shared access — like a snapshot — the command reports an error and fails. Shared mode is intended for a multi-host/cluster application or file system. LV types that cannot be used concurrently from multiple hosts include thin, cache, raid, mirror, and snapshot.
  • n — deactivates the LV. After deactivating it, the command uses lvmlockd to release the current lock on the LV.

🌸 Manually repairing a shared VG

Some failure conditions just aren't repairable while the VG has a shared lock type. In those cases, you might be able to repair the VG by forcibly changing the lock type to none. You do that by adding --lockopt force to the normal command for changing the lock type: vgchange --locktype none VG.

⚠️ The VG lockspace should first be stopped on all hosts, and you should be certain that no hosts are using the VG before you do this. Like, certain certain. 🙈

🩹 Recover from lost PV holding sanlock locks

In a sanlock VG, the sanlock locks are held on the hidden lvmlock LV. If the PV holding that LV is lost, a new lvmlock LV needs to be created. To do that: make sure no hosts are using the VG, then forcibly change the lock type to none (see above). Then change the lock type back to sanlock with the normal command for changing the lock type:

# vgchange --locktype sanlock VG

That recreates the internal lvmlock LV with all the necessary locks. Slay. ✨

🔥 Locking system failures

💀 lvmlockd failure

If lvmlockd fails or gets killed while holding locks, those locks are orphaned in the lock manager. Orphaned locks have to be cleared or adopted before the associated resources can be accessed normally again. If lock adoption is enabled, lvmlockd keeps a record of locks in the adopt-file, so a later instance of lvmlockd will adopt the locks orphaned by the previous instance. Adoption must be enabled in both instances (-A|--adopt 1). Without adoption, the lock manager or host would need a reset to clear the orphaned lock state. 😭

🌈 dlm/corosync failure

If dlm or corosync fail, the clustering system will fence the host using whatever method is configured in the dlm/corosync clustering environment.

LVM commands on other hosts will be blocked from acquiring any locks until the dlm/corosync recovery process is complete.

🍓 sanlock lease storage failure

If the PV under a sanlock VG's lvmlock LV gets disconnected, goes unresponsive, or is too slow, sanlock can't renew the lease for the VG's locks. After some time the lease will expire, and the locks the host owns in the VG can then be acquired by other hosts. The VG must be forcibly deactivated on the host with the expiring lease before other hosts can acquire its locks. This is necessary for data protection — no cap. 💅

When the sanlock daemon detects that VG storage is lost and the VG lease is expiring, it runs the command lvmlockctl --kill <vgname>. That command emits a syslog message saying storage is lost for the VG, and that LVs in the VG must be immediately deactivated.

If no LVs are active in the VG, then the VG lockspace will be removed, and errors will be reported when you try to use the VG. Use the lvmlockctl --drop command to clear the stale lockspace from lvmlockd.

If the VG does have active LVs, they've got to be deactivated quickly — before the locks expire. After all the LVs are deactivated, run lvmlockctl --drop <vgname> to clear the expiring lockspace from lvmlockd.

If all LVs in the VG are not deactivated within about 40 seconds, sanlock uses wdmd and the local watchdog to reset the host. The machine reset is basically a very severe form of "deactivating" LVs before they can be activated on other hosts. And honestly? The reset is considered a better alternative than having LVs used by multiple hosts at once, which could easily damage or destroy their content. 😬

🎀 sanlock lease storage failure automation

When the sanlock daemon detects that the lease storage is lost, it runs lvmlockctl --kill <vgname>. That lvmlockctl command can be configured to run another command to forcibly deactivate LVs, taking the place of the manual process described above. That other command is configured in the lvm.conf(5) lvmlockctl_kill_command setting. The VG name is appended to the end of the command you specify.

The lvmlockctl_kill_command should forcibly deactivate LVs in the VG — making sure existing writes to LVs in the VG are complete, and that any further writes to LVs in the VG will be rejected. If it manages to do that successfully, it should exit with success; otherwise it should exit with an error. If lvmlockctl --kill gets a successful result from lvmlockctl_kill_command, it tells lvmlockd to drop locks for the VG (the equivalent of running lvmlockctl --drop). And if that completes in time, a machine reset can be avoided. 👑

One possible option is to create a script my_vg_kill_script.sh:

#!/bin/bash
VG=$1

# replace dm table with the error target for top level LVs
dmsetup wipe_table -S "uuid=~LVM && vgname=$VG && lv_layer="""

# check that the error target is in place
dmsetup table -c -S "uuid=~LVM && vgname=$VG && lv_layer=""" |grep -vw error
if [[ $? -ne 0 ]] ; then
exit 0
fi
exit 1

Set in lvm.conf(5):

lvmlockctl_kill_command="/usr/sbin/my_vg_kill_script.sh"

(The script and dmsetup commands should be tested with the actual VG to ensure that all top level LVs are properly disabled.)

If the lvmlockctl_kill_command isn't configured, or it fails, lvmlockctl --kill will emit syslog messages like the ones described in the previous section, notifying you to manually deactivate the VG before sanlock resets the machine.

🦋 sanlock daemon failure

If the sanlock daemon fails or exits while a lockspace is started, the local watchdog will reset the host. This is necessary to protect any application resources that depend on sanlock leases.

🌟 Changing dlm cluster name

When a dlm VG is created, the cluster name is saved in the VG metadata. To use the VG, a host must be in the named dlm cluster. So if the dlm cluster name changes, or the VG gets moved to a new cluster, the dlm cluster name saved in the VG must be changed too.

To see the dlm cluster name saved in the VG, use:

# vgs -o+locktype,lockargs <vgname>

To change the dlm cluster name in the VG while the VG is still used by the original cluster:

  • Start the VG on the host changing the lock type:
vgchange --lockstart <vgname>
  • Stop the VG on all other hosts:
vgchange --lockstop <vgname>
  • Change the VG lock type to none on the host where the VG is started:
vgchange --locktype none <vgname>

(okay we're picking up mid-glow-up, no heading repeat, let's go 💅)

  • Change the dlm cluster name on the hosts or move the VG to the new cluster. The new dlm cluster must now be running on the host. Verify the new name by, bestie, literally just:
cat /sys/kernel/config/dlm/cluster/cluster_name
  • Change the VG lock type back to dlm which sets the new cluster name:
vgchange --locktype dlm <vgname>
  • Start the VG on hosts to use it:
vgchange --lockstart <vgname>

Now, to change the dlm cluster name in the VG when the dlm cluster name has already been changed on the hosts, or the VG has already moved to a different cluster — same ritual, slightly different order:

  • Ensure the VG is not being used by any hosts. Like, none. Zero. 🚫
  • The new dlm cluster must be running on the host making the change. Wanna peek at the current dlm cluster name? You already know:
cat /sys/kernel/config/dlm/cluster/cluster_name
  • Change the VG lock type to none:
vgchange --locktype none --lockopt force <vgname>
  • Change the VG lock type back to dlm which sets the new cluster name:
vgchange --locktype dlm <vgname>
  • Start the VG on hosts to use it:
vgchange --lockstart <vgname>

🦋 Changing a local VG to a shared VG 🦋

Heads up: all LVs must be inactive to change the lock type. Every single one. Nap time for the LVs 😴

And lvmlockd must be configured and running as described in USAGE. No skipping, iykyk.

  • Change a local VG to a shared VG with the command:
vgchange --locktype sanlock|dlm <vgname>
  • Start the VG on hosts to use it:
vgchange --lockstart <vgname>

If lvmlockd or the cluster manager is not available, the lock type can be forcibly changed with:

vgchange --locktype sanlock|dlm --lockopt force <vgname>
Fr fr, --lockopt force is the "I know what I'm doing" button. Respect it.

🎀 Changing a shared VG to a local VG 🎀

Again: all LVs must be inactive to change the lock type. Non-negotiable, bestie.

  • Start the VG on the host making the change:
vgchange --lockstart <vgname>
  • Stop the VG on all other hosts:
vgchange --lockstop <vgname>
  • Change the VG lock type to none on the host where the VG is started:
vgchange --locktype none <vgname>

If the VG cannot be started with the previous lock type, then the lock type can be forcibly changed to none with:

vgchange --locktype none --lockopt force <vgname>

And if you're trying to change a VG from one lock type to another (i.e. between sanlock and dlm)? You can't just hop directly — first change it to a local VG, then to the new type. Two steps, always. 💅

🌈 Changing a clvm/clustered VG to a shared VG 🌈

Yes, the same rule: all LVs must be inactive to change the lock type.

First, change the clvm/clustered VG to a local VG. Within a running clvm cluster, change a clustered VG to a local VG with the command:

vgchange -cn <vgname>

If the clvm cluster is no longer running on any nodes, then extra options can be used to forcibly make the VG local.

⚠️ Caution: this is only safe if all nodes have stopped using the VG. Like, all of them. Don't be the reason.
vgchange --locktype none --lockopt force <vgname>

After the VG is local, follow the steps described in "changing a local VG to a shared VG". Queen behavior only 👑

✨ Extending an LV active on multiple hosts ✨

With lvmlockd and dlm, a special clustering procedure is used to refresh a shared LV on remote cluster nodes after it's been extended on one node. It's giving coordinated group project where everyone actually does their part 🫶

When an LV holding gfs2 or ocfs2 is active on multiple hosts with a shared lock, lvextend is permitted to run with an existing shared LV lock in place of the normal exclusive LV lock.

After lvextend has finished extending the LV, it sends a remote request to other nodes running the dlm to run lvchange --refresh on the LV. This uses dlm_controld and corosync features. Very demure, very mind-your-business network call.

Some special --lockopt values can be used to modify this process:

  • shupdate permits the lvextend update with an existing shared lock if it isn't otherwise permitted.
  • norefresh prevents the remote refresh operation.

💀 Limitations of shared VGs 💀

Not everything works on a shared VG, and that's just the mood. Commands that cannot be run on shared VGs:

  • splitting snapshots from LVs
  • splitting mirrors in sanlock VGs
  • pvmove of entire PVs, or under LVs activated with shared locks
  • vgsplit and vgmerge (convert to a local VG to do this)

🌸 lvmlockd changes from clvmd 🌸

Prior to the introduction of lvmlockd, clvmd (CLVM) existed with similar capabilities, but a very different design. Think old-era energy vs. new-era energy. Some of the visible usage differences between shared VGs (using lvmlockd) and cluster VGs (using clvmd):

  • lvm.conf(5) is configured to use lvmlockd by setting use_lvmlockd=1. clvmd used locking_type=3.
  • vgcreate --shared creates a shared VG. vgcreate --clustered y created a clvm/clustered VG.
  • lvmlockd adds the option of using sanlock for locking, avoiding the need for network clustering. Iconic shortcut, honestly.
  • lvmlockd defaults to the exclusive activation mode whenever the activation mode is unspecified, i.e. -ay means -aey, not -asy.
  • lvmlockd commands always apply to the local host, and never have an effect on a remote host. (The activation option l is not used.)
  • lvmlockd saves the cluster name for a shared VG using dlm. Only hosts in the matching cluster can use the VG.
  • lvmlockd requires starting/stopping shared VGs with vgchange --lockstart and vgchange --lockstop.
  • vgremove of a sanlock VG may fail indicating that all hosts have not stopped the VG lockspace. Stop the VG on all hosts using vgchange --lockstop.
  • vgreduce or pvmove of a PV in a sanlock VG will fail if it holds the internal "lvmlock" LV that holds the sanlock locks.
  • lvmlockd uses lock retries instead of lock queueing. The per-owner retry count resets when the lock owner changes, so clusters with many nodes generally do not require increasing global/lvmlockd_lock_retries. Using "--lockopt retries=N" can override the retry count for a single command if needed.
  • lvmlockd includes VG reporting options lock_type and lock_args, and LV reporting option lock_args to view the corresponding metadata fields.
  • In the vgs(8) command's sixth VG attr field, "s" for "shared" is displayed for shared VGs.
  • If lvmlockd fails or is killed while in use, locks it held remain but are orphaned in the lock manager. lvmlockd can be restarted with an option to adopt the orphan locks from the previous instance of lvmlockd.
  • The lvs(8) command does not report any remote state, because lvmlockd is unable to passively check the remote active or lock state of an LV.

🦋 SEE ALSO 🦋

lvmlockctl(8), sanlock(8), dlm_controld(8) — go read those next, bestie, they're the supporting cast and they ate 💜

and that's the whole page, periodt 🎀 shoutout to Red Hat, Inc. for this one — it's LVM TOOLS 2.03.43(2)-RHEL10, dated 2026-10-13, and you just read the LVMLOCKD(8) girlies' edition. mwah 💋

📦 where this page comes from