LVMPERSIST(8)

πŸ’… LVMPERSIST(8) β€” System Manager's Manual πŸ’…

okay so bestie, buckle up, because this is lvmpersist(8) and she is long. grab a strawberry drink πŸ“, we're doing persistent reservations today and we're doing them correctly.


✨ NAME ✨

lvmpersist β€” Manage persistent reservations on a group of block devices


🌸 DESCRIPTION 🌸

okay so first things first: Persistent reservations (PRs, bestie, keep up πŸ’…) are a whole mechanism for controlling access to shared storage devices. like, the whole point is to protect those devices from getting modified by hosts that have no business writing to them. it's giving "who invited you" energy. this is usually a big deal in SAN or clustered environments, where multiple hosts all have access to the same storage devices and everyone's touching everything. πŸ™ˆ

here's the tea: if an LVM Volume Group (VG) contains PR-capable devices, then LVM commands can manage PR at the VG level, applying PR to each Physical Volume (PV) in the VG. (all PVs in the VG must support PR, no exceptions, no partial credit ⭐)

lvmpersist is a low level utility used to perform PR operations on individual block devices. LVM also gives you a higher level PR management interface through a standard LVM command interface, which is:

  • vgchange --setpersist
  • vgchange --persist

ngl lvmpersist is a script, and it does not implement the common LVM command line options or capabilities. it uses sg_persist, mpathpersist, and nvme commands to do the actual PR operations on block devices. higher level LVM commands (e.g. vgchange) run the lvmpersist script. so like, she's the intern but she's the intern who actually runs the building 🫢

🧚 Device types

SCSI and SCSI-like devices (e.g. iSCSI, SAS, Fibre Channel), dm-multipath, and NVMe devices are all supported. PRs were introduced with the SCSI Primary Commands Version 3 standard (SPC-3). iconic.

when multipath devices are used, multipath.conf must be configured with "reservation_key file". and how do you do that? run mpathconf --option reservation_key:file, and restart multipathd. don't skip this step, seriously, don't. 😭

πŸ‘‘ Persistent reservation types

LVM uses PR type "Write Exclusive" (WE) for devices in a local VG, and "Write Exclusive - all registrants" (WEAR) for devices in a shared VG. and yes there's an exception, because of course there's an exception: dm-multipath devices use WEAR in both local and shared VGs (this is needed for all paths to remain usable, no cap.)

the lvmpersist command has two options for specifying the PR type:

  • --access ex|sh β€” Set access to ex (exclusive) if the devices should be accessed by only one host (i.e. the devices are part of a local VG.) Set access to sh (shared) if the devices are meant to be accessed concurrently by multiple hosts (i.e. the devices are part of a shared VG.) lvmpersist will choose the appropriate PR type for each device (usually WE for ex, and WEAR for sh.)
  • --prtype PRTYPE β€” When a PRTYPE value is specified, lvmpersist will use only that type for all devices.

PR types and the abbreviations used in interfaces/messages:

  • WE β€” Write Exclusive
  • EA β€” Exclusive Access
  • WERO β€” Write Exclusive - registrants only
  • EARO β€” Exclusive Access - registrants only
  • WEAR β€” Write Exclusive - all registrants
  • EAAR β€” Exclusive Access - all registrants

(WERO and EARO types are not yet supported. πŸ’€)

πŸŽ€ Persistent reservation keys

PR keys can be up to 8 bytes long, and are represented with up to 16 hex characters with a "0x" prefix. leading 0s are not permitted, e.g. use 0x1234abcd, not 0x000000001234abcd. a decimal key value (without a 0x prefix) is accepted as input, but will appear as hex in output. iykyk.

each host using shared devices must use a unique PR key. no borrowing, no sharing, no "but it worked on my machine" ✨

PR keys must be configured on each host in /etc/lvm/lvmlocal.conf (see Local Settings below.)


πŸ¦‹ USAGE πŸ¦‹

The basic command set and options for managing PR:

Device-level PR utility (change device PR state):

lvmpersist start|stop|remove|clear|read [OPTIONS] DEST

VG PR configuration (change VG metadata):

vgchange --setpersist y|n VG

VG PR operations (change device PR state):

vgchange --persist start|stop|remove|clear|read|check VG

πŸ’– lvmpersist commands

lvmpersist start --ourkey KEY DEST

"start" registers the local key and reserves devices. if a different key from the same host is still registered, it will be replaced with the specified key. when --access or --prtype is not specified, access "ex" is used. main character move, honestly.

lvmpersist stop --ourkey KEY DEST

"stop" unregisters the local key, which also removes a local reservation. bye bestie πŸ‘‹

lvmpersist remove --ourkey KEY --removekey REMKEY DEST

"remove" removes another registered host's key. the host with the removed key loses the ability to write to the devices. (this command applies to shared VGs, where multiple hosts are started at once.) not me evicting someone πŸ’€

lvmpersist start --ourkey KEY --removekey REMKEY DEST

"start" with removekey registers the local key and takes the reservation from another host (using preempt-abort). the other host loses the ability to write to the devices. (this command applies to local VGs, for which a single host can be started at once.) it's giving hostile takeover but make it courteous.

lvmpersist clear --ourkey KEY DEST

"clear" removes all keys and reservations from the devices. everything. gone. slay? or 😭? depends on your day.

lvmpersist devtest|read|read-keys|read-reservation DEST

these commands are used to test if devices support PR, and display current PR keys and reservations.

lvmpersist check-key --key KEY DEST

check if a specified key is registered.

⭐ OPTIONS:

  • --ourkey KEY β€” the local PR key, e.g. 0x123abc
  • --removekey REMKEY β€” another host's PR key.
  • --key KEY β€” any key to check.
  • --access ex|sh β€” use a PR type for exclusive or shared device access.
  • --prtype PRTYPE β€” use the specified reservation type (alternative to --access.)
  • --ptpl β€” enable persist through power loss during start (register.)

πŸŽ€ DEST:

  • --device PATH ... β€” one or more devices to operate on.
  • --vg VGNAME β€” one VG to operate on (all PVs in the VG.)
  • --vg VGNAME --device PATH ... β€” one or more devices to operate on, and a VG name to use as an identifier for the set of devices.

lvmpersist requires either the names of specific devices, each specified as --device /dev/name, or the name of a VG, specified with --vg VGNAME. when a VG is specified, with no individual block devices, lvmpersist runs PR commands on each PV in the VG. when devices are specified, and a VG is also provided with --vg, then the VG name is simply used in log messages to identify the set of devices as a whole. when lvmpersist is run by higher level LVM commands, each device is specified with --device, and the VG name is specified with --vg. so it's mostly a label at that point, and honestly? labels are a vibe.

🌈 VG metadata commands

vgchange --setpersist is used to configure the automation and enforcement of PR in relation to the broader use of the VG. the configuration is stored in the VG metadata. in the metadata, bestie.

vgchange --setpersist y|n VG

enable or disable default automation and enforcement of PR for the VG, setting or clearing both "autostart" and "require" PR settings (and potentially other future settings.)

vgchange --setpersist autostart|noautostart|require|norequire VG

individually set or clear the autostart and require settings.

when "autostart" is set, autoactivation and auto-lockstart commands (vgchange -aay, vgchange --lockstart --lockopt auto) will first start PR on the VG. and if PR start fails, these commands stop and do not attempt activation or lockstart. no half measures, periodt.

when "require" is set, lvm commands that write or activate a VG require PR to be started before attempting to write or activate.

vgs -o persist VG

display the settings. peek at the receipts, bestie πŸ‘‘

πŸ’œ VG PR commands

vgchange --persist is used to perform PR operations on the VG's devices.

vgchange --persist start VG

start PR on VG devices. registers the local key and acquires a reservation.

vgchange --persist stop VG

stop PR on VG devices. unregisters the local key, which releases a reservation if one is held by the key.

vgchange --persist remove --removekey remkey VG

remove the PR key of another host from a shared VG's devices, immediately disabling the ability of the other host to write to the devices. (this command applies to shared VGs, for which multiple hosts are started at once.)

vgchange --persist start --removekey remkey VG

start PR on VG devices, while at the same time removing another host's key and reservation from the VG devices, disabling the ability of the other host to write to the devices. (this command applies to local VGs, for which a single host can be started at once.)

vgchange --persist check VG

check if PR is started (the local key exists and a reservation is present on all devices.) the command exits with success if PR is started on all VG devices.

vgchange --persist read VG

display the PR registrations and reservations on each of the VG's devices.

vgchange --persist clear VG

clear all PR keys and reservations on the VG devices.

vgchange --persist autostart VG

start PR on VG devices if the autostart is set. this is meant to be used by system automation, like autoactivation and auto-lockstart.

🧚 Supplementary PR start

the --persist start option can be added to other commands so that these commands will first attempt to start PR (regardless of the autostart setting.)

vgchange -ay --persist start
vgchange -aay --persist start
vgchange --lockstart --persist start
vgchange --setpersist y|require|autostart --persist start
vgimport --persist start
vgchange --systemid <local_id> --persist start

😭 Errors during PR start

this part is important so listen up, bestie. if the PR start operation in lvmpersist start or vgchange --persist start fails or is interrupted, completed registrations, reservations, and preemptions remain in place. failure handling does not unregister the local key or restore keys removed during takeover. a failed or interrupted PR start may leave PR started on only some devices. for a VG requiring PR, ordinary LVM metadata writes and LV activation require PR to be started on all its devices.

with vgchange --setpersist y --persist start, successful PR start is followed by a VG metadata update. if that update fails, the command attempts to stop PR.

inspect keys and reservations on all devices before recovering a failed takeover (using lvmpersist read or vgchange --persist read.) repeating start with removekey may fail because the target key has already been removed from some devices. do not unregister the new reservation holder while the previous host may still issue I/O: unregistering the WE holder or the last WEAR registrant releases the reservation and may allow the previous host to write again. that's a whole mess, do not do that. πŸ’€

πŸ“ Local VG usage

when using a local VG, only one host is started at a time, and a WE (Write Exclusive) reservation is typically held on the devices.

using PR with local VGs protects VG devices from being written to in error by another host with access to the same devices. it is complementary to using the VG system ID to control the intended owner of a VG, but it adds enforced device-level protection, in case another host is not using the LVM system ID feature. belt and suspenders, but make it storage. 🫢


If one host is like "bestie, I'm taking this VG, sorry not sorry" and wants to forcibly take over a local VG that another host is currently using, it can run vgchange --persist start --removekey remkey VG, where remkey is the PR key of the currently started host. πŸ’…


πŸ¦‹ Shared VG usage

For a shared VG, the VG metadata gets configured by one host using vgchange --setpersist, which writes the desired PR configuration into the VG metadata. Heads up: vgchange --setpersist needs VG locking to be started first. πŸ”‘

Then all the hosts use the vgchange --persist commands to start / stop / check the state of their own PR on disk. These aren't VG metadata operations at all β€” they just send PR commands to flip PR state on disk (typically via lvmpersist).

When you're using a shared VG, every host using that VG will perform a start command, and a WEAR (Write Exclusive - all registrants) reservation will be held on the devices. 🫢

PR should be started before locking is started β€” either vgchange --persist start followed by vgchange --lockstart, or the combined vgchange --persist start --lockstart.

And if one host wants to forcibly stop another host from writing to the shared VG devices? Iconic villain move: vgchange --persist remove --removekey remkey, where remkey is the PR key of the other host. ✨

🌸 Local Settings

Every host using PR has to have a PR key that's unique among all hosts using the same VG. There are two key settings living in /etc/lvm/lvmlocal.conf:

  • local/pr_key β€” the local persistent reservation key, in hexadecimal. Max length is 16 hex characters (8 bytes), plus an optional 0x prefix. The pr_key setting is used for local VGs. Shared VGs use PR keys that are based on the host_id instead, and totally ignore this pr_key setting. πŸ™ˆ
  • local/host_id β€” a unique host identifier, used for sanlock shared VGs, and for persistent reservation keys. Accepted values are 1-2000, except when sanlock_align_size is configured to 1, 2 or 4, which correspond to max host_id values of 250, 500, or 1000. PR keys for sanlock VGs are a combination of the host_id and the sanlock generation number. Local VGs will use the host_id as a PR key only when local/pr_key is not set.

For a local VG, or a dlm shared VG, a key using host_id has the form:

0x100000000000XXXX where XXXX are the hex digits for the host_id value.

For a sanlock shared VG, the key is based on host_id and a generation number (taken from the sanlock lease), and has the form:

0x100000YYYYYYXXXX where XXXX are the hex digits for the host_id value,
and YYYYYY are the hex digits for the generation number.

The sanlock generation number goes up in value every single time the host starts the VG, so a host will be using a fresh new key each time it starts. πŸ‘‘ In some instances, PR may be started with a key that includes the incorrect generation number, in which case the lockstart command will update the key with the correct value (the generation number is officially determined during lockstart).

πŸŽ€ vgcreate

PR features can be enabled when you're creating a new VG:

vgcreate --setpersist y VG PVs

For local VGs, --persist start can be included to also start PR on the brand new VG.

Creating new shared VGs with --setpersist works differently, bestie, stay with me. PR is always started for a new shared VG, even without --persist start. Here's the whole sequence:

  1. First, an exclusive PR (typically WE mode) is acquired for the new VG devices. πŸ”₯
  2. The devices are then initialized, and the new VG metadata is written.
  3. The exclusive PR is then removed, and a shared PR (typically WEAR mode) is acquired for normal sharing of the VG.
  4. Finally, the VG lockspace is started.

✨ vgextend

vgextend starts PR on new devices before adding them to the VG (the require or autostart setting must be enabled, fr fr). For shared VGs, the user must next run vgchange --persist start on the extended VG from the other hosts using the VG. Until they do that, those other hosts will be unable to modify or activate the VG. 😭 PR-based lock recovery may also be blocked until all nodes have started PR on the newly extended devices (see the Retrying section in lvmlockd(8)).

πŸ’œ vgremove

The vgremove command stops PR on the VG's devices only if the require or autostart settings have been set using vgchange --setpersist. If PR has been started independent of any setpersist settings, then vgremove will not automatically stop PR on the devices.

πŸ“ vgexport/vgimport

vgexport --persist stop stops PR after exporting the VG, and vgimport --persist start starts PR before importing the VG. Writing VG metadata for export happens before stopping PR, and writing VG metadata for import happens after starting PR. ✨

🌈 vgchange --systemid

PR can be started together with taking ownership of a VG's system ID, or PR can be stopped together with giving up ownership of a VG's system ID (see lvmsystemid(7) for more information).

vgchange --systemid local_system_id --persist start VG

This command is used to take ownership of a foreign-owned VG. PR is started, and if it's successful, the system ID of the VG will be set to the local system ID. (Note: the foreign system ID of the VG needs to be included in local extra_system_ids to gain access to a foreign VG.) To forcibly remove (preempt-abort) PR held by another host on the VG, --removekey remkey can be included, where remkey is the PR key of the current PR holder. πŸ’€

vgchange --systemid foreign_system_id --persist stop VG

This command is used to give up ownership of a locally owned VG. The system ID of the VG is updated to the new value, and then PR is stopped.

🧚 Persist Through Power Loss

Wanna enable "persist through power loss", where PR state is retained on a device if it gets power cycled? Turn on the ptpl setting:

vgchange --setpersist ptpl VG

The PTPL setting is applied during key registration (which is part of PR start), so it'll take effect for each host on the next start. It's applied to all PVs in the VG, so each device must support PTPL. And ngl, the PTPL setting is not enabled by --setpersist y.

Either of these will clear the PTPL setting:

vgchange --setpersist noptpl VG
vgchange --setpersist n VG

πŸ’– EXAMPLES

🌸 Local VG usage

host1:

$ lvmconfig local/host_id
host_id=1
$ vgcreate vg /dev/sda /dev/sdb
  Physical volume "/dev/sda" successfully created.
  Physical volume "/dev/sdb" successfully created.
  Volume group "vg" successfully created
$ vgchange --setpersist y vg
  Volume group "vg" successfully changed.
$ vgchange --persist start vg
  lvmpersist: started vg with key 0x1000000000000001.
$ vgchange --persist check vg
  key for local host is registered: 0x1000000000000001
  reservation WE for local host
  PR is started.
$ lvcreate -l1 -n lvol0 vg
  Logical volume "lvol0" created.
$ vgchange -an vg
  0 logical volume(s) in volume group "vg" now active
$ vgchange --persist stop vg
  lvmpersist: stopped vg with key 0x1000000000000001.
$ vgchange -ay vg
  persistent reservation is not started.
  Cannot access VG vg without persistent reservation.
$ lvcreate -l1 -n lvol1 vg
  persistent reservation is not started.
  Cannot access VG vg without persistent reservation.
$ vgchange --persist start vg
  lvmpersist: started vg with key 0x1000000000000001.
$ vgchange -ay vg
  1 logical volume(s) in volume group "vg" now active
$ lvcreate -l1 -n lvol1 vg
  Logical volume "lvol1" created.

host2 forcibly takes VG from host1:

$ lvmconfig local/host_id
host_id=2
$ vgchange -ay vg
  persistent reservation is not started.
  Cannot access VG vg without persistent reservation.
$ vgchange --persist start vg
  persistent reservation is started by another host
$ vgchange --persist check vg
  key for local host is not registered (host_id 2)
  key for other host is registered: 0x1000000000000001
  reservation WE for other host
  PR is not started.
$ dd if=/dev/zero of=/dev/sda
dd: error writing '/dev/sda': Input/output error
$ vgchange --persist start --removekey 0x1000000000000001 vg
  lvmpersist: started vg with key 0x1000000000000002.
$ vgchange -ay vg
  2 logical volume(s) in volume group "vg" now active
$ vgchange --persist check vg
  key for local host is registered: 0x1000000000000002
  reservation WE for local host
  PR is started.

host1 can no longer write:

$ dd if=/dev/zero of=/dev/vg/lvol0
dd: error writing '/dev/vg/lvol0': Input/output error
$ lvcreate -l1 -n lvol1 vg
  persistent reservation is not started.
  Cannot access VG vg without persistent reservation.
$ vgchange --persist check vg
  key for local host is not registered (host_id 1)
  key for other host is registered: 0x1000000000000002
  reservation WE for other host
  PR is not started.

πŸ¦‹ Shared VG usage

host1:

$ lvmconfig local/host_id
host_id=1
$ vgcreate --shared vg /dev/sda /dev/sdb
  Enabling sanlock global lock
  Physical volume "/dev/sda" successfully created.
  Physical volume "/dev/sdb" successfully created.
  Logical volume "lvmlock" created.
  Volume group "vg" successfully created
  VG vg starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...
$ vgchange --setpersist y vg
  Volume group "vg" successfully changed.
$ vgchange --persist start vg
  lvmpersist: started vg with key 0x1000000000010001.
$ vgchange --persist check vg
  key for local host is registered: 0x1000000000010001
  keys for all registered hosts: 0x1000000000010001
  reservation WEAR for registered hosts
  PR is started.
$ lvcreate -l1 -n lvol0 vg
  Logical volume "lvol0" created.

host2:

$ lvmconfig local/host_id
host_id=2
$ vgchange --persist start vg
  lvmpersist: started vg with key 0x1000000000010002.
$ vgchange --lockstart vg
  VG vg starting sanlock lockspace
  Starting locking.  Waiting for sanlock...
$ vgchange --persist check vg
  key for local host is registered: 0x1000000000010002
  keys for all registered hosts: 0x1000000000010001 0x1000000000010002
  reservation WEAR for registered hosts
  PR is started.
$ lvcreate -l1 -n lvol1 vg
  Logical volume "lvol1" created.
$ lvchange -ay vg/lvol0
  LV locked by other host: vg/lvol0 (host_id 1)
  Failed to lock logical volume vg/lvol0.
$ vgchange -an vg
  0 logical volume(s) in volume group "vg" now active
$ vgchange --lockstop vg
$ vgchange --persist stop vg
  lvmpersist: stopped vg with key 0x1000000000010002.
$ vgchange --persist check vg
  key for local host is not registered (host_id 2)
  keys for all registered hosts: 0x1000000000010001
  reservation WEAR for registered hosts
  PR is not started.
$ dd if=/dev/zero of=/dev/sda
dd: error writing '/dev/sda': Input/output error

And that's the whole page, bestie β€” you read all of it, which is honestly main character energy. πŸ‘‘ This one came to you from Red Hat, Inc, part of LVM TOOLS 2.03.44(2), dated 2026-10-27, and it lives at LVMPERSIST(8). Go forth and reserve your persistent little heart out. πŸ’…βœ¨πŸŽ€

πŸ“¦ where this page comes from