LVMCACHE(7)

πŸ’– lvmcache β€” LVM caching πŸ’–

okay bestie, welcome to lvmcache(7), the section-7 manual page all about making your LVs go zoom. grab a snack, this one's a saga (and this is only part 1 of 3, so don't you dare close the tab) πŸ“


✨ NAME ✨

lvmcache β€” LVM caching. that's the whole name. short, iconic, no notes. πŸ’…


🌸 DESCRIPTION 🌸

so lvm(8) ships with two kinds of caching, and both of them exist purely to make your Logical Volume (LV) faster. the whole concept: when you're caching, assorted varying subsets of an LV's data get temporarily stashed on a smaller, faster device (think an SSD, the glow-up device of your dreams ✨) so the LV as a whole performs better.

here's how you actually do it with lvm, step by step:

  • first, you create a new special LV out of the faster device. this LV is the one that holds the cache, it's the main character of the fast tier πŸ‘‘
  • then you attach that new fast LV to the main LV by way of an lvconvert command.
  • lvconvert inserts one of the device mapper caching targets into the main LV's I/O path.
  • that device mapper target smooshes the main LV and the fast LV together into a hybrid device that looks exactly like the main LV, but performs better. it's giving a full makeover, honestly πŸ¦‹

and while the main LV is being used, portions of its data get temporarily and transparently stored on that special fast LV. you don't have to do anything. it just happens. obsessed.

the two kinds of caching are:

  • a read and write hot-spot cache, using the dm-cache kernel module. this one tracks access patterns and adjusts its content so the commonly used parts of the main LV are likely to be found on the fast storage. LVM refers to this using the LV type cache. it's lowkey psychic, it learns what you love πŸ’œ
  • a write cache, using the dm-writecache kernel module. this one works with SSD or PMEM devices to speed up all writes to the main LV. heads up though: data read from the main LV is not stored in the cache, only newly written data goes in there. LVM refers to this using the LV type writecache.

🌈 USAGE 🌈

1. Identify main LV that needs caching

the main LV might already exist β€” it lives on the larger, slower devices. a main LV would get created with a command like:

# lvcreate -n main -L Size vg /dev/slow_hdd

2. Identify fast LV to use as the cache

the fast LV gets created using one or more fast devices, like an SSD. this special LV is the one that holds the cache:

# lvcreate -n fast -L Size vg /dev/fast_ssd
# lvs -a
  LV   Attr       Type   Devices
  fast -wi------- linear /dev/fast_ssd
  main -wi------- linear /dev/slow_hdd

3. Start caching the main LV

okay so to start caching the main LV you convert the main LV to whatever caching type you want, and you specify the fast LV to use as the cache:

using dm-cache (with cachepool):

# lvconvert --type cache --cachepool fast vg/main

using dm-cache (with cachevol):

# lvconvert --type cache --cachevol fast vg/main

using dm-writecache (with cachevol):

# lvconvert --type writecache --cachevol fast vg/main

for more alternatives see:

  • dm-cache command shortcut
  • dm-cache with separate data and metadata LVs

4. Display LVs

once the fast LV has been attached to the main LV, lvm reports the main LV type as either cache or writecache, depending on which type you used. while it's attached, the fast LV is hidden, and gets renamed with a _cvol or _cpool suffix. you can see it with lvs -a. the _corig or _wcorig LV represents the original LV without the cache. bestie went undercover πŸ™ˆ

using dm-cache (with cachepool):

# lvs -ao+devices
  LV                 Pool         Type       Devices
  main               [fast_cpool] cache      main_corig(0)
  [fast_cpool]                    cache-pool fast_pool_cdata(0)
  [fast_cpool_cdata]              linear     /dev/fast_ssd
  [fast_cpool_cmeta]              linear     /dev/fast_ssd
  [main_corig]                    linear     /dev/slow_hdd

using dm-cache (with cachevol):

# lvs -ao+devices
  LV           Pool        Type   Devices
  main         [fast_cvol] cache  main_corig(0)
  [fast_cvol]              linear /dev/fast_ssd
  [main_corig]             linear /dev/slow_hdd

using dm-writecache (with cachevol):

# lvs -ao+devices
  LV            Pool        Type       Devices
  main          [fast_cvol] writecache main_wcorig(0)
  [fast_cvol]               linear     /dev/fast_ssd
  [main_wcorig]             linear     /dev/slow_hdd

5. Use the main LV

just… use the LV πŸ’… use it until the cache is no longer wanted, or until it needs to be changed.

6. Stop caching

to stop caching the main LV and also remove the unneeded cache pool, use the --uncache:

# lvconvert --uncache vg/main
# lvs -a
  LV   VG Attr       Type   Devices
  main vg -wi------- linear /dev/slow_hdd

to stop caching the main LV while keeping things around, separate the fast LV from the main LV. this changes the type of the main LV back to what it was before the cache got attached. so iconic, it reverts ✨

# lvconvert --splitcache vg/main
# lvs -a
  LV   VG Attr       Type   Devices
  fast vg -wi------- linear /dev/fast_ssd
  main vg -wi------- linear /dev/slow_hdd

7. Create a new LV with caching

you can also make a brand new LV with caching attached at the moment of creation, using this command:

# lvcreate --type cache|writecache -n Name -L Size
     --cachedevice /dev/fast_ssd vg /dev/slow_hdd

the main LV is created with the specified Name and Size from the slow_hdd. a hidden fast LV is created on the fast_ssd and is then attached to the new main LV. if the fast_ssd is unused, the entire disk will be used as the cache β€” unless you use the --cachesize option to specify a size for the fast LV. and yes, the --cachedevice option can be repeated to use multiple disks for the fast LV. stacking devices like it's nothing πŸ”₯


🫢 OPTIONS 🫢

option args

  • --cachepool CachePoolLV|LV β€” pass this option a cachepool LV or a standard LV. when you use a cache pool, lvm places cache data and cache metadata on different LVs. those two LVs together are what's called a cache pool. this gives a bit better performance for dm-cache and permits specific placement and segment type selection for the data and metadata volumes. a cache pool is represented as a special type of LV that cannot be used directly. if you pass a standard LV with this option, lvm will first convert it to a cache pool by combining it with another LV to use for metadata. this option can be used with dm-cache.
  • --cachevol LV β€” pass this option a fast LV that should be used to hold the cache. with a cachevol, cache data and metadata are stored in different parts of the same fast LV. this option can be used with dm-writecache or dm-cache.
  • --cachedevice PV β€” this option can be used in place of --cachevol, in which case a cachevol LV will be created using the specified device. this option can be repeated to create a cachevol using multiple devices, or a tag name can be specified β€” in which case the cachevol will be created using any of the devices with the given tag. if a named cache device is unused, the entire device will be used to create the cachevol. and to create a cachevol of a specific size from the cache devices, include the --cachesize option.

dm-cache block size

a cache pool will have a logical block size of 4096 bytes if it is created on a device with a logical block size of 4096 bytes.

⚠️ and here's the drama: if a main LV has logical block size 512 (with an existing xfs file system using that size), then it cannot use a cache pool with a 4096 logical block size. if the cache pool is attached anyway, the main LV will likely fail to mount. 😭

to dodge this whole problem, use a mkfs option to specify a 4096 block size for the file system, or attach the cache pool before running mkfs.

dm-writecache block size

the dm-writecache block size can be 4096 bytes (which is the default) or 512 bytes. the default 4096 has better performance and should be used β€” except when 512 is necessary for compatibility. the dm-writecache block size is specified with --cachesettings block_size=4096|512 when caching is started.

when a file system like xfs already exists on the main LV prior to caching, and that file system is using a block size of 512, then the writecache block size should be set to 512. (the file system will likely fail to mount if a writecache block size of 4096 is used in this case. πŸ™ˆ)

check the xfs sector size while the fs is mounted:

# xfs_info /dev/vg/main

look for sectsz=512 or sectsz=4096.

the writecache block size should be chosen to match the xfs sectsz value.

it's also possible to specify a sector size of 4096 to mkfs.xfs when creating the file system. in that case, the writecache block size of 4096 can be used.

the writecache block size is displayed by the command:

# lvs -o writecacheblocksize VG/LV

dm-writecache memory usage

the amount of main system memory used by dm-writecache can totally be a factor when you're selecting the writecache cachevol size and the writecache block size. plan accordingly bestie 🧠

  • writecache block size 4096: each 100 GiB of writecache cachevol uses slightly over 2 GiB of system memory.
  • writecache block size 512: each 100 GiB of writecache cachevol uses a little over 16 GiB of system memory.

dm-writecache settings

to specify dm-writecache tunable settings on the command line, use:

--cachesettings 'option=N' or

--cachesettings 'option1=N option2=N ...'

for example, --cachesettings 'high_watermark=90 writeback_jobs=4'.

to include settings when caching is started, run:

# lvconvert --type writecache --cachevol fast \
     --cachesettings 'option=N' vg/main

to change settings for an existing writecache, run:

# lvchange --cachesettings 'option=N' vg/main

to clear all settings that have been applied, run:

# lvchange --cachesettings '' vg/main

to view the settings that are applied to a writecache LV, run:

# lvs -o cachesettings vg/main

the tunable settings are:

  • high_watermark = <percent> β€” start writeback when the writecache usage reaches this percent (0-100).
  • low_watermark = <percent> β€” stop writeback when the writecache usage reaches this percent (0-100).
  • writeback_jobs = <count> β€” limit the number of blocks that are in flight during writeback. setting this value reduces writeback throughput, but it may improve latency of read requests.
  • autocommit_blocks = <count> β€” when the application writes this amount of blocks without issuing the FLUSH request, the blocks are automatically committed.
  • autocommit_time = <milliseconds> β€” the data is automatically committed if this time passes and no FLUSH request is received.
  • fua = 0|1 β€” use the FUA flag when writing data from persistent memory back to the underlying device. applicable only to persistent memory.
  • nofua = 0|1 β€” don't use the FUA flag when writing back data, and send the FLUSH request afterwards. some underlying devices perform better with fua, some with nofua. testing is necessary to determine which. applicable only to persistent memory.
  • cleaner = 0|1 β€” set cleaner=1 and the writecache cleaner mode turns on, meaning data gets gradually flushed out of the cache πŸ’§. Do this before you detach the writecache and splitcache will have little or no flushing left to do. Don't do it beforehand and splitcache enables cleaner mode for you and waits for the flushing to finish before it detaches the writecache. Add cleaner=0 to the splitcache command and you skip cleaner mode entirely β€” any flushing that's still required happens in device suspend.
  • max_age = <milliseconds> β€” the maximum age of a block, in milliseconds. If a block chills in the cache for too long, it gets written to the underlying device and cleaned up. ✨
  • metadata_only = 0|1 β€” only metadata gets promoted to the cache. This one improves performance for heavier REQ_META workloads.
  • pause_writeback = <milliseconds> β€” pause writeback if there was some write I/O redirected to the origin volume in the last however-many milliseconds. 🫢

πŸŽ€ dm-writecache using metadata profiles πŸ’–

Setting writecache settings on the command line is not the only way, bestie β€” they can also be set in lvm.conf(5), or in a profile file, using the allocation/cache_settings/writecache config structure down below. πŸ‘‡

And you can literally prepare a whole roster of different profile files in the /etc/lvm/profile directory and just name the profile file you want when you start writecache. So organised of you.

Example

# cat <<EOF > /etc/lvm/profile/cache_writecache.profile
allocation {
       cache_settings {
              writecache {
                     high_watermark=60
                     writeback_jobs=1024
              }
       }
}
EOF

# lvcreate -an -L10G --name fast vg /dev/fast_ssd
# lvcreate --type writecache -L10G --name main  --cachevol fast \
   --metadataprofile cache_writecache vg /dev/slow_hdd

πŸ’… dm-cache with separate data and metadata LVs πŸ’…

The preferred way to use dm-cache is putting the cache metadata and the cache data on separate LVs. To pull that off, you make a "cache pool", which is a special LV that references two sub LVs β€” one for data and one for metadata. Two little babies, one pool. 😭

To create a cache pool of a given data size and let LVM2 calculate an appropriate metadata size:

# lvcreate --type cache-pool -L DataSize -n fast vg /dev/fast_ssd1

To create a cache pool from a separate LV and let LVM2 calculate an appropriate cache metadata size:

# lvcreate -n fast -L DataSize vg /dev/fast_ssd1
# lvconvert --type cache-pool vg/fast /dev/fast_ssd1

To create a cache pool from two separate LVs:

# lvcreate -n fast -L DataSize vg /dev/fast_ssd1
# lvcreate -n fastmeta -L MetadataSize vg /dev/fast_ssd2
# lvconvert --type cache-pool --poolmetadata fastmeta vg/fast

Then you take that cache pool LV and use it to start caching the main LV:

# lvconvert --type cache --cachepool fast vg/main

There's a variation of the same procedure where the cache pool gets created automatically the moment caching starts. πŸͺ„ For that, use a standard LV as the --cachepool (this one holds the cache data), and use another standard LV as the --poolmetadata (this one holds the cache metadata). LVM will create a cache pool LV out of the two specified LVs, and use that cache pool to start caching the main LV.

# lvcreate -n fast -L DataSize vg /dev/fast_ssd1
# lvcreate -n fastmeta -L MetadataSize vg /dev/fast_ssd2
# lvconvert --type cache --cachepool fast \
        --poolmetadata fastmeta vg/main

🌈 dm-cache cache modes 🌈

Default dm-cache cache mode? That's "writethrough" and it's giving safety. Writethrough makes sure any data written gets stored both in the cache and on the origin LV. So if a device associated with the cache goes poof in this mode, that does not mean you lose any data. πŸ›‘οΈ

Second cache mode is "writeback". Writeback delays writing data blocks from the cache back to the origin LV. This mode bumps performance, but losing a cache device can result in lost data. ⚠️ Not a vibe. Choose accordingly.

With the --cachemode option the cache mode can be set when caching is started, or changed on an LV that's already cached. And you can display the current cache mode with the cache_mode reporting option:

# lvs -o+cache_mode VG/LV

lvm.conf(5) allocation/cache_mode – defines the default cache mode.

# lvconvert --type cache --cachemode writethrough \
        --cachepool fast vg/main
# lvconvert --type cache --cachemode writethrough \
        --cachevol fast vg/main

πŸ¦‹ dm-cache chunk size πŸ¦‹

The size of the data blocks dm-cache manages can be specified with the --chunksize option when caching is started. Default unit is KiB πŸ’–. The value has to be a multiple of 32 KiB between 32 KiB and 1 GiB. Cache chunks bigger than 512 KiB should only be used when necessary β€” like, really.

Going too large on your chunk size can result in wasteful use of the cache, where small reads and writes cause large sections of an LV to get stored in the cache. It can also require increasing the migration threshold, which defaults to 2048 sectors (1 MiB). LVM2 ensures the migration threshold is at least 8 chunks in size. This may in some cases result in very high bandwidth load of transferring data between the cache LV and its cache origin LV. But going too small on the chunk size can result in more overhead trying to manage the numerous chunks that get mapped into the cache. That overhead can include both excessive CPU time searching for chunks, and excessive memory tracking chunks. It's giving pick your struggle. 😭

Command to display the chunk size:

# lvs -o+chunksize VG/LV

lvm.conf(5) allocation/cache_pool_chunk_size

controls the default chunk size.

The default value is shown by:

# lvmconfig --type default allocation/cache_pool_chunk_size

Checking migration threshold (in sectors) of running cached LV:

# lvs -o+kernel_cache_settings VG/LV

πŸ’œ dm-cache cache settings πŸ’œ

To set a dm-cache cache setting, use:

--cachesettings 'option1=N option2=N ...'

To unset/drop a cache setting and restore its default kernel value, use the special keyword 'default' as the option parameter:

--cachesettings 'option1=default option2=default ...'

⭐ dm-cache migration threshold cache setting ⭐

Migrating data between the origin and cache LV uses bandwidth. ✨ The user can set a throttle so no more than a certain amount of migration happens at any one time. Currently dm-cache is not taking any account of normal I/O traffic going to the devices. Ngl, it's just not watching.

The user can set the migration threshold via cache policy settings as "migration_threshold=<#sectors>" to set the maximum number of sectors being migrated, the default being 2048 sectors (1 MiB) or 8 cache chunks, whichever of those two values is larger.

Command to set migration threshold to 2 MiB (4096 sectors):

# lvchange --cachesettings 'migration_threshold=4096' VG/LV

Command to display the migration threshold:

# lvs -o+kernel_cache_settings,cache_settings VG/LV
# lvs -o+chunksize VG/LV

Command to restore/revert to default value:

# lvchange --cachesettings 'migration_threshold=default' VG/LV

🧚 dm-cache cache policy 🧚

The dm-cache subsystem has extra per-LV parameters: the cache policy to use, and possibly tunable parameters for that cache policy. Three policies are available right now: "smq" is the default policy, "mq" is an older implementation, and "cleaner" is used to force the cache to write back (flush) all cached writes to the origin LV. πŸ‘‘

The older "mq" policy has a number of tunable parameters. The defaults are chosen to be suitable for the majority of systems, but in special circumstances, changing the settings can improve performance. Newer kernels, however, alias this policy with the "smq" policy. Cache settings used to configure "mq" policy [random_threshold, sequential_threshold, discard_promote_adjustment, read_promote_adjustment, write_promote_adjustment] are thus silently ignored, and the performance matches that of "smq". πŸ’€ Your tuning is just going into the void.

With the --cachepolicy and --cachesettings options, the cache policy and settings can be set when caching is started, or changed on an existing cached LV (both options can be used together 🫢). The current cache policy and settings can be displayed with the cache_policy and cache_settings reporting options:

# lvs -o+cache_policy,cache_settings VG/LV

Change the cache policy and settings of an existing LV.

# lvchange --cachepolicy mq --cachesettings \
     'migration_threshold=2048 random_threshold=4' vg/main

lvm.conf(5) allocation/cache_policy – defines the default cache policy.

lvm.conf(5) allocation/cache_settings – defines the default cache settings.


🌸 dm-cache using metadata profiles 🌸

Cache pools allow a whole variety of options to be set. Many of those settings can be specified in lvm.conf or profile settings. You can prepare a number of different profiles in the /etc/lvm/profile directory and just specify the metadata profile file name when you're caching an LV or creating a cache-pool. Check the output of:

# lvmconfig --type default --withcomments

for a detailed description of all individual cache settings. πŸ‘€

Example

# cat <<EOF > /etc/lvm/profile/cache_big_chunk.profile
allocation {
       cache_pool_metadata_require_separate_pvs=0
       cache_pool_chunk_size=512
       cache_metadata_format=2
       cache_mode="writethrough"
       cache_policy="smq"
       cache_settings {
              smq {
                     migration_threshold=8192
              }
       }
}
EOF

# lvcreate --cache -L10G --metadataprofile cache_big_chunk vg/main \
        /dev/fast_ssd
# lvcreate --cache -L10G vg/main --config \
        'allocation/cache_pool_chunk_size=512' /dev/fast_ssd

πŸŽ€ dm-cache spare metadata LV πŸŽ€

See lvmthin(7) for a description of the "pool metadata spare" LV. The exact same concept is used for cache pools. Reuse, recycle, iconic. ♻️


🌈 dm-cache metadata formats 🌈

There are two disk formats for dm-cache metadata. The metadata format can be specified with --cachemetadataformat when caching is started, and it cannot be changed afterwards. Format 2 has better performance; it is more compact, and stores dirty bits in a separate btree, which improves the speed of shutting down the cache. With auto, lvm selects the best option provided by the current dm-cache kernel module. ✨


πŸ‘‘ RAID1 cache device πŸ‘‘

RAID1 can be used to create the fast LV holding the cache so it can tolerate a device failure. πŸ’ͺ (When using dm-cache with separate data and metadata LVs, each of the sub-LVs can use RAID1.)

# lvcreate -n main -L Size vg /dev/slow
# lvcreate --type raid1 -m 1 -n fast -L Size vg /dev/ssd1 /dev/ssd2
# lvconvert --type cache --cachevol fast vg/main

πŸ’– dm-cache command shortcut πŸ’–

A single command can be used to cache the main LV with automatic creation of a cache-pool:

# lvcreate --cache --size CacheDataSize VG/LV [FastPVs]

or the longer variant

okay so</p> you've got an LV that's already living its life, and now you want to give it a glow-up with a cache pool attached, here's the moment πŸ’…

# lvcreate --type cache --size CacheDataSize \
        --name NameCachePool VG/LV [FastPVs]

So what's actually happening here? The LV you named already exists β€” it's your main character LV, the one getting cached. ✨ This command spins up a brand-new cache pool with the size you gave (CacheDataSize) and either the name you handed over (NameCachePool) or, if you didn't name it, a name auto-picked from the sequence lvolX_cpool 😌 β€” using the fast PV(s) you optionally listed (FastPVs, which is typically an SSD, because obviously 🫢). Then it attaches that shiny new cache pool to your existing main LV so the caching can begin. slay. πŸ’–

(Note: make sure the main LV you specified is a standard LV. If you accidentally point it at a cache pool LV instead, this command does something different. not the vibe you want. πŸ™ˆ)
(Note: this command reads the --type option differently than normal lvcreate commands do. Usually --type says what type of LV is being created β€” but here, an LV with type cache-pool is being created, and the existing main LV is being converted to type cache. same flag, different lore, iykyk 🧚)

🌈 SEE ALSO 🌈

go read these when you're done here bestie, they're all part of the same friend group:


and that's the whole page, no cap πŸ’œ this one came from Red Hat, Inc., shipped as part of LVM TOOLS 2.03.44(2), dated 2026-10-27, and it is literally titled LVMCACHE(7). periodt. πŸŽ€πŸ¦‹πŸ’–

πŸ“¦ where this page comes from