ZFS(4)

🌸 NAME 🌸

zfs — tuning of the ZFS kernel module. (Yes bestie, this is the ZFS(4) kernel interfaces manual page, and we're about to tune the ZFS kernel module like it's our main character moment. 💅)

✨ DESCRIPTION ✨

okay so the ZFS module supports these parameters, and every single one is a little knob you can turn. no cap. Let's get into it. 🦋

• dbuf_cache_max_bytes=UINT64_MAXB (u64) — Maximum size in bytes of the dbuf cache. The target size is decided by the MIN versus 1/2^dbuf_cache_shift (1/32nd) of the target ARC size. You can spy on how the dbuf cache and its little settings are doing via the /proc/spl/kstat/zfs/dbufstats kstat. 👀
• dbuf_metadata_cache_max_bytes=UINT64_MAXB (u64) — Maximum size in bytes of the metadata dbuf cache. The target size is decided by the MIN versus 1/2^dbuf_metadata_cache_shift (1/64th) of the target ARC size. The behavior of the metadata dbuf cache and its associated settings can be observed via the /proc/spl/kstat/zfs/dbufstats kstat. 🎀
• dbuf_cache_hiwater_pct=10% (uint) — The percentage over dbuf_cache_max_bytes when dbufs must be evicted directly. no waiting, it's giving urgency. 💅
• dbuf_cache_lowater_pct=10% (uint) — The percentage below dbuf_cache_max_bytes when the evict thread stops evicting dbufs. so it can chill. 🧚
• dbuf_cache_shift=5 (uint) — Set the size of the dbuf cache (dbuf_cache_max_bytes) to a log2 fraction of the target ARC size. math but make it cache. ✨
• dbuf_metadata_cache_shift=6 (uint) — Set the size of the dbuf metadata cache (dbuf_metadata_cache_max_bytes) to a log2 fraction of the target ARC size. metadata stays iconic. 💜
• dbuf_mutex_cache_shift=0 (uint) — Set the size of the mutex array for the dbuf cache. When set to 0 the array is dynamically sized based on total system memory. lowkey adaptive. 🦋
• dmu_object_alloc_chunk_shift=7 (128) (uint) — dnode slots allocated in a single operation as a power of 2. The default value minimizes lock contention for the bulk operation performed. slay, less fighting. 🌈
• dmu_ddt_copies=3 (uint) — Controls the number of copies stored for DeDup Table (DDT) objects. Reducing the number of copies to 1 from the previous default of 3 can reduce the write inflation caused by deduplication. This assumes redundancy for this data is provided by the vdev layer. If the DDT is damaged, space may be leaked (not freed) when the DDT can not report the correct reference count. so like, be careful bestie. ⚠️
• dmu_prefetch_max=134217728B (128 MiB) (uint) — Limit the amount we can prefetch with one call to this amount in bytes. This helps to limit the amount of memory that can be used by prefetching. keep it cute, keep it capped. 🍓
• l2arc_feed_again=1|0 (int) — Turbo L2ARC warm-up. When the L2ARC is cold the fill interval will be set as fast as possible. zoom zoom. 🔥
• l2arc_feed_min_ms=200 (u64) — Min feed interval in milliseconds. Requires l2arc_feed_again=1 and only applicable in related situations. iykyk. 💫
• l2arc_feed_secs=1 (u64) — Seconds between L2ARC writing. periodt. 🕐
• l2arc_headroom=8 (u64) — How far through the ARC lists to search for L2ARC cacheable content, expressed as a multiplier of l2arc_write_max. ARC persistence across reboots can be achieved with persistent L2ARC by setting this parameter to 0, allowing the full length of ARC lists to be searched for cacheable content. full send. 🫶
• l2arc_headroom_boost=200% (u64) — Scales l2arc_headroom by this percentage when L2ARC contents are being successfully compressed before writing. A value of 100 disables this feature. compression ate. 💅
• l2arc_exclude_special=0|1 (int) — Controls whether buffers present on special vdevs are eligible for caching into L2ARC. If set to 1, exclude dbufs on special vdevs from being cached to L2ARC. special vdevs stay in their lane. 🎀
• l2arc_mfuonly=0|1|2 (int) — Controls whether only MFU metadata and data are cached from ARC into L2ARC. This may be desired to avoid wasting space on L2ARC when reading/writing large amounts of data that are not expected to be accessed more than once.

The default is 0, meaning both MRU and MFU data and metadata are cached. When turning off this feature (setting it to 0), some MRU buffers will still be present in ARC and eventually cached on L2ARC. If l2arc_noprefetch=0, some prefetched buffers will be cached to L2ARC, and those might later transition to MRU, in which case the l2arc_mru_asize arcstat will not be 0.

Setting it to 1 means to L2 cache only MFU data and metadata.

Setting it to 2 means to L2 cache all metadata (MRU+MFU) but only MFU data (i.e. MRU data are not cached). This can be the right setting to cache as much metadata as possible even when having high data turnover.

Regardless of l2arc_noprefetch, some MFU buffers might be evicted from ARC, accessed later on as prefetches and transition to MRU as prefetches. If accessed again they are counted as MRU and the l2arc_mru_asize arcstat will not be 0.

The ARC status of L2ARC buffers when they were first cached in L2ARC can be seen in the l2arc_mru_asize, l2arc_mfu_asize, and l2arc_prefetch_asize arcstats when importing the pool or onlining a cache device if persistent L2ARC is enabled.

The evict_l2_eligible_mru arcstat does not take into account if this option is enabled as the information provided by the evict_l2_eligible_m[rf]u arcstats can be used to decide if toggling this option is appropriate for the current workload. 💀

• l2arc_meta_percent=33% (uint) — Percent of ARC size allowed for L2ARC-only headers. Since L2ARC buffers are not evicted on memory pressure, too many headers on a system with an irrationally large L2ARC can render it slow or unusable. This parameter limits L2ARC writes and rebuilds to achieve the target. protect your peace. 🌸
• l2arc_trim_ahead=0% (u64) — Trims ahead of the current write size (l2arc_write_max) on L2ARC devices by this percentage of write size if we have filled the device. If set to 100 we TRIM twice the space required to accommodate upcoming writes. A minimum of 64 MiB will be trimmed. It also enables TRIM of the whole L2ARC device upon creation or addition to an existing pool or if the header of the device is invalid upon importing a pool or onlining a cache device. A value of 0 disables TRIM on L2ARC altogether and is the default as it can put significant stress on the underlying storage devices. This will vary depending on how well the specific device handles these commands. 💅
• l2arc_noprefetch=1|0 (int) — Do not write buffers to L2ARC if they were prefetched but not used by applications. In case there are prefetched buffers in L2ARC and this option is later set, we do not read the prefetched buffers from L2ARC. Unsetting this option is useful for caching sequential reads from the disks to L2ARC and serve those reads from L2ARC later on. This may be beneficial in case the L2ARC device is significantly faster in sequential reads than the disks of the pool.

Use 1 to disable and 0 to enable caching/reading prefetches to/from L2ARC. no cap. 📖

• l2arc_norw=0|1 (int) — No reads during writes. periodt. ✋
• l2arc_write_boost=33554432B (32 MiB) (u64) — Cold L2ARC devices will have l2arc_write_max increased by this amount while they remain cold. glow-up for cold cache. ❄️
• l2arc_write_max=33554432B (32 MiB) (u64) — Max write bytes per interval. keep it bounded. 🎀
• l2arc_rebuild_enabled=1|0 (int) — Rebuild the L2ARC when importing a pool (persistent L2ARC). This can be disabled if there are problems importing a pool or attaching an L2ARC device (e.g. the L2ARC device is slow in reading stored log metadata, or the metadata has become somehow fragmented/unusable). sometimes you gotta turn it off, bestie. 🙈
• l2arc_rebuild_blocks_min_l2size=1073741824B (1 GiB) (u64) — Minimum size of an L2ARC device required in order to write log blocks in it. The log blocks are used upon importing the pool to rebuild the persistent L2ARC.

For L2ARC devices less than 1 GiB, the amount of data l2arc_evict() evicts is significant compared to the amount of restored L2ARC data. In this case, do not write log blocks in L2ARC in order not to waste space. tiny device, no log blocks. 🧚

• metaslab_aliquot=2097152B (2 MiB) (u64) — Metaslab group's per child vdev allocation granularity, in bytes. This is roughly similar to what would be referred to as the "stripe size" in traditional RAID arrays. In normal operation, ZFS will try to write this amount of data to each child of a top-level vdev before moving on to the next top-level vdev. it's giving RAID but make it ZFS. 🌈
• metaslab_bias_enabled=1|0 (int) — Enable metaslab groups biasing based on their over- or under-utilization relative to the metaslab class average. If disabled, each metaslab group will receive allocations proportional to its capacity. fair and balanced. ⭐
• metaslab_perf_bias=1|0|2 (int) — Controls metaslab groups biasing based on their write performance. Setting to 0 makes all metaslab groups receive fixed amounts of allocations. Setting to 2 allows faster metaslab groups to allocate more. Setting to 1 equals to 2 if the pool is write-bound or 0 otherwise. That is, if the pool is limited by write throughput, then allocate more from faster metaslab groups, but if not, try to evenly distribute the allocations. performance-based main character energy. 👑
• metaslab_force_ganging=16777217B (16 MiB + 1 B) (u64) — Make some blocks above a certain size be gang blocks. This option is used by the test suite to facilitate testing. test suite things. 💀
• metaslab_force_ganging_pct=3% (uint) — For blocks that could be forced to be a gang block (due to metaslab_force_ganging), force this many of them to be gang blocks. gang gang. 🔥
• brt_zap_prefetch=1|0 (int) — Controls prefetching BRT records for blocks which are going to be cloned. clone ahead. 🦋
• brt_zap_default_bs=13 (8 KiB) (int) — Default BRT ZAP data block size as a power of 2. Note that changing this after creating a BRT on the pool will not affect existing BRTs, only newly created ones. new BRTs only, bestie. 💅
• brt_zap_default_ibs=13 (8 KiB) (int) — Default BRT ZAP indirect block size as a power of 2. Note that changing this after creating a BRT on the pool will not affect existing BRTs, only newly created ones. same energy, indirect this time. ✨
• ddt_zap_default_bs=15 (32 KiB) (int) — Your default DDT ZAP data block size, bestie, expressed as a power of 2. Heads up: changing this after a DDT has been created on the pool will not affect existing DDTs, only the newly created ones. 💅
• ddt_zap_default_ibs=15 (32 KiB) (int) — Same energy but for the default DDT ZAP indirect block size, also as a power of 2. And again: changing it after creating a DDT on the pool won't touch existing DDTs, only newly created ones. 🦋
• zfs_default_bs=9 (512 B) (int) — Default dnode block size, as a power of 2. ✨
• zfs_default_ibs=17 (128 KiB) (int) — Default dnode indirect block size, as a power of 2. 🎀
• zfs_dio_enabled=1|0 (int) — Turn Direct I/O on or off. If this setting is 0, then all I/O requests get directed through the ARC acting as though the dataset property direct was set to disabled. 🫶
• zfs_dio_strict=0|1 (int) — Strictly enforce alignment for Direct I/O requests, returning EINVAL if not page-aligned instead of silently falling back to uncached I/O. No sneaky fallback, periodt. 💅
• zfs_history_output_max=1048576B (1 MiB) (u64) — When trying to log an output nvlist of an ioctl in the on-disk history, the output will not be stored if it is larger than this size (in bytes). This must be less than DMU_MAX_ACCESS (64 MiB). This applies primarily to zfs_ioc_channel_program() (cf. zfs-program(8)). 🔥
• zfs_keep_log_spacemaps_at_export=0|1 (int) — Prevent log spacemaps from being destroyed during pool exports and destroys. Protect the spacemaps at all costs. 🫶
• zfs_metaslab_segment_weight_enabled=1|0 (int) — Enable or disable segment-based metaslab selection. 👑
• zfs_metaslab_switch_threshold=2 (int) — When using segment-based metaslab selection, keep allocating from the active metaslab until this option's worth of buckets have been exhausted. Slay. ✨
• metaslab_debug_load=0|1 (int) — Load all metaslabs during pool import. 🧚
• metaslab_debug_unload=0|1 (int) — Prevent metaslabs from being unloaded. 💀
• metaslab_fragmentation_factor_enabled=1|0 (int) — Enable use of the fragmentation metric when computing metaslab weights. 🌸
• metaslab_df_max_search=16777216B (16 MiB) (uint) — Maximum distance to search forward from the last offset. Without this limit, fragmented pools can see >100`000 iterations and metaslab_block_picker() becomes the performance limiting factor on high-performance storage. With the default setting of 16 MiB, we typically see less than 500 iterations, even with very fragmented ashift=9 pools. The maximum number of iterations possible is metaslab_df_max_search / 2^(ashift+1). With the default setting of 16 MiB this is 16*1024 (with ashift=9) or 2*1024 (with ashift=12). 🔥
• metaslab_df_use_largest_segment=0|1 (int) — If not searching forward (due to metaslab_df_max_search, metaslab_df_free_pct, or metaslab_df_alloc_threshold), this tunable controls which segment is used. If set, we will use the largest free segment. If unset, we will use a segment of at least the requested size. 🦋
• zfs_metaslab_max_size_cache_sec=3600s (1 hour) (u64) — When we unload a metaslab, we cache the size of the largest free chunk. We use that cached size to determine whether or not to load a metaslab for a given allocation. As more frees accumulate in that metaslab while it's unloaded, the cached max size becomes less and less accurate. After a number of seconds controlled by this tunable, we stop considering the cached max size and start considering only the histogram instead. 🧚
• zfs_metaslab_mem_limit=25% (uint) — When we are loading a new metaslab, we check the amount of memory being used to store metaslab range trees. If it is over a threshold, we attempt to unload the least recently used metaslab to prevent the system from clogging all of its memory with range trees. This tunable sets the percentage of total system memory that is the threshold. 💀
• zfs_metaslab_try_hard_before_gang=0|1 (int) — If unset, we will first try normal allocation. If that fails then we will do a gang allocation. If that fails then we will do a "try hard" gang allocation. If that fails then we will have a multi-layer gang block. If set, we will first try normal allocation. If that fails then we will do a "try hard" allocation. If that fails we will do a gang allocation. If that fails we will do a "try hard" gang allocation. If that fails then we will have a multi-layer gang block. 👑
• zfs_metaslab_find_max_tries=100 (uint) — When not trying hard, we only consider this number of the best metaslabs. This improves performance, especially when there are many metaslabs per vdev and the allocation can't actually be satisfied (so we would otherwise iterate all metaslabs). ✨
• zfs_vdev_default_ms_count=200 (uint) — When a vdev is added, target this number of metaslabs per top-level vdev. 🎀
• zfs_vdev_default_ms_shift=29 (512 MiB) (uint) — Default lower limit for metaslab size. 🌸
• zfs_vdev_max_ms_shift=34 (16 GiB) (uint) — Default upper limit for metaslab size. 🦋
• zfs_vdev_max_auto_ashift=14 (uint) — Maximum ashift used when optimizing for logical → physical sector size on new top-level vdevs. May be increased up to ASHIFT_MAX (16), but this may negatively impact pool space efficiency. Choose wisely, bestie. 💅
• zfs_vdev_direct_write_verify=Linux 1 | FreeBSD 0 (uint) — If non-zero, then a Direct I/O write's checksum will be verified every time the write is issued and before it is committed to the block pointer. In the event the checksum is not valid then the I/O operation will return EIO. This module parameter can be used to detect if the contents of the user's buffer have changed in the process of doing a Direct I/O write. It can also help to identify if reported checksum errors are tied to Direct I/O writes. Each verify error causes a dio_verify_wr zevent. Direct Write I/O checksum verify errors can be seen with zpool status -d. The default value for this is 1 on Linux, but is 0 for FreeBSD because user pages can be placed under write protection in FreeBSD before the Direct I/O write is issued. 🌈
• zfs_vdev_min_auto_ashift=ASHIFT_MIN (9) (uint) — Minimum ashift used when creating new top-level vdevs. ✨
• zfs_vdev_min_ms_count=16 (uint) — Minimum number of metaslabs to create in a top-level vdev. 🫶
• vdev_validate_skip=0|1 (int) — Skip label validation steps during pool import. Changing this is not recommended unless you know what you're doing and are recovering a damaged label. ⚠️
• zfs_vdev_ms_count_limit=131072 (128k) (uint) — Practical upper limit of total metaslabs per top-level vdev. 💀
• metaslab_preload_enabled=1|0 (int) — Enable metaslab group preloading. 🧚
• metaslab_preload_limit=10 (uint) — Maximum number of metaslabs per group to preload
• metaslab_preload_pct=50 (uint) — Percentage of CPUs to run a metaslab preload taskq
• metaslab_lba_weighting_enabled=1|0 (int) — Give more weight to metaslabs with lower LBAs, assuming they have greater bandwidth, as is typically the case on a modern constant angular velocity disk drive. 💖
• metaslab_unload_delay=32 (uint) — After a metaslab is used, we keep it loaded for this many TXGs, to attempt to reduce unnecessary reloading. Note that both this many TXGs and metaslab_unload_delay_ms milliseconds must pass before unloading will occur. 🦋
• metaslab_unload_delay_ms=600000ms (10 min) (uint) — After a metaslab is used, we keep it loaded for this many milliseconds, to attempt to reduce unnecessary reloading. Note, that both this many milliseconds and metaslab_unload_delay TXGs must pass before unloading will occur. 🌸
• reference_history=3 (uint) — Maximum reference holders being tracked when reference_tracking_enable is active. ✨
• raidz_expand_max_copy_bytes=160MB (ulong) — Max amount of memory to use for RAID-Z expansion I/O. This limits how much I/O can be outstanding at once. 🔥
• raidz_expand_max_reflow_bytes=0 (ulong) — For testing, pause RAID-Z expansion when reflow amount reaches this value. 🧪
• raidz_io_aggregate_rows=4 (ulong) — For expanded RAID-Z, aggregate reads that have more rows than this. 👑
• reference_history=3 (int) — Maximum reference holders being tracked when reference_tracking_enable is active. (Yes, again, bestie — it's in the manpage twice. Iconic. 💅)
• reference_tracking_enable=0|1 (int) — Track reference holders to refcount_t objects (debug builds only). 🫶
• send_holes_without_birth_time=1|0 (int) — When set, the hole_birth optimization will not be used, and all holes will always be sent during a zfs send. This is useful if you suspect your datasets are affected by a bug in hole_birth. 💀
• spa_config_path=/etc/zfs/zpool.cache (charp) — SPA config file. 🌈
• spa_asize_inflation=24 (uint) — Multiplication factor used to estimate actual disk consumption from the size of data being written. The default value is a worst case estimate, but lower values may be valid for a given pool depending on its configuration. Pool administrators who understand the factors involved may wish to specify a more realistic inflation factor, particularly if they operate close to quota or capacity limits. 📏
• spa_load_print_vdev_tree=0|1 (int) — Whether to print the vdev tree in the debugging message buffer during pool import. 🧚
• spa_load_verify_data=1|0 (int) — Whether to traverse data blocks during an "extreme rewind" (-X) import. An extreme rewind import normally performs a full traversal of all blocks in the pool for verification. If this parameter is unset, the traversal skips non-metadata blocks. It can be toggled once the import has started to stop or start the traversal of non-metadata blocks. ✨
• spa_load_verify_metadata=1|0 (int) — Whether to traverse blocks during an "extreme rewind" (-X) pool import. 🦋

and just to wrap that thought up — an extreme rewind import normally performs a full traversal of all blocks in the pool for verification, like a whole vibe check on every single block 💅. If this parameter is unset, the traversal is not performed. And bestie, you can toggle it once the import has already started, to stop or start the traversal. not me micromanaging mid-import 😌

• spa_load_verify_shift=4 (1/16th) (uint) — sets the maximum number of bytes to consume during pool import to the log2 fraction of the target ARC size.
• spa_slop_shift=5 (1/32nd) (int) — okay so normally we don't allow the last 3.2% (1/2^spa_slop_shift) of space in the pool to be consumed 🫶. This ensures we don't run the pool completely out of space because of unaccounted changes (e.g. to the MOS). It also limits the worst-case time to allocate space. If we have less than this amount of free space, most ZPL operations (e.g. write, create) will return ENOSPC 😭.
• spa_num_allocators=4 (int) — determines the number of block allocators to use per spa instance. Capped by the number of actual CPUs in the system via spa_cpus_per_allocator.

Note that setting this value too high could result in performance degradation and/or excess fragmentation. Set value only applies to pools imported/created after that. so like, don't be greedy with it 💀

• spa_cpus_per_allocator=4 (int) — determines the minimum number of CPUs in a system for block allocator per spa instance. Set value only applies to pools imported/created after that.
• spa_upgrade_errlog_limit=0 (uint) — limits the number of on-disk error log entries that will be converted to the new format when enabling the head_errlog feature. The default is to convert all log entries, no crumbs left ✨.
• vdev_read_sit_out_secs=600s (10 min) (ulong) — when a slow disk outlier is detected, it gets placed in a sit out state. While sitting out, the disk will not participate in normal reads, and instead its data will be reconstructed as needed from parity. Scrub operations will always read from a disk, even if it's sitting out. A number of disks in a RAID-Z or dRAID vdev may sit out at the same time, up to the number of parity devices. Writes will still be issued to a disk which is sitting out to maintain full redundancy. Defaults to 600 seconds, and a value of zero disables disk sit-outs in general, including slow disk outlier detection. it's giving timeout corner 🪑 but make it respectful.
• vdev_raidz_outlier_check_interval_ms=1000ms (1 sec) (ulong) — how often each RAID-Z and dRAID vdev will check for slow disk outliers. Increasing this interval will reduce the sensitivity of detection (since all I/Os since the last check are included in the statistics), but will slow the response to a disk developing a problem. Defaults to once per second; setting extremely small values may cause negative performance effects 🫠.
• vdev_raidz_outlier_insensitivity=50 (uint) — when performing slow outlier checks for RAID-Z and dRAID vdevs, this value is used to determine how far out an outlier must be before it counts as an event worth consdering. This is phrased as "insensitivity" because larger values result in fewer detections. Smaller values will result in more aggressive sitting out of disks that may have problems, but may significantly increase the rate of spurious sit-outs 👀.

To provide a more technical definition of this parameter, this is the multiple of the inter-quartile range (IQR) that is being used in a Tukey's Fence detection algorithm. This is much higher than a normal Tukey's Fence k-value, because the distribution under consideration is probably an extreme-value distribution, rather than a more typical Gaussian distribution. math girlies rise 🧚

• vdev_removal_max_span=32768B (32 KiB) (uint) — during top-level vdev removal, chunks of data are copied from the vdev which may include free space in order to trade bandwidth for IOPS. This parameter determines the maximum span of free space, in bytes, which will be included as "unnecessary" data in a chunk of copied data.

The default value here was chosen to align with zfs_vdev_read_gap_limit, which is a similar concept when doing regular reads (but there's no reason it has to be the same). friendship bracelet behavior fr 🫶

• vdev_file_logical_ashift=9 (512 B) (u64) — logical ashift for file-based devices.
• vdev_file_physical_ashift=9 (512 B) (u64) — physical ashift for file-based devices. twins 💅
• zap_iterate_prefetch=1|0 (int) — if set, when we start iterating over a ZAP object, prefetch the entire object (all leaf blocks). However, this is limited by dmu_prefetch_max. manifesting the whole thing upfront, lowkey genius 💖
• zap_micro_max_size=131072B (128 KiB) (int) — maximum micro ZAP size. A "micro" ZAP is upgraded to a "fat" ZAP once it grows beyond the specified size. Sizes higher than 128KiB will be clamped to 128KiB unless the large_microzap feature is enabled. glow-up arc, honestly 🌸
• zap_shrink_enabled=1|0 (int) — if set, adjacent empty ZAP blocks will be collapsed, reducing disk space. decluttering is self care periodt 🎀
• zfetch_min_distance=4194304B (4 MiB) (uint) — min bytes to prefetch per stream. Prefetch distance starts from the demand access size and quickly grows to this value, doubling on each hit. After that it may grow further by 1/8 per hit, but only if some prefetch since last time haven't completed in time to satisfy demand request, i.e. prefetch depth didn't cover the read latency or the pool got saturated. it starts small and then doubles like it's on main character energy ✨
• zfetch_max_distance=67108864B (64 MiB) (uint) — max bytes to prefetch per stream.
• zfetch_max_idistance=67108864B (64 MiB) (uint) — max bytes to prefetch indirects for per stream.
• zfetch_max_reorder=16777216B (16 MiB) (uint) — requests within this byte distance from the current prefetch stream position are considered parts of the stream, reordered due to parallel processing. Such requests do not advance the stream position immediately unless zfetch_hole_shift fill threshold is reached, but saved to fill holes in the stream later. held in the queue like a saved draft 🙈
• zfetch_max_streams=8 (uint) — max number of streams per zfetch (prefetch streams per file).
• zfetch_min_sec_reap=1 (uint) — min time before inactive prefetch stream can be reclaimed
• zfetch_max_sec_reap=2 (uint) — max time before inactive prefetch stream can be deleted
• zfs_abd_scatter_enabled=1|0 (int) — enables ARC from using scatter/gather lists and forces all allocations to be linear in kernel memory. Disabling can improve performance in some code paths at the expense of fragmented kernel memory. trade-offs, the eternal story 💀
• zfs_abd_scatter_max_order=MAX_ORDER-1 (uint) — maximum number of consecutive memory pages allocated in a single block for scatter/gather lists.

The value of MAX_ORDER depends on kernel configuration. so it's a "your mileage may vary" situation, not a fixed number, iykyk 🦋

• zfs_abd_scatter_min_size=1536B (1.5 KiB) (uint) — this is the minimum allocation size that will use scatter (page-based) ABDs. Smaller allocations will use linear ABDs.
• zfs_arc_dnode_limit=0B (u64) — when the number of bytes consumed by dnodes in the ARC exceeds this number of bytes, try to unpin some of it in response to demand for non-metadata. This value acts as a ceiling to the amount of dnode metadata, and defaults to 0, which indicates that a percent which is based on zfs_arc_dnode_limit_percent of the ARC meta buffers that may be used for dnodes.
• zfs_arc_dnode_limit_percent=10% (u64) — percentage that can be consumed by dnodes of ARC meta buffers.

See also zfs_arc_dnode_limit, which serves a similar purpose but has a higher priority if nonzero. 👑 the other one is the main character here.

• zfs_arc_dnode_reduce_percent=10% (u64) — percentage of ARC dnodes to try to scan in response to demand for non-metadata when the number of bytes consumed by dnodes exceeds zfs_arc_dnode_limit.
• zfs_arc_average_blocksize=8192B (8 KiB) (uint) — the ARC's buffer hash table is sized based on the assumption of an average block size of this value. This works out to roughly 1 MiB of hash table per 1 GiB of physical memory with 8-byte pointers. For configurations with a known larger average block size, this value can be increased to reduce the memory footprint. math is mathing 🔥
• zfs_arc_eviction_pct=200% (uint) — when arc_is_overflowing(), arc_get_data_impl() waits for this percent of the requested amount of data to be evicted. For example, by default, for every 2 KiB that's evicted, 1 KiB of it may be "reused" by a new allocation. Since this is above 100%, it ensures that progress is made towards getting arc_size under arc_c. Since this is finite, it ensures that allocations can still happen, even during the potentially long time that arc_size is more than arc_c. over 100% is the whole point bestie, it's about making progress 💜
• zfs_arc_evict_batch_limit=10 (uint) — number ARC headers to evict per sub-list before proceeding to another sub-list. This batch-style operation prevents entire sub-lists from being evicted at once but comes at a cost of additional unlocking and locking. one bite at a time, not the whole plate 🍓
• zfs_arc_evict_batches_limit=5 (uint) — number of zfs_arc_evict_batch_limit batches to process per parallel eviction task under heavy load to reduce number of context switches.
• zfs_arc_evict_threads=0 (int) — sets the number of ARC eviction threads to be used.

So like, the whole thread situation, here's the tea 🍵 — if you set it greater than 0, ZFS will dedicate up to that many threads to ARC eviction. Each thread will process one sub-list at a time, until the eviction target is reached or all sub-lists have been processed. And when it's set to 0, ZFS will compute a reasonable number of eviction threads based on the number of CPUs. It's giving "girl math but for CPUs":

• 1-4 CPUs → 1 Threads
• 5-8 CPUs → 2 Threads
• 9-15 CPUs → 3 Threads
• 16-31 CPUs → 4 Threads
• 32-63 CPUs → 6 Threads
• 64-95 CPUs → 8 Threads
• 96-127 CPUs → 9 Threads
• 128-160 CPUs → 11 Threads
• 160-191 CPUs → 12 Threads
• 192-223 CPUs → 13 Threads
• 224-255 CPUs → 14 Threads
• 256+ CPUs → 16 Threads

More threads may improve the responsiveness of ZFS to memory pressure. This can be important for performance when eviction from the ARC becomes a bottleneck for reads and writes.

Heads up bestie: this parameter can only be set at module load time. No changing it later, that's the rule. 💅

• zfs_arc_grow_retry=0s (uint) — If you set it to a non zero value, it will replace the arc_grow_retry value with this value. And what is arc_grow_retry even, you ask? It's the number of seconds the ARC will wait before trying to resume growth after a memory pressure event (default 5s). So it's the "gimme a sec, catching my breath" timer. 🫶
• zfs_arc_lotsfree_percent=10% (int) — Throttle I/O when free system memory drops below this percentage of total system memory. And if you set this value to 0? Throttle's gone, disabled, bye. ✨
• zfs_arc_max=0B (u64) — Max size of ARC in bytes, no cap. Well, there's a cap. If it's 0, then the max size of ARC is determined by the amount of system memory installed. The larger of all_system_memory - 1 GiB and 5/8 × all_system_memory will be used as the limit. This value must be at least 67108864B (64 MiB).

This value can be changed dynamically, with some caveats. It cannot be set back to 0 while running, and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking.

• zfs_arc_meta_balance=500 (uint) — Balance between metadata and data on ghost hits. Values above 100 increase metadata caching by proportionally reducing effect of ghost data hits on target data/metadata rate. It's literally a vibe balance slider between the two. ⚖️
• zfs_arc_min=0B (u64) — Min size of ARC in bytes. If set to 0, arc_c_min will default to consuming the larger of 32 MiB and all_system_memory / 32.
• zfs_arc_min_prefetch_ms=0ms(≡1s) (uint) — Minimum time prefetched blocks are locked in the ARC. Short but mighty. ✨
• zfs_arc_min_prescient_prefetch_ms=0ms(≡6s) (uint) — Minimum time "prescient prefetched" blocks are locked in the ARC. These blocks are meant to be prefetched fairly aggressively ahead of the code that may use them. Prescient = it already knows, main character energy fr. 👑
• zfs_arc_prune_task_threads=1 (int) — Number of arc_prune threads. FreeBSD does not need more than one. Linux may theoretically use one per mount point up to number of CPUs, but that was not proven to be useful. So, not worth it bestie. 💀
• zfs_max_missing_tvds=0 (int) — Number of missing top-level vdevs which will be allowed during pool import (only in read-only mode).
• zfs_max_nvlist_src_size= 0 (u64) — Maximum size in bytes allowed to be passed as zc_nvlist_src_size for ioctls on /dev/zfs. This prevents a user from causing the kernel to allocate an excessive amount of memory. When the limit is exceeded, the ioctl fails with EINVAL and a description of the error is sent to the zfs-dbgmsg log. This parameter should not need to be touched under normal circumstances. If 0, equivalent to a quarter of the user-wired memory limit under FreeBSD and to 134217728B (128 MiB) under Linux. Put it down and back away slowly. 🙈
• zfs_multilist_num_sublists=0 (uint) — To allow more fine-grained locking, each ARC state contains a series of lists for both data and metadata objects. Locking is performed at the level of these "sub-lists". This parameters controls the number of sub-lists per ARC state, and also applies to other uses of the multilist data structure.

If 0, equivalent to the greater of the number of online CPUs and 4.

• zfs_arc_overflow_shift=8 (int) — The ARC size is considered to be overflowing if it exceeds the current ARC target size (arc_c) by thresholds determined by this parameter. Exceeding by (arc_c >> zfs_arc_overflow_shift) / 2 starts ARC reclamation process. If that appears insufficient, exceeding by (arc_c >> zfs_arc_overflow_shift) × 1.5 blocks new buffer allocation until the reclaim thread catches up. Started reclamation process continues till ARC size returns below the target size.

The default value of 8 causes the ARC to start reclamation if it exceeds the target size by 0.2% of the target size, and block allocations by 0.6%.

• zfs_arc_shrink_shift=0 (uint) — If nonzero, this will update arc_shrink_shift (default 7) with the new value.
• zfs_arc_pc_percent=0% (off) (uint) — Percent of pagecache to reclaim ARC to. 🎀

This tunable allows the ZFS ARC to play more nicely with the kernel's LRU pagecache. It can guarantee that the ARC size won't collapse under scanning pressure on the pagecache, yet still allows the ARC to be reclaimed down to zfs_arc_min if necessary. This value is specified as percent of pagecache size (as measured by NR_ACTIVE_FILE + NR_INACTIVE_FILE), where that percent may exceed 100. This only operates during memory pressure/reclaim.

• zfs_arc_shrinker_limit=0 (int) — This is a limit on how many pages the ARC shrinker makes available for eviction in response to one page allocation attempt. Note that in practice, the kernel's shrinker can ask us to evict up to about four times this for one allocation attempt. To reduce OOM risk, this limit is applied for kswapd reclaims only.

For example a value of 10000 (in practice, 160 MiB per allocation attempt with 4 KiB pages) limits the amount of time spent attempting to reclaim ARC memory to less than 100 ms per allocation attempt, even with a small average compressed block size of ~8 KiB.

The parameter can be set to 0 (zero) to disable the limit, and only applies on Linux.

• zfs_arc_shrinker_seeks=2 (int) — Relative cost of ARC eviction on Linux, AKA number of seeks needed to restore evicted page. Bigger values make ARC more precious and evictions smaller, comparing to other kernel subsystems. Value of 4 means parity with page cache. So like, protect the ARC at all costs. 🔥
• zfs_arc_sys_free=0B (u64) — The target number of bytes the ARC should leave as free memory on the system. If zero, equivalent to the bigger of 512 KiB and all_system_memory/64.
• zfs_checksum_events_per_second=20/s (uint) — Rate limit checksum events to this many per second. Note that this should not be set below the ZED thresholds (currently 10 checksums over 10 seconds) or else the daemon may not trigger any action.

Do not set this below the ZED thresholds, seriously. Your daemon will just sit there doing nothing and you'll be so confused. 😭

• zfs_commit_timeout_pct=10% (uint) — This controls the amount of time that a ZIL block (lwb) will remain "open" when it isn't "full", and it has a thread waiting for it to be committed to stable storage. The timeout is scaled based on a percentage of the last lwb latency to avoid significantly impacting the latency of each individual transaction record (itx).
• zfs_condense_indirect_commit_entry_delay_ms=0ms (int) — Vdev indirection layer (used for device removal) sleeps for this many milliseconds during mapping generation. Intended for use with the test suite to throttle vdev removal speed.
• zfs_condense_indirect_obsolete_pct=25% (uint) — Minimum percent of obsolete bytes in vdev mapping required to attempt to condense (see zfs_condense_indirect_vdevs_enable). Intended for use with the test suite to facilitate triggering condensing as needed.
• zfs_condense_indirect_vdevs_enable=1|0 (int) — Enable condensing indirect vdev mappings. When set, attempt to condense indirect vdev mappings if the mapping uses more than zfs_condense_min_mapping_bytes bytes of memory and if the obsolete space map object uses more than zfs_condense_max_obsolete_bytes bytes on-disk. The condensing process is an attempt to save memory by removing obsolete mappings. Cleaning house, but make it storage. 🧹
• zfs_condense_max_obsolete_bytes=1073741824B (1 GiB) (u64) — Only attempt to condense indirect vdev mappings if the on-disk size of the obsolete space map object is greater than this number of bytes (see zfs_condense_indirect_vdevs_enable).
• zfs_condense_min_mapping_bytes=131072B (128 KiB) (u64) — Minimum size vdev mapping to attempt to condense (see zfs_condense_indirect_vdevs_enable).
• zfs_dbgmsg_enable=1|0 (int) — Internally ZFS keeps a small log to facilitate debugging. The log is enabled by default, and can be disabled by unsetting this option. The contents of the log can be accessed by reading /proc/spl/kstat/zfs/dbgmsg. Writing 0 to the file clears the log. 🫧

This setting does not influence debug prints due to zfs_flags.

• zfs_dbgmsg_maxsize=4194304B (4 MiB) (uint) — Maximum size of the internal ZFS debug log.
• zfs_dbuf_state_index=0 (int) — Historically used for controlling what reporting was available under /proc/spl/kstat/zfs. No effect. It's giving relic of the past, lowkey iconic but useless. 💀
• zfs_deadman_checktime_ms=60000ms (1 min) (u64) — Check time in milliseconds. This defines the frequency at which we check for hung I/O requests and potentially invoke the zfs_deadman_failmode behavior. Deadman said bestie, I'm watching. 👀
• zfs_deadman_enabled=1|0 (int) — okay so when a pool sync operation takes longer than zfs_deadman_synctime_ms, or when an individual I/O operation takes longer than zfs_deadman_ziotime_ms, then the operation is considered to be "hung" 😱. If zfs_deadman_enabled is set, then the deadman behavior is invoked as described by zfs_deadman_failmode. By default, the deadman is enabled and set to wait which results in "hung" I/O operations only being logged. The deadman is automatically disabled when a pool gets suspended. 💀
• zfs_deadman_events_per_second=1/s (int) — Rate limit deadman zevents (which report hung I/O operations) to this many per second. ✨
• zfs_deadman_failmode=wait (charp) — Controls the failure behavior when the deadman detects a "hung" I/O operation. Valid values are:
  • wait — Wait for a "hung" operation to complete. For each "hung" operation a "deadman" event will be posted describing that operation.
  • continue — Attempt to recover from a "hung" operation by re-dispatching it to the I/O pipeline if possible.
  • panic — Panic the system. This can be used to facilitate automatic fail-over to a properly configured fail-over partner.
• zfs_deadman_synctime_ms=600000ms (10 min) (u64) — Interval in milliseconds after which the deadman is triggered and also the interval after which a pool sync operation is considered to be "hung". Once this limit is exceeded the deadman will be invoked every zfs_deadman_checktime_ms milliseconds until the pool sync completes. ⏳
• zfs_deadman_ziotime_ms=300000ms (5 min) (u64) — Interval in milliseconds after which the deadman is triggered and an individual I/O operation is considered to be "hung". As long as the operation remains "hung", the deadman will be invoked every zfs_deadman_checktime_ms milliseconds until the operation completes. ⏱️
• zfs_dedup_prefetch=0|1 (int) — Enable prefetching dedup-ed blocks which are going to be freed. 🧹
• zfs_dedup_log_flush_min_time_ms=1000(uint) — Minimum time to spend on dedup log flush each transaction.

At least this long will be spent flushing dedup log entries each transaction, up to zfs_txg_timeout. This occurs even if doing so would delay the transaction, that is, other IO completes under this time. fr fr

• zfs_dedup_log_flush_entries_min=100(uint) — Flush at least this many entries each transaction.

OpenZFS will flush a fraction of the log every TXG, to keep the size proportional to the ingest rate (see zfs_dedup_log_flush_txgs). This sets the minimum for that estimate, which prevents the backlog from completely draining if the ingest rate falls. Raising it can force OpenZFS to flush more aggressively, reducing the backlog to zero more quickly, but can make it less able to back off if log flushing would compete with other IO too much. 💅

• zfs_dedup_log_flush_entries_max=UINT_MAX(uint) — Flush at most this many entries each transaction.

Mostly used for debugging purposes. 🐛

• zfs_dedup_log_flush_txgs=100(uint) — Target number of TXGs to process the whole dedup log.

Every TXG, OpenZFS will process the inverse of this number times the size of the DDT backlog. This will keep the backlog at a size roughly equal to the ingest rate times this value. This offers a balance between a more efficient DDT log, with better aggregation, and shorter import times, which increase as the size of the DDT log increases. Increasing this value will result in a more efficient DDT log, but longer import times. 📚

• zfs_dedup_log_cap=UINT_MAX(uint) — Soft cap for the size of the current dedup log.

If the log is larger than this size, we increase the aggressiveness of the flushing to try to bring it back down to the soft cap. Setting it will reduce import times, but will reduce the efficiency of the DDT log, increasing the expected number of IOs required to flush the same amount of data. 📉

• zfs_dedup_log_hard_cap=0|1 (uint) — Whether to treat the log cap as a firm cap or not.

When set to 0 (the default), the zfs_dedup_log_cap will increase the maximum number of log entries we flush in a given txg. This will bring the backlog size down towards the cap, but not at the expense of making TXG syncs take longer. If this is set to 1, the cap acts more like a hard cap than a soft cap; it will also increase the minimum number of log entries we flush per TXG. Enabling it will reduce worst-case import times, at the cost of increased TXG sync times. ⚖️

• zfs_dedup_log_flush_flow_rate_txgs=10(uint) — Number of transactions to use to compute the flow rate.

OpenZFS will estimate number of entries changed (ingest rate), number of entries flushed (flush rate) and time spent flushing (flush time rate) and combining these into an overall "flow rate". It will use an exponential weighted moving average over some number of recent transactions to compute these rates. This sets the number of transactions to compute these averages over. Setting it higher can help to smooth out the flow rate in the face of spiky workloads, but will take longer for the flow rate to adjust to a sustained change in the ingress rate. 🌊

• zfs_dedup_log_txg_max=8(uint) — Max transactions to before starting to flush dedup logs.

OpenZFS maintains two dedup logs, one receiving new changes, one flushing. If there is nothing to flush, it will accumulate changes for no more than this many transactions before switching the logs and starting to flush entries out. 🔄

• zfs_dedup_log_mem_max=0(u64) — Max memory to use for dedup logs.

OpenZFS will spend no more than this much memory on maintaining the in-memory dedup log. Flushing will begin when around half this amount is being spent on logs. The default value of 0 will cause it to be set by zfs_dedup_log_mem_max_percent instead. 🧠

• zfs_dedup_log_mem_max_percent=1% (uint) — Max memory to use for dedup logs, as a percentage of total memory.

If zfs_dedup_log_mem_max is not set, it will be initialized as a percentage of the total memory in the system. 📊

• zfs_delay_min_dirty_percent=60% (uint) — Start to delay each transaction once there is this amount of dirty data, expressed as a percentage of zfs_dirty_data_max. This value should be at least zfs_vdev_async_write_active_max_dirty_percent. See “ZFS TRANSACTION DELAY”. 🐢
• zfs_delay_scale=500000 (int) — This controls how quickly the transaction delay approaches infinity. Larger values cause longer delays for a given amount of dirty data.

For the smoothest delay, this value should be about 1 billion divided by the maximum number of operations per second. This will smoothly handle between ten times and a tenth of this number. See “ZFS TRANSACTION DELAY”.

zfs_delay_scale × zfs_dirty_data_max must be smaller than 2^64. ⚠️

• zfs_dio_write_verify_events_per_second=20/s (uint) — Rate limit Direct I/O write verify events to this many per second. ✍️
• zfs_disable_ivset_guid_check=0|1 (int) — Disables requirement for IVset GUIDs to be present and match when doing a raw receive of encrypted datasets. Intended for users whose pools were created with OpenZFS pre-release versions and now have compatibility issues. 🕰️
• zfs_key_max_salt_uses=400000000 (4*10^8) (ulong) — Maximum number of uses of a single salt value before generating a new one for encrypted datasets. The default value is also the maximum. 🔐
• zfs_object_mutex_size=64 (uint) — Size of the znode hashtable used for holds.

Due to the need to hold locks on objects that may not exist yet, kernel mutexes are not created per-object and instead a hashtable is used where collisions will result in objects waiting when there is not actually contention on the same object. 🔒

• zfs_slow_io_events_per_second=20/s (int) — Rate limit delay zevents (which report slow I/O operations) to this many per second. 🐌
• zfs_unflushed_max_mem_amt=1073741824B (1 GiB) (u64) — Upper-bound limit for unflushed metadata changes to be held by the log spacemap in memory, in bytes. 🧮
• zfs_unflushed_max_mem_ppm=1000ppm (0.1%) (u64) — Part of overall system memory that ZFS allows to be used for unflushed metadata changes by the log spacemap, in millionths. 🔬
• zfs_unflushed_log_block_max=131072 (128k) (u64) — Describes the maximum number of log spacemap blocks allowed for each pool. The default value means that the space in all the log spacemaps can add up to no more than 131072 blocks (which means 16 GiB of logical space before compression and ditto blocks, assuming that blocksize is 128 KiB).

This tunable is important because it involves a trade-off between import time after an unclean export and the frequency of flushing metaslabs. The higher this number is, the more log blocks we allow when the pool is active which means that we flush metaslabs less often and thus decrease the number of I/O operations for spacemap updates per TXG. At the same time though, that means that in the event of an unclean export, there will be more log spacemap blocks for us to read, inducing overhead in the import time of the pool. The lower the number, the amount of flushing increases, destroying log blocks quicker as they become obsolete faster, which leaves less blocks to be read during import time after a crash.

Each log spacemap block existing during pool import leads to approximately one extra logical I/O issued. This is the reason why this tunable is exposed in terms of blocks rather than space used. 📦

• zfs_unflushed_log_block_min=1000 (u64) — bestie, picture this 💭: the metaslab count is smol but the incoming rate is loud, and next thing you know we're flushing every single metaslab every TXG. not the vibe. so this sets a floor — we always allow at least this many log blocks, no cap. slay ✨
• zfs_unflushed_log_block_pct=400% (u64) — the tunable that decides how many blocks can be used for the spacemap log, expressed as a percentage of the total number of unflushed metaslabs in the pool. it's giving budget allocation 💅
• zfs_unflushed_log_txg_max=1000 (u64) — caps the max time, measured in TXGs, that any metaslab is allowed to stay unflushed. what it effectively does is limit the maximum number of unflushed per-TXG spacemap logs you'll have to read after an unclean pool export 😭 yeah, your post-crash cleanup homework has a ceiling now.
• zfs_unlink_suspend_progress=0|1 (uint) — when this is on, files will NOT be asynchronously removed from the list of pending unlinks, and the space they consume gets leaked. once you disable it again and remount the dataset, those pending unlinks get processed and the freed space goes back to the pool. lowkey this one is used by the test suite, so it's not your everyday girly 🫶
• zfs_delete_blocks=20480 (ulong) — defines what counts as a "large file" for deletion purposes. files with more than zfs_delete_blocks get deleted asynchronously, while smaller ones get deleted synchronously. decreasing this value reduces the time spent in an unlink(2) system call, but at the expense of a longer delay before the freed space is available. and just so you know — this only applies on Linux 🐧
• zfs_dirty_data_max= (int) — the dirty space limit, in bytes 😌. once you exceed it, new writes get halted until space frees up. this parameter takes precedence over zfs_dirty_data_max_percent. see “ZFS TRANSACTION DELAY”.

defaults to physical_ram/10, capped at zfs_dirty_data_max_max.

• zfs_dirty_data_max_max= (int) — the maximum allowable value of zfs_dirty_data_max, expressed in bytes. important: this limit is only enforced at module load time, and it gets ignored if zfs_dirty_data_max is changed later. takes precedence over zfs_dirty_data_max_max_percent. see “ZFS TRANSACTION DELAY”.

defaults to min(physical_ram/4, 4GiB), or min(physical_ram/4, 1GiB) for 32-bit systems 💀

• zfs_dirty_data_max_max_percent=25% (uint) — same idea, but expressed as a percentage of physical RAM. also only enforced at module load time, also ignored if zfs_dirty_data_max is changed later. and zfs_dirty_data_max_max takes precedence over this one. see “ZFS TRANSACTION DELAY”.
• zfs_dirty_data_max_percent=10% (uint) — the dirty space limit expressed as a percentage of all memory. once exceeded, new writes get halted until space frees up. but zfs_dirty_data_max takes precedence over this one. see “ZFS TRANSACTION DELAY”. also subject to zfs_dirty_data_max_max 💜
• zfs_dirty_data_sync_percent=20% (uint) — start syncing out a transaction group if there's at least this much dirty data (as a percentage of zfs_dirty_data_max). should be less than zfs_vdev_async_write_active_min_dirty_percent. just sayin', keep it in order 🙈
• zfs_wrlog_data_max= (int) — the upper limit of write-transaction ZIL log data size, in bytes. write ops get throttled as you approach this limit, until log data is cleared out after a transaction group sync. bc of some overhead, it should be set to at least 2 times the size of zfs_dirty_data_max to prevent harming normal write throughput. it also should be smaller than the size of the slog device if slog is present 🎀

defaults to zfs_dirty_data_max*2

• zfs_fallocate_reserve_percent=110% (uint) — okay so since ZFS is a copy-on-write filesystem with snapshots, blocks can't really be preallocated for a file to guarantee later writes won't run out of space 😅. instead, fallocate(2) space preallocation just checks that there's currently enough space in the pool or the user's project quota allocation, and then creates a sparse file of the requested size. the requested space gets multiplied by zfs_fallocate_reserve_percent to leave room for indirect blocks and other internal metadata. set this to 0 and it disables fallocate(2) support entirely, making it return EOPNOTSUPP. iconic behavior tbh ✨
• zfs_fletcher_4_impl=fastest (string) — pick a fletcher 4 implementation, bestie.

supported selectors are: fastest, scalar, sse2, ssse3, avx2, avx512f, avx512bw, and aarch64_neon. all except fastest and scalar require instruction set extensions to be available, and will only show up if ZFS detects they're present at runtime. if multiple fletcher 4 implementations are available, the fastest one gets picked using a micro benchmark 🏃‍♀️. choosing scalar means the original CPU-based calculation is used. choosing anything other than fastest or scalar means vector instructions from the respective CPU instruction set get used. periodt 💅

• zfs_bclone_enabled=1|0 (int) — turns access to the block cloning feature on/off. if this is 0, then even if feature@block_cloning is enabled, any functions and system calls that try to clone blocks will act like the feature is disabled. the audacity, honestly 💀
• zfs_bclone_wait_dirty=1|0 (int) — when set to 1, the FICLONE and FICLONERANGE ioctls wait for any dirty data to be written to disk before proceeding. this makes sure the clone operation reliably succeeds, even if a file is modified and then immediately cloned 😌. heads up tho: for small files this may be slower than just copying the file. when set to 0, the clone operation immediately fails if it hits any dirty blocks. by default, waiting is enabled. queen behavior 👑
• zfs_blake3_impl=fastest (string) — pick a BLAKE3 implementation ✨

supported selectors are: cycle, fastest, generic, sse2, sse41, avx2, avx512. all except cycle, fastest and generic require instruction set extensions to be available, and only show up if ZFS detects them at runtime. if multiple BLAKE3 implementations are available, the fastest gets picked via a micro benchmark 💫. you can peep the benchmark results by reading this kstat file: /proc/spl/kstat/zfs/chksum_bench 🍓

• zfs_free_bpobj_enabled=1|0 (int) — enable/disable the processing of the free_bpobj object. simple as 💅
• zfs_async_block_max_blocks=UINT64_MAX (unlimited) (u64) — maximum number of blocks freed in a single TXG. keep it movin' 🦋
• zfs_max_async_dedup_frees=250000 (u64) — maximum number of dedup, clone or gang blocks freed in a single TXG. these frees may require additional I/O, which makes them more expensive, so there's a cap, bestie 💸
• zfs_async_free_zio_wait_interval=2000 (u64) — after freeing this many dedup, clone or gang blocks, wait for all pending I/Os to complete before continuing. lil breather moment 🌸
• zfs_vdev_async_read_max_active=3 (uint) — maximum asynchronous read I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_async_read_min_active=1 (uint) — minimum asynchronous read I/O operation active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_async_write_active_max_dirty_percent=60% (uint) — when the pool has more than this much dirty data, use zfs_vdev_async_write_max_active to limit active async writes. if the dirty data is between the minimum and maximum, the active I/O limit gets linearly interpolated 📈. see “ZFS I/O SCHEDULER”.
• zfs_vdev_async_write_active_min_dirty_percent=30% (uint) — when the pool has less than this much dirty data, use zfs_vdev_async_write_min_active to limit active async writes. if the dirty data is between the minimum and maximum, the active I/O limit gets linearly interpolated 📉. see “ZFS I/O SCHEDULER”.
• zfs_vdev_async_write_max_active=10 (uint) — maximum asynchronous write I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_async_write_min_active=2 (uint) — minimum asynchronous write I/O operations active to each device. see “ZFS I/O SCHEDULER”.

lower values are associated with better latency on rotational media, but poorer resilver performance 😬. the default of 2 was chosen as a compromise. a value of 3 has been shown to improve resilver performance further, at the cost of further increasing latency. pick your struggle, bestie 🫠

• zfs_vdev_initializing_max_active=1 (uint) — maximum initializing I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_initializing_min_active=1 (uint) — minimum initializing I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_max_active=1000 (uint) — the maximum number of I/O operations active to each device. ideally it's at least the sum of each queue's max_active. see “ZFS I/O SCHEDULER”. math girlies rise up 🧚
• zfs_vdev_open_timeout_ms=1000 (uint) — timeout value to wait before deciding a device is missing during import. super helpful for transient missing paths, like when links get briefly removed and recreated in response to udev events 👀
• zfs_vdev_rebuild_max_active=3 (uint) — maximum sequential resilver I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_rebuild_min_active=1 (uint) — minimum sequential resilver I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_removal_max_active=2 (uint) — maximum removal I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_removal_min_active=1 (uint) — minimum removal I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_scrub_max_active=2 (uint) — maximum scrub I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_scrub_min_active=1 (uint) — minimum scrub I/O operations active to each device. see “ZFS I/O SCHEDULER”.
• zfs_vdev_sync_read_max_active=10 (uint) — the max number of synchronous read I/O operations allowed to be active on each device at once. Go peek at “ZFS I/O SCHEDULER” if you want the full lore. 💫
• zfs_vdev_sync_read_min_active=10 (uint) — the min number of synchronous read I/O operations active to each device. Yeah bestie, the floor. See “ZFS I/O SCHEDULER”.
• zfs_vdev_sync_write_max_active=10 (uint) — maximum synchronous write I/O operations active to each device. See “ZFS I/O SCHEDULER”.
• zfs_vdev_sync_write_min_active=10 (uint) — minimum synchronous write I/O operations active to each device. See “ZFS I/O SCHEDULER”.
• zfs_vdev_trim_max_active=2 (uint) — max trim/discard I/O operations active to each device. See “ZFS I/O SCHEDULER”. 🧹
• zfs_vdev_trim_min_active=1 (uint) — min trim/discard I/O operations active to each device. See “ZFS I/O SCHEDULER”.
• zfs_vdev_nia_delay=5 (uint) — okay this one is genuinely so clever, listen. For non-interactive I/O (scrub, resilver, removal, initialize and rebuild), the number of concurrently-active I/O operations is capped at zfs_*_min_active, unless the vdev is "idle". And when is it idle? When there are no interactive I/O operations active (synchronous or otherwise) and zfs_vdev_nia_delay operations have completed since the last interactive operation — then boom, the vdev is considered "idle" and the number of concurrently-active non-interactive operations gets bumped up to zfs_*_max_active. See “ZFS I/O SCHEDULER”. 🦋
• zfs_vdev_nia_credit=5 (uint) — some HDDs prioritize sequential I/O so hard that concurrent random I/O latency reaches several seconds 😭 and on some HDDs that happens even if the sequential I/O ops are submitted one at a time, so setting zfs_*_max_active= 1 doesn't even help. To stop non-interactive I/O (like scrub) from monopolizing the device, no more than zfs_vdev_nia_credit operations can be sent while there are outstanding incomplete interactive operations. That enforced wait is what makes sure the HDD services the interactive I/O within a reasonable amount of time. See “ZFS I/O SCHEDULER”.
• zfs_vdev_failfast_mask=1 (uint) — decides if the driver should retire on a given error type. These options may be bitwise-ored together:
  • 1 — Device — No driver retries on device errors
  • 2 — Transport — No driver retries on transport errors.
  • 4 — Driver — No driver retries on driver errors.
• zfs_vdev_disk_max_segs=0 (uint) — maximum number of segments to add to a BIO (minimum 4, ok, remember that 🫶). If this is higher than the maximum allowed by the device queue or by the kernel itself, it gets clamped. Setting it to zero makes the kernel's ideal size get used. Heads up: this parameter only applies on Linux. 🐧
• zfs_expire_snapshot=300s (int) — how long before .zfs/snapshot expires.
• zfs_admin_snapshot=0|1 (int) — lets the creation, removal, or renaming of entries in the .zfs/snapshot directory cause the creation, destruction, or renaming of snapshots. When it's enabled, this works both locally and over NFS exports that have the no_root_squash option set. 👑
• zfs_snapshot_no_setuid=0|1 (int) — whether to disable setuid/setgid support for snapshot mounts triggered by access to the .zfs/snapshot directory, by setting the nosuid mount option.
• zfs_flags=0 (int) — sets extra debugging flags. These may be bitwise-ored together:
  • 1 — ZFS_DEBUG_DPRINTF — Enable dprintf entries in the debug log.
  • 2 — ZFS_DEBUG_DBUF_VERIFY — Enable extra dbuf verifications. ⭐
  • 4 — ZFS_DEBUG_DNODE_VERIFY — Enable extra dnode verifications. ⭐
  • 8 — ZFS_DEBUG_SNAPNAMES — Enable snapshot name verification.
  • 16 — ZFS_DEBUG_MODIFY — Check for illegally modified ARC buffers. ⭐
  • 64 — ZFS_DEBUG_ZIO_FREE — Enable verification of block frees.
  • 128 — ZFS_DEBUG_HISTOGRAM_VERIFY — Enable extra spacemap histogram verifications.
  • 256 — ZFS_DEBUG_METASLAB_VERIFY — Verify space accounting on disk matches in-memory range_trees.
  • 512 — ZFS_DEBUG_SET_ERROR — Enable SET_ERROR and dprintf entries in the debug log.
  • 1024 — ZFS_DEBUG_INDIRECT_REMAP — Verify split blocks created by device removal.
  • 2048 — ZFS_DEBUG_TRIM — Verify TRIM ranges are always within the allocatable range tree.
  • 4096 — ZFS_DEBUG_LOG_SPACEMAP — Verify that the log summary is consistent with the spacemap log and enable zfs_dbgmsgs for metaslab loading and flushing.
  • 8192 — ZFS_DEBUG_METASLAB_ALLOC — Enable debugging messages when allocations fail.
  • 16384 — ZFS_DEBUG_BRT — Enable BRT-related debugging messages.
  • 32768 — ZFS_DEBUG_RAIDZ_RECONSTRUCT — Enabled debugging messages for raidz reconstruction.
  • 65536 — ZFS_DEBUG_DDT — Enable DDT-related debugging messages.

⭐ = Requires debug build. No shortcut around it, bestie, that's just how it is.

• zfs_btree_verify_intensity=0 (uint) — turns on btree verification. These settings are cumulative:
  • 1 — Verify height.
  • 2 — Verify pointers from children to parent.
  • 3 — Verify element counts.
  • 4 — Verify element order. (expensive) 💀
  • 5 — Verify unused memory is poisoned. (expensive) ⭐

⭐ = Requires debug build.

• zfs_free_leak_on_eio=0|1 (int) — okay this one's a whole situation, sit down 🍓. If destroy hits an EIO while reading metadata (e.g. indirect blocks), the space referenced by that missing metadata can not be freed. Normally that makes the background destroy become "stalled", since it can't make forward progress. And while it's stalled, all the remaining space to free from the error-encountering filesystem is "temporarily leaked". Set this flag and instead it ignores the EIO, permanently leaks the space from indirect blocks that can not be read, and keeps on freeing everything else that it can.

The default "stalling" behavior is genuinely useful if the storage partially fails (i.e. some but not all I/O operations fail) and then later recovers — in that case we'll be able to continue pool operations while it's partially failed, and when it recovers, we can continue to free the space, with no leaks. Note, however, that this case is actually fairly rare.

Typically pools either

1. fail completely (but perhaps temporarily, e.g. due to a top-level vdev going offline), or
2. have localized, permanent errors (e.g. disk returns the wrong data due to bit flip or firmware bug).

In the former case, this setting doesn't matter because the pool will be suspended and the sync thread won't be able to make forward progress regardless. In the latter, because the error is permanent, the best we can do is leak the minimum amount of space, which is exactly what setting this flag does. So it's totally reasonable for this flag to normally be set — but they chose the more conservative approach of not setting it, so that there's no possibility of leaking space in the "partial temporary" failure case. 🙈

• zfs_free_min_time_ms=500ms (1s) (uint) — during a zfs destroy operation using the async_destroy feature, a minimum of this much time will be spent working on freeing blocks per TXG.
• zfs_obsolete_min_time_ms=500ms (uint) — similar to zfs_free_min_time_ms, but for cleanup of old indirection records for removed vdevs. 🧚
• zfs_immediate_write_sz=32768B (32 KiB) (s64) — biggest write size that gets stashed directly into the ZIL, but only when logbias=latency 💅. Anything bigger than this might get written indirectly instead, kinda like what logbias=throughput does. BUT if there's an SLOG around? this whole param is ignored, acting exactly as if it were set to infinity — every single written byte goes into the ZIL so we're not at the mercy of regular vdev latency ✨.
• zil_special_is_slog=1|0 (int) — turn this on and, whenever written blocks are headed to normal vdevs, any special vdevs you have get treated like SLOGs 🫶. Blocks that actually do go to the special vdevs are still written indirectly, same as with logbias=throughput. Also: ignored entirely if an SLOG is present, bestie.
• zfs_import_defer_txgs=5 (uint) — how many transaction groups to wait after a pool import before kicking off background work 👑. That means stuff like asynchronous block freeing (from snapshots, clones, and deduplication) plus scrub or resilver operations. The point is letting pool import and filesystem mounting finish fast without background chaos interrupting. The default of 5 TXGs is generally enough time for import + mount to wrap up on most systems 💖.
• zfs_initialize_value=16045690984833335022 (0xDEADBEEFDEADBEEE) (u64) — the pattern that zpool-initialize(8) writes into vdev free space 🌸.
• zfs_initialize_chunk_size=1048576B (1 MiB) (u64) — the size of writes that zpool-initialize(8) uses. This one's for the test suite, iykyk 🧚.
• zfs_livelist_max_entries=500000 (5*10^5) (u64) — threshold size (measured in block pointers) where we spawn a new sub-livelist. Bigger sublists cost you more memory, but the fewer sublists there are, the cheaper insertion gets 💀.
• zfs_livelist_min_percent_shared=75% (int) — if the shared space between a snapshot and its clone drops below this, the clone switches the livelist off and goes back to the old deletion method 🙈. Why? because once a clone's been overwritten enough, livelists stop giving us any benefit, periodt.
• zfs_livelist_condense_new_alloc=0 (int) — bumped up every time an extra ALLOC blkptr gets added to a livelist entry while it's being condensed. Test suite uses this to track race conditions 👀.
• zfs_livelist_condense_sync_cancel=0 (int) — bumped up every time livelist condensing gets canceled while in spa_livelist_condense_sync(). Again: test suite, race conditions ⭐.
• zfs_livelist_condense_sync_pause=0|1 (int) — set it and the livelist condense process pauses indefinitely right before running the synctask — that's spa_livelist_condense_sync(). Test suite uses this to trigger race conditions 🔥.
• zfs_livelist_condense_zthr_cancel=0 (int) — incremented every time livelist condensing is canceled while in spa_livelist_condense_cb(). Test suite, race conditions, you know the drill 💜.
• zfs_livelist_condense_zthr_pause=0|1 (int) — set it and the livelist condense process pauses indefinitely before executing the open context condensing work inside spa_livelist_condense_cb(). Test suite uses it to trigger race conditions ✨.
• zfs_lua_max_instrlimit=100000000 (10^8) (u64) — the max execution time limit you can set for a ZFS channel program, expressed as a number of Lua instructions 🎀.
• zfs_lua_max_memlimit=104857600 (100 MiB) (u64) — the max memory limit you can set for a ZFS channel program, in bytes 💖.
• zfs_max_dataset_nesting=50 (int) — deepest you can nest datasets. You can tune this temporarily to fix existing datasets that have blown past the predefined limit 🦋.
• zfs_max_log_walking=5 (u64) — how many past TXGs the flushing algorithm of the log spacemap feature looks at to estimate incoming log blocks 🌈.
• zfs_max_logsm_summary_length=10 (u64) — max number of rows allowed in the spacemap log summary 👑.
• zfs_max_recordsize=16777216 (16 MiB) (uint) — right now we support block sizes from 512 (512 B) all the way up to 16777216 (16 MiB) 🌸. But like… the perks of bigger blocks and therefore bigger I/O have to be weighed against the cost of COWing a giant block just to change one byte 😭. Also, very large blocks can mess with I/O latency and potentially the memory allocator too. So back in the day we straight-up forbade creating blocks bigger than 1M. You could make larger blocks by changing this, and pools with larger blocks can always be imported and used no matter what this setting says 💅.

Note that on x86_32 it's still limited to 1 MiB by default, because Linux's 3/1 memory split doesn't leave much room for 16M chunks 🤏.

• zfs_allow_redacted_dataset_mount=0|1 (int) — lets datasets received with redacted send/receive be mounted. Normally disabled because those datasets might be missing key data 👀.
• zfs_min_metaslabs_to_flush=1 (u64) — minimum number of metaslabs to flush per dirty TXG 🧚.
• zfs_metaslab_fragmentation_threshold=77% (uint) — lets metaslabs keep their active state as long as their fragmentation percentage stays at or below this value. Once an active metaslab exceeds it, it loses that active status so better metaslabs can get picked ⭐.
• zfs_mg_fragmentation_threshold=95% (uint) — a metaslab group is considered eligible for allocations if its fragmentation metric (as a percentage) is less than or equal to this. If a metaslab group goes over the threshold, it gets skipped — unless every metaslab group in the metaslab class has also crossed it 💀.
• zfs_mg_noalloc_threshold=0% (uint) — sets the threshold at which metaslab groups should be eligible for allocations, expressed as a percentage of free space beyond which a metaslab group is always eligible 🫶. If a group's free space is less than or equal to the threshold, the allocator avoids allocating to that group — unless all groups in the pool have hit the threshold. Once all of them have, all of them are allowed to accept allocations. The default value of 0 disables the feature and makes all metaslab groups eligible 🔥.

This parameter is how you deal with pools that have heavily imbalanced vdevs, like when a new vdev just got added. Setting the threshold to a non-zero percentage stops allocations from going to vdevs that aren't filled to that percentage, letting the lesser-filled vdevs grab more allocations than they would under the old zfs_mg_alloc_failures facility 💖.

• zfs_ddt_data_is_special=1|0 (int) — enable it and ZFS puts DDT data into the special allocation class ✨.
• zfs_user_indirect_is_special=1|0 (int) — enable it and ZFS puts user data indirect blocks into the special allocation class 🎀.
• zfs_multihost_history=0 (uint) — historical stats for this many of the latest multihost updates will be available in /proc/spl/kstat/zfs/⟨pool⟩/multihost 📊.
• zfs_multihost_interval=1000ms (1 s) (u64) — controls how often multihost writes happen when the multihost pool property is on. It's also one of the factors deciding the length of the activity check during import 🌈.

The multihost write period is zfs_multihost_interval / leaf-vdevs. On average, a multihost write gets issued for each leaf vdev every zfs_multihost_interval milliseconds. In practice the observed period can shift with I/O load, and that observed value is the delay stored in the uberblock 👑.

• zfs_multihost_import_intervals=20 (uint) — controls how long the activity test runs on import. Smaller values of zfs_multihost_import_intervals cut the import time but raise the risk of failing to detect an active pool. The total activity check time is never allowed to drop below one second 🚨.

On import, the activity check waits a minimum amount of time determined by zfs_multihost_interval × zfs_multihost_import_intervals, or the same product computed on the host that last had the pool imported — whichever is greater. The activity check time may get extended even further if the MMP delay value in the best uberblock says actual multihost updates happened at longer intervals than zfs_multihost_interval. A minimum of 100 ms is enforced 💅.

0 is equivalent to 1.

• zfs_multihost_fail_intervals=10 (uint) — controls what the pool does when multihost write failures or delays get detected 👀.

When 0, multihost write failures or delays are ignored. They still get reported to the ZED, which — depending on its configuration — may take action like suspending the pool or offlining a device.

Otherwise, the pool gets suspended if zfs_multihost_fail_intervals × zfs_multihost_interval milliseconds pass without a successful MMP write. That guarantees the activity test will see MMP writes if the pool is imported 🫶. 1 is equivalent to 2; that's necessary so the pool doesn't get suspended over normal, small I/O latency variations 😭.

• zfs_no_scrub_io=0|1 (int) — set it to disable scrub I/O. Result: scrubs don't actually scrub data and just do a metadata crawl of the pool instead 💀.
• zfs_no_scrub_prefetch=0|1 (int) — set it to disable block prefetching for scrubs 💜.
• zfs_nocacheflush=0|1 (int) — disables cache flush operations on disks when writing. > Setting this will cause pool corruption on power loss if a volatile out-of-order write cache is enabled. No cap, do not play with this one 🙈.
• zfs_nopwrite_enabled=1|0 (int) — allows no-operation writes. Whether nopwrites actually happen also depends on other pool properties, i.a. the checksumming and compression algorithms ✨.

zfs_dmu_offset_next_sync=1|0 (int)

Bestie, this one is the "pls actually find my holes" switch 💅. Turn it on and ZFS is forced to sync data whenever the SEEK_HOLE or SEEK_DATA flags get used, which means holes in a file get reported accurately. Leave it off and holes just won't show up in files that were dirtied recently. That's it, that's the whole vibe.

zfs_pd_bytes_max=52428800B (50 MiB) (int)

How many bytes should get prefetched while you're doing a pool traversal — think zfs send or any other data-crawling situation. Lowkey the "how much do I grab ahead of time" number.

zfs_traverse_indirect_prefetch_limit=32 (uint)

The number of blocks pointed at by an indirect (non-L0) block that should be prefetched during a pool traversal, like zfs send or other data crawling ops. Yes, indirect blocks get their own prefetch budget. Iconic.

zfs_per_txg_dirty_frees_percent=30% (u64)

Controls what percentage of dirtied indirect blocks (from frees) are allowed into one TXG 🎀. Cross that threshold and any extra frees just sit there and wait for the next TXG. Set it to 0 and this throttle is fully disabled — no cap, no chill.

zfs_prefetch_disable=0|1 (int)

Kills predictive prefetch. ✨ Important though: it leaves "prescient" prefetch completely intact — that's the kind used for stuff like zfs send. And the difference matters: unlike predictive prefetch, prescient prefetch never issues I/O that ends up not being needed, so it literally cannot hurt performance. Bestie said "leave prescient alone, it did nothing wrong."

zfs_qat_checksum_disable=0|1 (int)

Disables QAT hardware acceleration for SHA256 checksums. Fun fact: you can unset this after the ZFS modules have been loaded in order to initialize the QAT hardware — as long as support is compiled in and the QAT driver is actually present. iykyk.

zfs_qat_compress_disable=0|1 (int)

Same energy, but for QAT hardware acceleration of gzip compression. May also be unset after the ZFS modules are loaded to initialize the QAT hardware, provided support is compiled in and the QAT driver is present.

zfs_qat_encrypt_disable=0|1 (int)

And the third one 🫶 — disables QAT hardware acceleration for AES-GCM encryption. Yep, you guessed it: may be unset after the ZFS modules have loaded to initialize the QAT hardware, as long as support is compiled in and the QAT driver is present.

zfs_vnops_read_chunk_size=33554432B (32 MiB) (u64)

Bytes to read per chunk. That's the whole description. Short and slaying.

zfs_read_history=0 (uint)

Set this to a number and you get historical statistics for that many of the most recent reads 📊, all available in /proc/spl/kstat/zfs/⟨pool⟩/reads. Your little read receipts, fr fr.

zfs_read_history_hits=0|1 (int)

Include cache hits in read history. Yup. So even the reads that hit cache get their moment. 🎀

zfs_rebuild_max_segment=1048576B (1 MiB) (u64)

The maximum read segment size to issue when you're sequentially resilvering a top-level vdev. Bigger isn't always it, bestie.

zfs_rebuild_scrub_enabled=1|0 (int)

When the last active sequential resilver finishes, this automatically kicks off a pool scrub 🧚 so it can verify the checksums of all the blocks that got resilvered. It's on by default and strongly recommended — like, do not turn this off, I'm begging.

zfs_rebuild_vdev_limit=67108864B (64 MiB) (u64)

The max amount of I/O that can be issued concurrently for a sequential resilver, per leaf device, given in bytes. Your per-device concurrency budget, essentially 💅.

zfs_reconstruct_indirect_combinations_max=4096 (int)

Okay so picture an indirect split block with a wild number of possible unique combinations while it's being reconstructed. If it's got more than this many, ZFS goes "nah, that's too computationally expensive to check them all" and instead only tries at most this many randomly selected combinations each time the block is accessed. The point? Every segment copy gets to participate fairly in the reconstruction when checking everything is off the table, and it stops one bad copy from being used over and over. Balanced. Fair. Iconic. ✨

zfs_recover=0|1 (int)

Set this to attempt recovery from fatal errors. ⚠️ But bestie, listen: this is a last resort move only. It typically results in leaked space, or worse. Not a normal-use thing, not a vibe, do not casually enable.

zfs_removal_ignore_errors=0|1 (int)

Ignore hard I/O errors during device removal. When it's set, if a device hits a hard I/O error mid-removal, the removal will not be canceled. Sounds convenient? It can result in a normally recoverable block becoming permanently damaged, which is why it is not recommended 🚫. Only, only, only as a last resort — when the pool can't be brought back to a healthy state before removing the device.

zfs_removal_suspend_progress=0|1 (uint)

This one exists purely so the test suite can make sure certain actions happen while a removal is mid-flight. Testing tool behavior, that's the whole job. 🧪

zfs_remove_max_segment=16777216B (16 MiB) (uint)

The largest contiguous segment ZFS will attempt to allocate when removing a device. If you're seeing performance drama from trying to allocate large blocks, consider decreasing it. Also note: the default value is also the maximum. So 16 MiB is the ceiling, periodt. 👑

zfs_resilver_disable_defer=0|1 (int)

Ignores the resilver_defer feature entirely. The consequence? Any operation that would start a resilver will immediately restart the one that's currently in progress instead of waiting its turn. No deferring, no patience.

zfs_resilver_defer_percent=10% (uint)

So if the currently running resilver's progress is below this threshold, a brand new resilver will just restart from scratch instead of being deferred until after the current one finishes — and it does this even when the resilver_defer feature is enabled. Low progress? We're starting over, bestie. 😭

zfs_resilver_min_time_ms=1500ms (uint)

Resilvers are handled by the sync thread. While it's resilvering, it'll spend at least this much time working on a resilver between TXG flushes. A little dedicated time slot for the resilver, every round.

zfs_scan_ignore_errors=0|1 (int)

If set, ZFS removes the DTL (dirty time list) once a pool scan (scrub) completes — even if there were unrepairable errors. This is intended for pool repair or recovery situations, to stop resilvering when the pool is next imported. Not a casual toggle 🫶.

zfs_scrub_after_expand=1|0 (int)

After a RAIDZ expansion completes, this automatically starts a pool scrub 🧚 to verify the checksums of all blocks that were copied during the expansion. Enabled by default and strongly recommended — same energy as the rebuild one, do not skip it.

zfs_scrub_min_time_ms=750ms (uint)

Scrubs are processed by the sync thread too. While scrubbing, it will spend at least this much time working on a scrub between TXG flushes. Scrubbing gets its time slice, every time. ⭐

zfs_scrub_error_blocks_per_txg=4096 (uint)

Error blocks to be scrubbed in one txg. Yep! The number of error blocks per txg, exactly as it sounds.

zfs_scan_checkpoint_intval=7200s (2 hour) (uint)

To preserve progress across reboots, the sequential scan algorithm periodically has to stop scanning metadata and issue all the verification I/O out to disk 💾. This tunable is what determines how often that flushing happens. Checkpoint the progress, keep the receipts.

zfs_scan_fill_weight=3 (uint)

This affects the order that scrub and resilver I/O segments get sorted in. A higher number = we care more about how filled in a segment is. A lower number = we care more about the size of the extent, without thinking about the gaps inside a segment. 📊 Heads up though: this value is only tunable upon module insertion. Change it afterwards and it will have literally no effect on scrub or resilver performance. Sorry bestie, that's just the rules.

zfs_scan_issue_strategy=0 (uint)

Determines the order data gets verified in while scrubbing or resilvering:

• 1 — Data will be verified as sequentially as possible, given the amount of memory reserved for scrubbing (see zfs_scan_mem_lim_fact). This may improve scrub performance when the pool's data is very fragmented. 🦋
• 2 — The largest mostly-contiguous chunk of found data gets verified first. By deferring the scrubbing of small segments, we might later find adjacent data to coalesce and increase the segment size.
• 0 — Use strategy 1 during normal verification, and strategy 2 while taking a checkpoint.

zfs_scan_legacy=0|1 (int)

If unset, it means scrubs and resilvers gather metadata in memory before issuing sequential I/O. Otherwise it means the legacy algorithm gets used, where I/O is kicked off as soon as it's discovered. ⚠️ Also important: unsetting this will not affect scrubs or resilvers that are already in progress. They're doing their thing, leave them.

zfs_scan_max_ext_gap=2097152B (2 MiB) (int)

Sets the largest gap (in bytes) between scrub/resilver I/O operations that still counts as sequential for sorting purposes. Same warning as above, bestie: changing this value will not affect scrubs or resilvers already in progress.

zfs_scan_mem_lim_fact=20^-1 (uint)

The maximum fraction of RAM used for I/O sorting by the sequential scan algorithm. This tunable is what sets the hard limit on I/O sorting memory usage. When we hit that hard limit, we stop scanning metadata and start issuing data verification I/O — and we keep doing that until we get back below the soft limit. 🎀

zfs_scan_mem_lim_soft_fact=20^-1 (uint)

The fraction of the hard limit that's used to determine the soft limit for I/O sorting by the sequential scan algorithm. Here's the behavior: cross this limit from below and nothing happens, no action taken. Cross it from above and that means we're currently issuing verification I/O — in that case (unless the metadata scan is done) we stop issuing verification I/O and go back to scanning metadata again, until we hit the hard limit. A little back-and-forth dance 💃.

zfs_scan_report_txgs=0|1 (uint)

When reporting resilver throughput and estimated completion time, use the performance observed over roughly the last zfs_scan_report_txgs TXGs. Set it to zero and performance instead gets calculated over the time between checkpoints. Pick your measurement window. ⏱️

zfs_scan_strict_mem_lim=0|1 (int)

Enforces tight memory limits on pool scans while a sequential scan is in progress. When disabled, the memory limit can be exceeded by fast disks 🏃♀️💨. So if your disks are speedy and shameless, they might blow right past it.

zfs_scan_suspend_progress=0|1 (int)

Freezes a scrub or resilver that's in progress without actually pausing it. Yeah. Intended for testing/debugging purposes 🧪. Weird, but it's for the testers.

zfs_scan_vdev_limit=16777216B (16 MiB) (int)

The maximum amount of data that can be issued concurrently at once for scrubs and resilvers, per leaf device, given in bytes. Per-device concurrency cap, scrub edition. 💅

zfs_send_corrupt_data=0|1 (int)

Allows sending of corrupt data — meaning read/checksum errors get ignored while sending. ⚠️ Yes, really. That's what it does.

zfs_send_unmodified_spill_blocks=1|0 (int)

Includes unmodified spill blocks in the send stream. Okay, context: under certain circumstances, previous versions of ZFS could incorrectly remove the spill block from an existing object 😭. Including unmodified copies of the spill blocks creates a backwards-compatible stream that will recreate a spill block if it was incorrectly removed. Fixing the past's oopsies with a compatibility stream. Iconic behavior, honestly. 🫶

• zfs_send_no_prefetch_queue_ff=20^-1 (uint) — okay so this is the fill fraction of the internal queues that zfs send keeps around 💌. It's basically the little dial that decides when the internal threads get woken up, timing-wise.
• zfs_send_no_prefetch_queue_length=1048576B (1 MiB) (uint) — the max number of bytes that are allowed to chill in zfs send's internal queues. Cap it, bestie ✨.
• zfs_send_queue_ff=20^-1 (uint) — the fill fraction of the zfs send prefetch queue this time (yes, a different queue, stay with me 🫶). Same deal: it controls the timing of when internal threads get woken up.
• zfs_send_queue_length=16777216B (16 MiB) (uint) — the maximum bytes that zfs send is allowed to prefetch. Heads up though: this value must be at least twice the maximum block size in use. Not optional, fr fr.
• zfs_recv_queue_ff=20^-1 (uint) — the fill fraction of the zfs receive queue, and yep, it's the same vibe: it controls the timing with which internal threads get woken up.
• zfs_recv_queue_length=16777216B (16 MiB) (uint) — max bytes allowed in the zfs receive queue. And again: this value must be at least twice the maximum block size in use 💅.
• zfs_recv_write_batch_size=1048576B (1 MiB) (uint) — the maximum amount of data, in bytes, that zfs receive will write in one DMU transaction 🎀. Important: this is the uncompressed size, even when you're receiving a compressed send stream. Also this setting will not reduce the write size below a single block, and it's capped at a maximum of 32 MiB.
• zfs_recv_best_effort_corrective=0 (int) — when you set this to non-zero, you get a corrective receive that:
  1. Does not enforce the restriction of source & destination snapshot GUIDs matching.
  2. If there's an error during healing, the healing receive is not terminated — instead it moves on to the next record. Resilient queen behavior 👑.
• zfs_override_estimate_recordsize=0|1 (uint) — setting this overrides the default logic for estimating block sizes when doing a zfs send 🦋. The default heuristic is that the average block size will be the current recordsize. So if most of the data in your dataset is not that size and you need accurate zfs send size estimates, override it.
• zfs_sync_pass_deferred_free=2 (uint) — flushing data to disk happens in passes. This is the pass where deferred frees start.
• zfs_spa_discard_memory_limit=16777216B (16 MiB) (int) — the maximum memory used for prefetching a checkpoint's space map on each vdev while discarding the checkpoint.
• zfs_spa_note_txg_time=600 (uint) — this defines, in seconds, how often the TXG time database will record a new TXG if it has changed 🌸. Once that interval passes and the TXG number has changed, the new value gets recorded in the database. These timestamps can later be used for more granular operations, like scrubbing.
• zfs_spa_flush_txg_time=600 (uint) — how often, in seconds, ZFS flushes the TXG time database to disk 💾. It makes sure the data is actually written to persistent storage, which helps preserve the database in case of an unexpected shutdown. Bonus: the database is also automatically flushed during the export sequence.
• zfs_special_class_metadata_reserve_pct=25% (uint) — only allow small data blocks to be allocated on the special and dedup vdev types when the available free space percentage on these vdevs exceeds this value ✨. Why? To ensure reserved space is available for pool metadata as the special vdevs approach capacity.
• zfs_sync_pass_dont_compress=8 (uint) — starting in this sync pass, compression (including of metadata) gets disabled. With the default setting, in practice we don't have this many sync passes, so this has no effect.

The original intent was that disabling compression would help the sync passes converge 😭. But in practice, disabling compression increases the average number of sync passes — because when compression turns off, many blocks' size will change, so we have to re-allocate them (not overwrite them). It also increases the number of 128 KiB allocations (e.g. for indirect blocks and spacemaps) since those won't be compressed. Those 128 KiB allocations are especially detrimental to performance on highly fragmented systems, which may have very few free segments of this size and may need to load new metaslabs just to satisfy these allocations. It's giving "great idea, terrible execution" 💀.

• zfs_sync_pass_rewrite=2 (uint) — rewrite new block pointers starting in this pass.
• zfs_trim_extent_bytes_max=134217728B (128 MiB) (uint) — maximum size of a TRIM command. Bigger ranges get split into chunks no larger than this value before being issued.
• zfs_trim_extent_bytes_min=32768B (32 KiB) (uint) — minimum size of TRIM commands. TRIM ranges smaller than this get skipped, unless they're part of a larger range which was chunked. The reason? It's common for these small TRIMs to negatively impact overall performance 😬.
• zfs_trim_metaslab_skip=0|1 (uint) — skip uninitialized metaslabs during the TRIM process. This option is useful for pools constructed from large thinly-provisioned devices where TRIM operations are slow 🐢. As a pool ages, an increasing fraction of the pool's metaslabs will be initialized, progressively degrading the usefulness of this option. This setting is stored when starting a manual TRIM and will persist for the duration of the requested TRIM.
• zfs_trim_queue_limit=10 (uint) — max number of queued TRIMs outstanding per leaf vdev. The number of concurrent TRIM commands issued to the device is controlled by zfs_vdev_trim_min_active and zfs_vdev_trim_max_active.
• zfs_trim_txg_batch=32 (uint) — the number of transaction groups' worth of frees that should be aggregated before TRIM operations are issued to the device 🌈. This is a whole trade-off: larger, more efficient TRIM operations vs. the delay before the recently trimmed space is available for use by the device.

Increase this value and frees get aggregated for a longer time — giving you larger TRIM operations and potentially increased memory usage. Decrease it and you get the opposite effect. The default of 32 was determined to be a reasonable compromise 🎀.

• zfs_txg_history=100 (uint) — historical statistics for this many latest TXGs will be available in /proc/spl/kstat/zfs/⟨pool⟩/TXGs 📊.
• zfs_txg_timeout=5s (uint) — flush dirty data to disk at least every this many seconds (this is the maximum TXG duration, bestie ⏱️).
• zfs_vdev_aggregation_limit=1048576B (1 MiB) (uint) — max vdev I/O aggregation size.
• zfs_vdev_aggregation_limit_non_rotating=131072B (128 KiB) (uint) — max vdev I/O aggregation size for non-rotating media 💫.
• zfs_vdev_mirror_rotating_inc=0 (int) — a number by which the balancing algorithm increments the load calculation, for the purpose of selecting the least busy mirror member when an I/O operation immediately follows its predecessor on rotational vdevs, so it can make decisions based on load.
• zfs_vdev_mirror_rotating_seek_inc=5 (int) — a number by which the balancing algorithm increments the load calculation, again for picking the least busy mirror member, when an I/O operation lacks locality as defined by zfs_vdev_mirror_rotating_seek_offset 🧭. Operations within this that do not immediately follow the previous operation get incremented by half.
• zfs_vdev_mirror_rotating_seek_offset=1048576B (1 MiB) (int) — the maximum distance from the last queued I/O operation in which the balancing algorithm considers an operation to have locality. See “ZFS I/O SCHEDULER” 📚.
• zfs_vdev_mirror_non_rotating_inc=0 (int) — a number by which the balancing algorithm increments the load calculation for the purpose of selecting the least busy mirror member on non-rotational vdevs, when I/O operations do not immediately follow one another.
• zfs_vdev_mirror_non_rotating_seek_inc=1 (int) — a number by which the balancing algorithm increments the load calculation for selecting the least busy mirror member when an I/O operation lacks locality as defined by zfs_vdev_mirror_rotating_seek_offset 🦋. Operations within this that are not immediately following the previous operation are incremented by half.
• zfs_vdev_read_gap_limit=32768B (32 KiB) (uint) — aggregate read I/O operations if the on-disk gap between them is within this threshold.
• zfs_vdev_write_gap_limit=4096B (4 KiB) (uint) — aggregate write I/O operations if the on-disk gap between them is within this threshold.
• zfs_vdev_raidz_impl=fastest (string) — pick the raidz parity implementation to use ⭐.

Variants that don't depend on CPU-specific features may be selected on module load, since they're supported on all systems. The remaining options may only be set after the module is loaded, because they're available only if the implementations are compiled in and supported on the running system.

Once the module is loaded, /sys/module/zfs/parameters/zfs_vdev_raidz_impl will show the available options, with the currently selected one enclosed in square brackets 👀.

• fastest — selected by built-in benchmark
• original — original implementation
• scalar — scalar implementation
• sse2 — SSE2 instruction set, 64-bit x86
• ssse3 — SSSE3 instruction set, 64-bit x86
• avx2 — AVX2 instruction set, 64-bit x86
• avx512f — AVX512F instruction set, 64-bit x86
• avx512bw — AVX512F & AVX512BW instruction sets, 64-bit x86
• aarch64_neon — NEON, Aarch64/64-bit ARMv8
• aarch64_neonx2 — NEON with more unrolling, Aarch64/64-bit ARMv8
• powerpc_altivec — Altivec, PowerPC

• zfs_zevent_len_max=512 (uint) — max event queue length 📬. Events in the queue can be viewed with zpool-events(8).

🫧 zfs_zevent_retain_max=2000 (int)

How many recent zevent records we're willing to hang onto so we can spot duplicates, bestie. Default is 2000. And if you set it to 0? Duplicate detection is simply disabled. No checking, no dedup, gone. 💀

⏳ zfs_zevent_retain_expire_secs=900s (15 min) (int)

The lifespan for a recent ereport that got retained for duplicate checking. Fifteen minutes is the default attention span. ✨

🧹 zfs_zil_clean_taskq_maxalloc=1048576 (int)

The maximum number of taskq entries allowed to be cached. Once you go over this limit, transaction records (itxs) get cleaned synchronously. No more lounging in the cache, bestie, you're doing it live. 💅

🌱 zfs_zil_clean_taskq_minalloc=1024 (int)

How many taskq entries get pre-populated the moment the taskq is first created — already sitting there, immediately ready for use. No waiting around, they're born ready. 🎀

🧵 zfs_zil_clean_taskq_nthr_pct=100% (int)

This one controls how many threads dp_zil_clean_taskq is allowed to use. The default of 100% means a maximum of one thread per CPU. Iconic, generous, very main character. 👑

📦 zil_maxblocksize=131072B (128 KiB) (uint)

Sets the maximum block size the ZIL will use. And here's the tea: on very fragmented pools, lowering this — typically down to 36 KiB — can improve performance. Yes really. 💖

✍️ zil_maxcopied=7680B (7.5 KiB) (uint)

The maximum number of write bytes logged via WR_COPIED. It tunes a whole little tradeoff: extra memory copy and possibly worse log space efficiency on one side, extra range lock/unlock on the other. Pick your poison, gorgeous. ⚖️

🚫 zil_nocacheflush=0|1 (int)

Disables the cache flush commands that the ZIL normally sends to disk after an LWB write has completed.

Not playing around here: setting this will cause ZIL corruption on power loss if a volatile out-of-order write cache is enabled. That's not a vibe, that's a data loss situation. 😭

🙈 zil_replay_disable=0|1 (int)

Disables intent logging replay. You can flip this on for recovery from a corrupted ZIL — it's basically the "this thing is broken, let me in anyway" lever. 🔧

🐘 zil_slog_bulk=67108864B (64 MiB) (u64)

Limits the SLOG write size per commit executed with synchronous priority. Anything above that gets executed with lower (asynchronous) priority, so a single active ZIL writer can't just absolutely body your SLOG device. 🛑

🐣 zfs_zil_saxattr=1|0 (int)

Set this to zero and you disable ZIL logging of new xattr=sa records — but only if the org.openzfs:zilsaxattr feature is enabled on the pool. Realistically you'd only do this to work around bugs in the ZIL logging or replay code for this record type. And if the feature is disabled? The tunable does literally nothing. No effect, bestie. 🦋

🧲 zfs_embedded_slog_min_ms=64 (uint)

So usually, one metaslab from each normal and special class vdev gets dedicated to the ZIL for logging synchronous writes. But — if there are fewer than zfs_embedded_slog_min_ms metaslabs in the vdev, that functionality is disabled entirely. Why? Because we are not about to set aside an unreasonable amount of space for the ZIL. Boundaries. 💅

🗜️ zstd_earlyabort_pass=1 (uint)

Whether the heuristic for detecting incompressible data is enabled — specifically for zstd levels >= 3 using LZ4 and zstd-1 passes. Incompressible data, we see you, and we're not wasting time on you. 👀

📏 zstd_abort_size=131072 (uint)

The minimal uncompressed size (inclusive) of a record before that early abort heuristic will even be attempted. Small records don't qualify, sorry babe. 🌸

💀 zio_deadman_log_all=0|1 (int)

If this is non-zero, the zio deadman produces debugging messages (see zfs_dbgmsg_enable) for all zios, instead of only for leaf zios that possess a vdev. This is a developer thing — it's meant for gaining diagnostic information on hang conditions that don't involve a mutex or other locking primitive, typically the kind where a thread in the zio pipeline is looping indefinitely. Forever. Just spinning. 😵‍💫

🐌 zio_slow_io_ms=30000ms (30 s) (int)

When an I/O operation takes longer than this to complete, it gets marked as slow. Every single slow operation causes a delay zevent. You can peep your slow I/O counters with zpool status -s. Snitch on your own disks, honestly. 🕵️

🚦 zio_dva_throttle_enabled=1|0 (int)

Throttles block allocations in the I/O pipeline. This is what lets allocation get distributed dynamically based on how each device is actually performing. Democracy but for vdevs. ✨

🏷️ zfs_xattr_compat=0|1 (int)

This one controls the naming scheme used when setting new xattrs in the user namespace. Here's the breakdown:

• If 0 (the default on Linux), user namespace xattr names get prefixed with the namespace, so it stays backwards compatible with previous versions of ZFS on Linux.
• If 1 (the default on FreeBSD), user namespace xattr names are not prefixed, so it stays backwards compatible with previous versions of ZFS on illumos and FreeBSD.

Either naming scheme can be read on this and future versions of ZFS, regardless of the tunable. But — legacy ZFS on illumos or FreeBSD is unable to read user namespace xattrs written in the Linux format, and legacy versions of ZFS on Linux are unable to read user namespace xattrs written in the legacy ZFS format. Awkward little standoff, ngl. 😭

Also, an existing xattr with the alternate naming scheme gets removed when overwriting the xattr, so we don't just accumulate duplicates forever. Clean girl behavior. 🧼

⏭️ zio_requeue_io_start_cut_in_line=0|1 (int)

Prioritize requeued I/O. That's it. That's the whole thing. Requeued I/O skips the line. 🎟️

🗂️ zfs_delete_inode=0|1 (int)

Sets whether the kernel should free an inode structure when the last reference is released, or cache it in memory. Intended for testing/debugging, fyi.

A live inode structure "pins" various internal OpenZFS structures in memory, and that can result in large amounts of "unusable" memory on systems with lots of infrequently-accessed files — right up until the kernel's memory pressure mechanism asks OpenZFS to release them.

The default value of 0 always caches inodes that appear to still exist on disk. Setting it to 1 will immediately release unused inodes and their associated memory back to the dbuf cache or the ARC for reuse, but it may reduce performance if inodes are frequently evicted and reloaded. Pick your struggle. 💅

This parameter is only available on Linux.

📁 zfs_delete_dentry=0|1 (int)

Sets whether the kernel should free a dentry structure when it's no longer required, or hold it in the dentry cache. Intended for testing/debugging. And since a dentry structure holds an inode reference, a cached dentry can "pin" an inode in memory indefinitely, along with the associated OpenZFS structures (See zfs_delete_inode). Yes, they're a duo. 🫶

The default value of 0 instructs the kernel to cache entries and their associated inodes when they are no longer directly referenced. They'll get reclaimed as part of the kernel's normal cache management processes. Setting it to 1 instructs the kernel to release directory entries and their inodes as soon as they are no longer referenced by the filesystem. Snappy, but at a cost. ⚡

This parameter is only available on Linux.

🍰 zio_taskq_batch_pct=80% (uint)

The percentage of online CPUs that will run a worker thread for I/O. These workers are responsible for I/O work like compression, encryption, checksum and parity calculations. Fractional number of CPUs gets rounded down, no half-threads here.

The default of 80% was chosen to avoid using all the CPUs — because doing that can result in latency issues and inconsistent application performance, especially when slower compression and/or checksumming is enabled. Leave some CPU for the rest of us. 💖

Set value only applies to pools imported/created after that.

🧮 zio_taskq_batch_tpq=0 (uint)

Number of worker threads per taskq. Higher values improve I/O ordering and CPU utilization, while lower values reduce lock contention. Set value only applies to pools imported/created after that.

If 0, a system-dependent value close to 6 threads per taskq gets generated. And one more time for the people in the back: set value only applies to pools imported/created after that. 🎤

✏️ zio_taskq_write_tpq=16 (uint)

Determines the minimum number of threads per write issue taskq. Higher values improve CPU utilization on high throughput, while lower values reduce taskq locks contention on high IOPS. Set value only applies to pools imported/created after that. 🦋

📖 zio_taskq_read=fixed,1,8 null scale null (charp)

Sets the queue and thread configuration for the IO read queues. This is an advanced debugging parameter — don't change this unless you actually understand what it does. I'm serious, bestie, put it down. 🙈

Each of the four values corresponds to the issue, issue high-priority, interrupt, and interrupt high-priority queues. Valid values are:

• fixed,N,M — M queues with N threads each
• scale[,MIN] — scale with CPUs, minimum MIN total threads
• sync
• null

Set values only apply to pools imported/created after that. ✨

🖊️ zio_taskq_write=sync null scale null (charp)

Sets the queue and thread configuration for the IO write queues. Also an advanced debugging parameter, also not for casual fiddling — don't change this unless you understand what it does. 💀

Each of the four values corresponds to the issue, issue high-priority, interrupt, and interrupt high-priority queues. Valid values are:

• fixed,N,M — M queues with N threads each
• scale[,MIN] — scale with CPUs, minimum MIN total threads
• sync
• null

Set values only apply to pools imported/created after that. 🌈

🗑️ zio_taskq_free=scale,32 null null null (charp)

Sets the queue and thread configuration for the IO free queues. Yep, another advanced debugging parameter — don't change this unless you understand what it does, periodt.

Each of the four values corresponds to the issue, issue high-priority, interrupt, and interrupt high-priority queues. Valid values are:

• fixed,N,M — M queues with N threads each
• scale[,MIN] — scale with CPUs, minimum MIN total threads
• sync
• null

The default uses a minimum of 32 threads to improve parallelism for DDT and BRT metadata operations during frees. Set values only apply to pools imported/created after that. 🧚

🌫️ zvol_inhibit_dev=0|1 (uint)

Do not create zvol device nodes. Why would you want that? Because it may slightly improve startup time on systems with a very large number of zvols. Sometimes less is more, gorgeous. 💜

🔢 zvol_major=230 (uint)

The major number for zvol block devices. 230 is the vibe. ⭐

✂️ zvol_max_discard_blocks=16384 (long)

Discard (TRIM) operations done on zvols will be done in batches of this many blocks — where the block size is determined by the volblocksize property of a zvol. Batch it, don't dribble it. 🎀

• zvol_prefetch_bytes = 131072B (128 KiB) (uint) — when a zvol gets added to the system, we prefetch this many bytes from both the start and the end of the volume. Why bestie? Because those regions are likely to be accessed immediately by blkid(8) or by the kernel partitioner, so warming them up first is just 👩‍🍳💋 smart.
• zvol_request_sync = 0|1 (uint) — when we're processing I/O requests for a zvol, submit them synchronously. This effectively caps the queue depth at 1 for each individual I/O submitter. When it's unset, requests get handled asynchronously by a thread pool instead, and the number of requests that can be handled concurrently is controlled by zvol_threads. Heads up: this parameter applies regardless of the zvol_use_blk_mq setting. No cap, it just does its thing ✨
• zvol_num_taskqs = 0 (uint) — the number of zvol taskqs. If it's 0 (which is the default, periodt) then the scaling is done internally, preferring 6 threads per taskq. This only applies on Linux 🐧
• zvol_threads = 0 (uint) — how many system wide threads to use for processing zvol block IOs. If it's 0 (default, again) then internally zvol_threads gets set to the number of CPUs present, or 32 — whichever is greater. Math is math 💅
• zvol_blk_mq_threads = 0 (uint) — the number of threads per zvol used for queuing IO requests. This parameter only shows up if your kernel supports blk-mq, and it's only read and assigned to a zvol at zvol load time. If 0 (default), internally zvol_blk_mq_threads gets set to the number of CPUs present 🧠
• zvol_use_blk_mq = 0|1 (uint) — set it to 1 to use the blk-mq API for zvols. Set it to 0 (the default) to use the legacy zvol APIs instead. Lowkey this setting can give you better or worse zvol performance depending on the workload, so it's giving "it depends". This parameter only appears if your kernel supports blk-mq, and it's only read and assigned to a zvol at zvol load time.
• zvol_blk_mq_blocks_per_thread = 8 (uint) — if zvol_use_blk_mq is enabled, then process this many volblocksize-sized blocks per zvol thread. You can use this tunable to favor better performance for zvol reads (lower values) or writes (higher values). If it's set to 0, then the zvol layer will process the maximum number of blocks per thread that it can. This parameter only appears if your kernel supports blk-mq, and it's only applied at each zvol's load time.
• zvol_blk_mq_queue_depth = 0 (uint) — the queue_depth value for the zvol blk-mq interface. This parameter only appears if your kernel supports blk-mq, and it's only applied at each zvol's load time. If 0 (the default), then it uses the kernel's default queue depth. Values get clamped to the kernel's BLKDEV_MIN_RQ and BLKDEV_MAX_RQ/BLKDEV_DEFAULT_RQ limits — no escaping those boundaries bestie 🚧
• zvol_volmode = 1 (uint) — defines zvol block device behavior when volmode=default:
  • 1 — equivalent to full
  • 2 — equivalent to dev
  • 3 — equivalent to none
• zvol_enforce_quotas = 0|1 (uint) — enable strict ZVOL quota enforcement. Friendly lil warning: the strict quota enforcement may have a performance impact, so know that going in 🫶

---

🔥 ZFS I/O SCHEDULER 🔥

Okay so ZFS issues I/O operations to the leaf vdevs to satisfy and complete I/O operations, and the scheduler is the one deciding when and in what order those operations get issued. Not me hyping up a scheduler but it's kind of the main character here 👑

The scheduler divides operations into five I/O classes, prioritized in this order:

1. sync read
2. sync write
3. async read
4. async write
5. scrub/resilver

Every queue defines the minimum and maximum number of concurrent operations that may be issued to the device. On top of that, the device has an aggregate maximum, zfs_vdev_max_active. And here's the catch, bestie: the sum of the per-queue minima must not exceed the aggregate maximum. But if the sum of the per-queue maxima exceeds the aggregate maximum, then the number of active operations may reach zfs_vdev_max_active, and once that happens, no further operations will be issued — regardless of whether all per-queue minima have been met. It's giving "the math said no" 💀

For many physical devices, throughput increases with the number of concurrent operations, but latency typically suffers 😭 And more than that, physical devices typically have a limit at which more concurrent operations have no effect on throughput, or can actually cause it to decrease. So more isn't always more, no cap.

The scheduler picks the next operation to issue by first looking for an I/O class whose minimum has not been satisfied. Once all of them are satisfied and the aggregate maximum hasn't been hit, the scheduler looks for classes whose maximum hasn't been satisfied. Iteration through the I/O classes happens in the order specified above, every single time. And no further operations are issued if the aggregate maximum number of concurrent operations has been hit, or if there are no operations queued for an I/O class that hasn't hit its maximum. Every time an I/O operation is queued, or an operation completes, the scheduler looks for new operations to issue. It's literally always checking ✨

In general:

• smaller max_actives will lead to lower latency of synchronous operations
• larger max_actives may lead to higher overall throughput, depending on underlying storage

The ratio of the queues' max_actives determines the balance of performance between reads, writes, and scrubs. For example, increasing zfs_vdev_scrub_max_active will cause the scrub or resilver to complete more quickly — but reads and writes will have higher latency and lower throughput. Trade-offs are real 🙈

All I/O classes have a fixed maximum number of outstanding operations, except for the async write class. Asynchronous writes represent the data that is committed to stable storage during the syncing stage for transaction groups. Transaction groups enter the syncing state periodically, so the number of queued async writes will quickly burst up and then bleed down to zero. Rather than servicing them as quickly as possible, the I/O scheduler changes the maximum number of active async write operations according to the amount of dirty data in the pool. Since both throughput and latency typically increase with the number of concurrent operations issued to physical devices, reducing the burstiness in the number of simultaneous operations also stabilizes the response time of operations from other queues — in particular the synchronous ones. In broad strokes: the I/O scheduler will issue more concurrent operations from the async write queue as there is more dirty data in the pool. 🌈

---

💖 Async Writes 💖

The number of concurrent operations issued for the async write I/O class follows a piece-wise linear function defined by a few adjustable points:

              |              o---------| <-- zfs_vdev_async_write_max_active
         ^    |             /^         |
         |    |            / |         |
       active |           /  |         |
        I/O   |          /   |         |
       count  |         /    |         |
              |        /     |         |
              |-------o      |         | <-- zfs_vdev_async_write_min_active
             0|_______^______|_________|
              0%      |      |       100% of zfs_dirty_data_max
                      |      |
                      |      `-- zfs_vdev_async_write_active_max_dirty_percent
                      `--------- zfs_vdev_async_write_active_min_dirty_percent

Until the amount of dirty data exceeds a minimum percentage of the dirty data allowed in the pool, the I/O scheduler will limit the number of concurrent operations to the minimum. As that threshold gets crossed, the number of concurrent operations issued increases linearly up to the maximum, at the specified maximum percentage of the dirty data allowed in the pool. Obsessed with a clean slope honestly 📈

Ideally, the amount of dirty data on a busy pool will stay in the sloped part of the function, between zfs_vdev_async_write_active_min_dirty_percent and zfs_vdev_async_write_active_max_dirty_percent. If it exceeds the maximum percentage, that's a signal that the rate of incoming data is greater than the rate that the backend storage can handle. And when that happens, we must further throttle incoming writes, as described in the next section 👀

✨ ZFS TRANSACTION DELAY ✨

ok so bestie, this is the part where ZFS is like "girl, slow down" 💅 We delay transactions when we've determined that the backend storage isn't able to accommodate the rate of incoming writes. Like, the disk is struggling and we're being a considerate friend about it.

If there is already a transaction waiting, we delay relative to when that transaction will finish waiting. This way the calculated delay time is independent of the number of threads concurrently executing transactions. no cap, that's the whole point — the math stays cute no matter how many threads are in the group chat 🫶

If we are the only waiter, wait relative to when the transaction started, rather than the current time. This credits the transaction for "time already served", e.g. reading indirect blocks. it's giving "you already did the work, so here's your credit" 💖

The minimum time for a transaction to take is calculated as

min_time = min(zfs_delay_scale × (dirty - min) / (max - dirty), 100ms)

The delay has two degrees of freedom that can be adjusted via tunables:

• zfs_delay_min_dirty_percent — the percentage of dirty data at which we start to delay. This should typically be at or above zfs_vdev_async_write_active_max_dirty_percent, so that we only start to delay after writing at full speed has failed to keep up with the incoming write rate. literally the "ok we tried our best, now we throttle" line 🌸
• zfs_delay_scale — the scale of the curve. Roughly speaking, this variable determines the amount of delay at the midpoint of the curve. she's the one shaping the whole vibe ✨

and here's the curve, in all her main character glory:

delay
 10ms +-------------------------------------------------------------*+
      |                                                             *|
  9ms +                                                             *+
      |                                                             *|
  8ms +                                                             *+
      |                                                            * |
  7ms +                                                            * +
      |                                                            * |
  6ms +                                                            * +
      |                                                            * |
  5ms +                                                           *  +
      |                                                           *  |
  4ms +                                                           *  +
      |                                                           *  |
  3ms +                                                          *   +
      |                                                          *   |
  2ms +                                              (midpoint) *    +
      |                                                  |    **     |
  1ms +                                                  v ***       +
      |             zfs_delay_scale ---------->     ********         |
    0 +-------------------------------------*********----------------+
      0%                    <- zfs_dirty_data_max ->               100%

Note, that since the delay is added to the outstanding time remaining on the most recent transaction it's effectively the inverse of IOPS. Here, the midpoint of 500 us translates to 2000 IOPS 🔥 The shape of the curve was chosen such that small changes in the amount of accumulated dirty data in the first three quarters of the curve yield relatively small differences in the amount of delay. lowkey genius behavior tbh.

The effects can be easier to understand when the amount of delay is represented on a logarithmic scale:

delay
100ms +-------------------------------------------------------------++
      +                                                              +
      |                                                              |
      +                                                             *+
 10ms +                                                             *+
      +                                                           ** +
      |                                              (midpoint)  **  |
      +                                                  |     **    +
  1ms +                                                  v ****      +
      +             zfs_delay_scale ---------->        *****         +
      |                                             ****             |
      +                                          ****                +
100us +                                        **                    +
      +                                       *                      +
      |                                      *                       |
      +                                     *                        +
 10us +                                     *                        +
      +                                                              +
      |                                                              +
      +                                                              +
      +--------------------------------------------------------------+
      0%                    <- zfs_dirty_data_max ->               100%

Note here that only as the amount of dirty data approaches its limit does the delay start to increase rapidly. 👀 The goal of a properly tuned system should be to keep the amount of dirty data out of that range by first ensuring that the appropriate limits are set for the I/O scheduler to reach optimal throughput on the back-end storage, and then by changing the value of zfs_delay_scale to increase the steepness of the curve. it's giving "set the limits first, then fine-tune the drama" 💅

---

and that's a wrap on the whole page, bestie!! 🎀 this lovely manual page comes to us from OpenZFS, dated September 15, 2025 — go forth and tune your pools responsibly, mwah 💖✨🦋