1.\" 2.\" Copyright (c) 2013 by Turbo Fredriksson <turbo@bayour.com>. All rights reserved. 3.\" Copyright (c) 2019, 2021 by Delphix. All rights reserved. 4.\" Copyright (c) 2019 Datto Inc. 5.\" The contents of this file are subject to the terms of the Common Development 6.\" and Distribution License (the "License"). You may not use this file except 7.\" in compliance with the License. You can obtain a copy of the license at 8.\" usr/src/OPENSOLARIS.LICENSE or https://opensource.org/licenses/CDDL-1.0. 9.\" 10.\" See the License for the specific language governing permissions and 11.\" limitations under the License. When distributing Covered Code, include this 12.\" CDDL HEADER in each file and include the License file at 13.\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this 14.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your 15.\" own identifying information: 16.\" Portions Copyright [yyyy] [name of copyright owner] 17.\" 18.Dd January 10, 2023 19.Dt ZFS 4 20.Os 21. 22.Sh NAME 23.Nm zfs 24.Nd tuning of the ZFS kernel module 25. 26.Sh DESCRIPTION 27The ZFS module supports these parameters: 28.Bl -tag -width Ds 29.It Sy dbuf_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64 30Maximum size in bytes of the dbuf cache. 31The target size is determined by the MIN versus 32.No 1/2^ Ns Sy dbuf_cache_shift Pq 1/32nd 33of the target ARC size. 34The behavior of the dbuf cache and its associated settings 35can be observed via the 36.Pa /proc/spl/kstat/zfs/dbufstats 37kstat. 38. 39.It Sy dbuf_metadata_cache_max_bytes Ns = Ns Sy UINT64_MAX Ns B Pq u64 40Maximum size in bytes of the metadata dbuf cache. 41The target size is determined by the MIN versus 42.No 1/2^ Ns Sy dbuf_metadata_cache_shift Pq 1/64th 43of the target ARC size. 44The behavior of the metadata dbuf cache and its associated settings 45can be observed via the 46.Pa /proc/spl/kstat/zfs/dbufstats 47kstat. 48. 49.It Sy dbuf_cache_hiwater_pct Ns = Ns Sy 10 Ns % Pq uint 50The percentage over 51.Sy dbuf_cache_max_bytes 52when dbufs must be evicted directly. 53. 54.It Sy dbuf_cache_lowater_pct Ns = Ns Sy 10 Ns % Pq uint 55The percentage below 56.Sy dbuf_cache_max_bytes 57when the evict thread stops evicting dbufs. 58. 59.It Sy dbuf_cache_shift Ns = Ns Sy 5 Pq uint 60Set the size of the dbuf cache 61.Pq Sy dbuf_cache_max_bytes 62to a log2 fraction of the target ARC size. 63. 64.It Sy dbuf_metadata_cache_shift Ns = Ns Sy 6 Pq uint 65Set the size of the dbuf metadata cache 66.Pq Sy dbuf_metadata_cache_max_bytes 67to a log2 fraction of the target ARC size. 68. 69.It Sy dbuf_mutex_cache_shift Ns = Ns Sy 0 Pq uint 70Set the size of the mutex array for the dbuf cache. 71When set to 72.Sy 0 73the array is dynamically sized based on total system memory. 74. 75.It Sy dmu_object_alloc_chunk_shift Ns = Ns Sy 7 Po 128 Pc Pq uint 76dnode slots allocated in a single operation as a power of 2. 77The default value minimizes lock contention for the bulk operation performed. 78. 79.It Sy dmu_prefetch_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint 80Limit the amount we can prefetch with one call to this amount in bytes. 81This helps to limit the amount of memory that can be used by prefetching. 82. 83.It Sy ignore_hole_birth Pq int 84Alias for 85.Sy send_holes_without_birth_time . 86. 87.It Sy l2arc_feed_again Ns = Ns Sy 1 Ns | Ns 0 Pq int 88Turbo L2ARC warm-up. 89When the L2ARC is cold the fill interval will be set as fast as possible. 90. 91.It Sy l2arc_feed_min_ms Ns = Ns Sy 200 Pq u64 92Min feed interval in milliseconds. 93Requires 94.Sy l2arc_feed_again Ns = Ns Ar 1 95and only applicable in related situations. 96. 97.It Sy l2arc_feed_secs Ns = Ns Sy 1 Pq u64 98Seconds between L2ARC writing. 99. 100.It Sy l2arc_headroom Ns = Ns Sy 2 Pq u64 101How far through the ARC lists to search for L2ARC cacheable content, 102expressed as a multiplier of 103.Sy l2arc_write_max . 104ARC persistence across reboots can be achieved with persistent L2ARC 105by setting this parameter to 106.Sy 0 , 107allowing the full length of ARC lists to be searched for cacheable content. 108. 109.It Sy l2arc_headroom_boost Ns = Ns Sy 200 Ns % Pq u64 110Scales 111.Sy l2arc_headroom 112by this percentage when L2ARC contents are being successfully compressed 113before writing. 114A value of 115.Sy 100 116disables this feature. 117. 118.It Sy l2arc_exclude_special Ns = Ns Sy 0 Ns | Ns 1 Pq int 119Controls whether buffers present on special vdevs are eligible for caching 120into L2ARC. 121If set to 1, exclude dbufs on special vdevs from being cached to L2ARC. 122. 123.It Sy l2arc_mfuonly Ns = Ns Sy 0 Ns | Ns 1 Pq int 124Controls whether only MFU metadata and data are cached from ARC into L2ARC. 125This may be desired to avoid wasting space on L2ARC when reading/writing large 126amounts of data that are not expected to be accessed more than once. 127.Pp 128The default is off, 129meaning both MRU and MFU data and metadata are cached. 130When turning off this feature, some MRU buffers will still be present 131in ARC and eventually cached on L2ARC. 132.No If Sy l2arc_noprefetch Ns = Ns Sy 0 , 133some prefetched buffers will be cached to L2ARC, and those might later 134transition to MRU, in which case the 135.Sy l2arc_mru_asize No arcstat will not be Sy 0 . 136.Pp 137Regardless of 138.Sy l2arc_noprefetch , 139some MFU buffers might be evicted from ARC, 140accessed later on as prefetches and transition to MRU as prefetches. 141If accessed again they are counted as MRU and the 142.Sy l2arc_mru_asize No arcstat will not be Sy 0 . 143.Pp 144The ARC status of L2ARC buffers when they were first cached in 145L2ARC can be seen in the 146.Sy l2arc_mru_asize , Sy l2arc_mfu_asize , No and Sy l2arc_prefetch_asize 147arcstats when importing the pool or onlining a cache 148device if persistent L2ARC is enabled. 149.Pp 150The 151.Sy evict_l2_eligible_mru 152arcstat does not take into account if this option is enabled as the information 153provided by the 154.Sy evict_l2_eligible_m[rf]u 155arcstats can be used to decide if toggling this option is appropriate 156for the current workload. 157. 158.It Sy l2arc_meta_percent Ns = Ns Sy 33 Ns % Pq uint 159Percent of ARC size allowed for L2ARC-only headers. 160Since L2ARC buffers are not evicted on memory pressure, 161too many headers on a system with an irrationally large L2ARC 162can render it slow or unusable. 163This parameter limits L2ARC writes and rebuilds to achieve the target. 164. 165.It Sy l2arc_trim_ahead Ns = Ns Sy 0 Ns % Pq u64 166Trims ahead of the current write size 167.Pq Sy l2arc_write_max 168on L2ARC devices by this percentage of write size if we have filled the device. 169If set to 170.Sy 100 171we TRIM twice the space required to accommodate upcoming writes. 172A minimum of 173.Sy 64 MiB 174will be trimmed. 175It also enables TRIM of the whole L2ARC device upon creation 176or addition to an existing pool or if the header of the device is 177invalid upon importing a pool or onlining a cache device. 178A value of 179.Sy 0 180disables TRIM on L2ARC altogether and is the default as it can put significant 181stress on the underlying storage devices. 182This will vary depending of how well the specific device handles these commands. 183. 184.It Sy l2arc_noprefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int 185Do not write buffers to L2ARC if they were prefetched but not used by 186applications. 187In case there are prefetched buffers in L2ARC and this option 188is later set, we do not read the prefetched buffers from L2ARC. 189Unsetting this option is useful for caching sequential reads from the 190disks to L2ARC and serve those reads from L2ARC later on. 191This may be beneficial in case the L2ARC device is significantly faster 192in sequential reads than the disks of the pool. 193.Pp 194Use 195.Sy 1 196to disable and 197.Sy 0 198to enable caching/reading prefetches to/from L2ARC. 199. 200.It Sy l2arc_norw Ns = Ns Sy 0 Ns | Ns 1 Pq int 201No reads during writes. 202. 203.It Sy l2arc_write_boost Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq u64 204Cold L2ARC devices will have 205.Sy l2arc_write_max 206increased by this amount while they remain cold. 207. 208.It Sy l2arc_write_max Ns = Ns Sy 8388608 Ns B Po 8 MiB Pc Pq u64 209Max write bytes per interval. 210. 211.It Sy l2arc_rebuild_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 212Rebuild the L2ARC when importing a pool (persistent L2ARC). 213This can be disabled if there are problems importing a pool 214or attaching an L2ARC device (e.g. the L2ARC device is slow 215in reading stored log metadata, or the metadata 216has become somehow fragmented/unusable). 217. 218.It Sy l2arc_rebuild_blocks_min_l2size Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 219Mininum size of an L2ARC device required in order to write log blocks in it. 220The log blocks are used upon importing the pool to rebuild the persistent L2ARC. 221.Pp 222For L2ARC devices less than 1 GiB, the amount of data 223.Fn l2arc_evict 224evicts is significant compared to the amount of restored L2ARC data. 225In this case, do not write log blocks in L2ARC in order not to waste space. 226. 227.It Sy metaslab_aliquot Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 228Metaslab granularity, in bytes. 229This is roughly similar to what would be referred to as the "stripe size" 230in traditional RAID arrays. 231In normal operation, ZFS will try to write this amount of data to each disk 232before moving on to the next top-level vdev. 233. 234.It Sy metaslab_bias_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 235Enable metaslab group biasing based on their vdevs' over- or under-utilization 236relative to the pool. 237. 238.It Sy metaslab_force_ganging Ns = Ns Sy 16777217 Ns B Po 16 MiB + 1 B Pc Pq u64 239Make some blocks above a certain size be gang blocks. 240This option is used by the test suite to facilitate testing. 241. 242.It Sy zfs_ddt_zap_default_bs Ns = Ns Sy 15 Po 32 KiB Pc Pq int 243Default DDT ZAP data block size as a power of 2. Note that changing this after 244creating a DDT on the pool will not affect existing DDTs, only newly created 245ones. 246. 247.It Sy zfs_ddt_zap_default_ibs Ns = Ns Sy 15 Po 32 KiB Pc Pq int 248Default DDT ZAP indirect block size as a power of 2. Note that changing this 249after creating a DDT on the pool will not affect existing DDTs, only newly 250created ones. 251. 252.It Sy zfs_default_bs Ns = Ns Sy 9 Po 512 B Pc Pq int 253Default dnode block size as a power of 2. 254. 255.It Sy zfs_default_ibs Ns = Ns Sy 17 Po 128 KiB Pc Pq int 256Default dnode indirect block size as a power of 2. 257. 258.It Sy zfs_history_output_max Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 259When attempting to log an output nvlist of an ioctl in the on-disk history, 260the output will not be stored if it is larger than this size (in bytes). 261This must be less than 262.Sy DMU_MAX_ACCESS Pq 64 MiB . 263This applies primarily to 264.Fn zfs_ioc_channel_program Pq cf. Xr zfs-program 8 . 265. 266.It Sy zfs_keep_log_spacemaps_at_export Ns = Ns Sy 0 Ns | Ns 1 Pq int 267Prevent log spacemaps from being destroyed during pool exports and destroys. 268. 269.It Sy zfs_metaslab_segment_weight_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 270Enable/disable segment-based metaslab selection. 271. 272.It Sy zfs_metaslab_switch_threshold Ns = Ns Sy 2 Pq int 273When using segment-based metaslab selection, continue allocating 274from the active metaslab until this option's 275worth of buckets have been exhausted. 276. 277.It Sy metaslab_debug_load Ns = Ns Sy 0 Ns | Ns 1 Pq int 278Load all metaslabs during pool import. 279. 280.It Sy metaslab_debug_unload Ns = Ns Sy 0 Ns | Ns 1 Pq int 281Prevent metaslabs from being unloaded. 282. 283.It Sy metaslab_fragmentation_factor_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 284Enable use of the fragmentation metric in computing metaslab weights. 285. 286.It Sy metaslab_df_max_search Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 287Maximum distance to search forward from the last offset. 288Without this limit, fragmented pools can see 289.Em >100`000 290iterations and 291.Fn metaslab_block_picker 292becomes the performance limiting factor on high-performance storage. 293.Pp 294With the default setting of 295.Sy 16 MiB , 296we typically see less than 297.Em 500 298iterations, even with very fragmented 299.Sy ashift Ns = Ns Sy 9 300pools. 301The maximum number of iterations possible is 302.Sy metaslab_df_max_search / 2^(ashift+1) . 303With the default setting of 304.Sy 16 MiB 305this is 306.Em 16*1024 Pq with Sy ashift Ns = Ns Sy 9 307or 308.Em 2*1024 Pq with Sy ashift Ns = Ns Sy 12 . 309. 310.It Sy metaslab_df_use_largest_segment Ns = Ns Sy 0 Ns | Ns 1 Pq int 311If not searching forward (due to 312.Sy metaslab_df_max_search , metaslab_df_free_pct , 313.No or Sy metaslab_df_alloc_threshold ) , 314this tunable controls which segment is used. 315If set, we will use the largest free segment. 316If unset, we will use a segment of at least the requested size. 317. 318.It Sy zfs_metaslab_max_size_cache_sec Ns = Ns Sy 3600 Ns s Po 1 hour Pc Pq u64 319When we unload a metaslab, we cache the size of the largest free chunk. 320We use that cached size to determine whether or not to load a metaslab 321for a given allocation. 322As more frees accumulate in that metaslab while it's unloaded, 323the cached max size becomes less and less accurate. 324After a number of seconds controlled by this tunable, 325we stop considering the cached max size and start 326considering only the histogram instead. 327. 328.It Sy zfs_metaslab_mem_limit Ns = Ns Sy 25 Ns % Pq uint 329When we are loading a new metaslab, we check the amount of memory being used 330to store metaslab range trees. 331If it is over a threshold, we attempt to unload the least recently used metaslab 332to prevent the system from clogging all of its memory with range trees. 333This tunable sets the percentage of total system memory that is the threshold. 334. 335.It Sy zfs_metaslab_try_hard_before_gang Ns = Ns Sy 0 Ns | Ns 1 Pq int 336.Bl -item -compact 337.It 338If unset, we will first try normal allocation. 339.It 340If that fails then we will do a gang allocation. 341.It 342If that fails then we will do a "try hard" gang allocation. 343.It 344If that fails then we will have a multi-layer gang block. 345.El 346.Pp 347.Bl -item -compact 348.It 349If set, we will first try normal allocation. 350.It 351If that fails then we will do a "try hard" allocation. 352.It 353If that fails we will do a gang allocation. 354.It 355If that fails we will do a "try hard" gang allocation. 356.It 357If that fails then we will have a multi-layer gang block. 358.El 359. 360.It Sy zfs_metaslab_find_max_tries Ns = Ns Sy 100 Pq uint 361When not trying hard, we only consider this number of the best metaslabs. 362This improves performance, especially when there are many metaslabs per vdev 363and the allocation can't actually be satisfied 364(so we would otherwise iterate all metaslabs). 365. 366.It Sy zfs_vdev_default_ms_count Ns = Ns Sy 200 Pq uint 367When a vdev is added, target this number of metaslabs per top-level vdev. 368. 369.It Sy zfs_vdev_default_ms_shift Ns = Ns Sy 29 Po 512 MiB Pc Pq uint 370Default lower limit for metaslab size. 371. 372.It Sy zfs_vdev_max_ms_shift Ns = Ns Sy 34 Po 16 GiB Pc Pq uint 373Default upper limit for metaslab size. 374. 375.It Sy zfs_vdev_max_auto_ashift Ns = Ns Sy 14 Pq uint 376Maximum ashift used when optimizing for logical \[->] physical sector size on 377new 378top-level vdevs. 379May be increased up to 380.Sy ASHIFT_MAX Po 16 Pc , 381but this may negatively impact pool space efficiency. 382. 383.It Sy zfs_vdev_min_auto_ashift Ns = Ns Sy ASHIFT_MIN Po 9 Pc Pq uint 384Minimum ashift used when creating new top-level vdevs. 385. 386.It Sy zfs_vdev_min_ms_count Ns = Ns Sy 16 Pq uint 387Minimum number of metaslabs to create in a top-level vdev. 388. 389.It Sy vdev_validate_skip Ns = Ns Sy 0 Ns | Ns 1 Pq int 390Skip label validation steps during pool import. 391Changing is not recommended unless you know what you're doing 392and are recovering a damaged label. 393. 394.It Sy zfs_vdev_ms_count_limit Ns = Ns Sy 131072 Po 128k Pc Pq uint 395Practical upper limit of total metaslabs per top-level vdev. 396. 397.It Sy metaslab_preload_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 398Enable metaslab group preloading. 399. 400.It Sy metaslab_lba_weighting_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 401Give more weight to metaslabs with lower LBAs, 402assuming they have greater bandwidth, 403as is typically the case on a modern constant angular velocity disk drive. 404. 405.It Sy metaslab_unload_delay Ns = Ns Sy 32 Pq uint 406After a metaslab is used, we keep it loaded for this many TXGs, to attempt to 407reduce unnecessary reloading. 408Note that both this many TXGs and 409.Sy metaslab_unload_delay_ms 410milliseconds must pass before unloading will occur. 411. 412.It Sy metaslab_unload_delay_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq uint 413After a metaslab is used, we keep it loaded for this many milliseconds, 414to attempt to reduce unnecessary reloading. 415Note, that both this many milliseconds and 416.Sy metaslab_unload_delay 417TXGs must pass before unloading will occur. 418. 419.It Sy reference_history Ns = Ns Sy 3 Pq uint 420Maximum reference holders being tracked when reference_tracking_enable is 421active. 422. 423.It Sy reference_tracking_enable Ns = Ns Sy 0 Ns | Ns 1 Pq int 424Track reference holders to 425.Sy refcount_t 426objects (debug builds only). 427. 428.It Sy send_holes_without_birth_time Ns = Ns Sy 1 Ns | Ns 0 Pq int 429When set, the 430.Sy hole_birth 431optimization will not be used, and all holes will always be sent during a 432.Nm zfs Cm send . 433This is useful if you suspect your datasets are affected by a bug in 434.Sy hole_birth . 435. 436.It Sy spa_config_path Ns = Ns Pa /etc/zfs/zpool.cache Pq charp 437SPA config file. 438. 439.It Sy spa_asize_inflation Ns = Ns Sy 24 Pq uint 440Multiplication factor used to estimate actual disk consumption from the 441size of data being written. 442The default value is a worst case estimate, 443but lower values may be valid for a given pool depending on its configuration. 444Pool administrators who understand the factors involved 445may wish to specify a more realistic inflation factor, 446particularly if they operate close to quota or capacity limits. 447. 448.It Sy spa_load_print_vdev_tree Ns = Ns Sy 0 Ns | Ns 1 Pq int 449Whether to print the vdev tree in the debugging message buffer during pool 450import. 451. 452.It Sy spa_load_verify_data Ns = Ns Sy 1 Ns | Ns 0 Pq int 453Whether to traverse data blocks during an "extreme rewind" 454.Pq Fl X 455import. 456.Pp 457An extreme rewind import normally performs a full traversal of all 458blocks in the pool for verification. 459If this parameter is unset, the traversal skips non-metadata blocks. 460It can be toggled once the 461import has started to stop or start the traversal of non-metadata blocks. 462. 463.It Sy spa_load_verify_metadata Ns = Ns Sy 1 Ns | Ns 0 Pq int 464Whether to traverse blocks during an "extreme rewind" 465.Pq Fl X 466pool import. 467.Pp 468An extreme rewind import normally performs a full traversal of all 469blocks in the pool for verification. 470If this parameter is unset, the traversal is not performed. 471It can be toggled once the import has started to stop or start the traversal. 472. 473.It Sy spa_load_verify_shift Ns = Ns Sy 4 Po 1/16th Pc Pq uint 474Sets the maximum number of bytes to consume during pool import to the log2 475fraction of the target ARC size. 476. 477.It Sy spa_slop_shift Ns = Ns Sy 5 Po 1/32nd Pc Pq int 478Normally, we don't allow the last 479.Sy 3.2% Pq Sy 1/2^spa_slop_shift 480of space in the pool to be consumed. 481This ensures that we don't run the pool completely out of space, 482due to unaccounted changes (e.g. to the MOS). 483It also limits the worst-case time to allocate space. 484If we have less than this amount of free space, 485most ZPL operations (e.g. write, create) will return 486.Sy ENOSPC . 487. 488.It Sy spa_upgrade_errlog_limit Ns = Ns Sy 0 Pq uint 489Limits the number of on-disk error log entries that will be converted to the 490new format when enabling the 491.Sy head_errlog 492feature. 493The default is to convert all log entries. 494. 495.It Sy vdev_removal_max_span Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint 496During top-level vdev removal, chunks of data are copied from the vdev 497which may include free space in order to trade bandwidth for IOPS. 498This parameter determines the maximum span of free space, in bytes, 499which will be included as "unnecessary" data in a chunk of copied data. 500.Pp 501The default value here was chosen to align with 502.Sy zfs_vdev_read_gap_limit , 503which is a similar concept when doing 504regular reads (but there's no reason it has to be the same). 505. 506.It Sy vdev_file_logical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64 507Logical ashift for file-based devices. 508. 509.It Sy vdev_file_physical_ashift Ns = Ns Sy 9 Po 512 B Pc Pq u64 510Physical ashift for file-based devices. 511. 512.It Sy zap_iterate_prefetch Ns = Ns Sy 1 Ns | Ns 0 Pq int 513If set, when we start iterating over a ZAP object, 514prefetch the entire object (all leaf blocks). 515However, this is limited by 516.Sy dmu_prefetch_max . 517. 518.It Sy zap_micro_max_size Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq int 519Maximum micro ZAP size. 520A micro ZAP is upgraded to a fat ZAP, once it grows beyond the specified size. 521. 522.It Sy zfetch_array_rd_sz Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 523If prefetching is enabled, disable prefetching for reads larger than this size. 524. 525.It Sy zfetch_min_distance Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint 526Min bytes to prefetch per stream. 527Prefetch distance starts from the demand access size and quickly grows to 528this value, doubling on each hit. 529After that it may grow further by 1/8 per hit, but only if some prefetch 530since last time haven't completed in time to satisfy demand request, i.e. 531prefetch depth didn't cover the read latency or the pool got saturated. 532. 533.It Sy zfetch_max_distance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint 534Max bytes to prefetch per stream. 535. 536.It Sy zfetch_max_idistance Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq uint 537Max bytes to prefetch indirects for per stream. 538. 539.It Sy zfetch_max_streams Ns = Ns Sy 8 Pq uint 540Max number of streams per zfetch (prefetch streams per file). 541. 542.It Sy zfetch_min_sec_reap Ns = Ns Sy 1 Pq uint 543Min time before inactive prefetch stream can be reclaimed 544. 545.It Sy zfetch_max_sec_reap Ns = Ns Sy 2 Pq uint 546Max time before inactive prefetch stream can be deleted 547. 548.It Sy zfs_abd_scatter_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 549Enables ARC from using scatter/gather lists and forces all allocations to be 550linear in kernel memory. 551Disabling can improve performance in some code paths 552at the expense of fragmented kernel memory. 553. 554.It Sy zfs_abd_scatter_max_order Ns = Ns Sy MAX_ORDER\-1 Pq uint 555Maximum number of consecutive memory pages allocated in a single block for 556scatter/gather lists. 557.Pp 558The value of 559.Sy MAX_ORDER 560depends on kernel configuration. 561. 562.It Sy zfs_abd_scatter_min_size Ns = Ns Sy 1536 Ns B Po 1.5 KiB Pc Pq uint 563This is the minimum allocation size that will use scatter (page-based) ABDs. 564Smaller allocations will use linear ABDs. 565. 566.It Sy zfs_arc_dnode_limit Ns = Ns Sy 0 Ns B Pq u64 567When the number of bytes consumed by dnodes in the ARC exceeds this number of 568bytes, try to unpin some of it in response to demand for non-metadata. 569This value acts as a ceiling to the amount of dnode metadata, and defaults to 570.Sy 0 , 571which indicates that a percent which is based on 572.Sy zfs_arc_dnode_limit_percent 573of the ARC meta buffers that may be used for dnodes. 574.It Sy zfs_arc_dnode_limit_percent Ns = Ns Sy 10 Ns % Pq u64 575Percentage that can be consumed by dnodes of ARC meta buffers. 576.Pp 577See also 578.Sy zfs_arc_dnode_limit , 579which serves a similar purpose but has a higher priority if nonzero. 580. 581.It Sy zfs_arc_dnode_reduce_percent Ns = Ns Sy 10 Ns % Pq u64 582Percentage of ARC dnodes to try to scan in response to demand for non-metadata 583when the number of bytes consumed by dnodes exceeds 584.Sy zfs_arc_dnode_limit . 585. 586.It Sy zfs_arc_average_blocksize Ns = Ns Sy 8192 Ns B Po 8 KiB Pc Pq uint 587The ARC's buffer hash table is sized based on the assumption of an average 588block size of this value. 589This works out to roughly 1 MiB of hash table per 1 GiB of physical memory 590with 8-byte pointers. 591For configurations with a known larger average block size, 592this value can be increased to reduce the memory footprint. 593. 594.It Sy zfs_arc_eviction_pct Ns = Ns Sy 200 Ns % Pq uint 595When 596.Fn arc_is_overflowing , 597.Fn arc_get_data_impl 598waits for this percent of the requested amount of data to be evicted. 599For example, by default, for every 600.Em 2 KiB 601that's evicted, 602.Em 1 KiB 603of it may be "reused" by a new allocation. 604Since this is above 605.Sy 100 Ns % , 606it ensures that progress is made towards getting 607.Sy arc_size No under Sy arc_c . 608Since this is finite, it ensures that allocations can still happen, 609even during the potentially long time that 610.Sy arc_size No is more than Sy arc_c . 611. 612.It Sy zfs_arc_evict_batch_limit Ns = Ns Sy 10 Pq uint 613Number ARC headers to evict per sub-list before proceeding to another sub-list. 614This batch-style operation prevents entire sub-lists from being evicted at once 615but comes at a cost of additional unlocking and locking. 616. 617.It Sy zfs_arc_grow_retry Ns = Ns Sy 0 Ns s Pq uint 618If set to a non zero value, it will replace the 619.Sy arc_grow_retry 620value with this value. 621The 622.Sy arc_grow_retry 623.No value Pq default Sy 5 Ns s 624is the number of seconds the ARC will wait before 625trying to resume growth after a memory pressure event. 626. 627.It Sy zfs_arc_lotsfree_percent Ns = Ns Sy 10 Ns % Pq int 628Throttle I/O when free system memory drops below this percentage of total 629system memory. 630Setting this value to 631.Sy 0 632will disable the throttle. 633. 634.It Sy zfs_arc_max Ns = Ns Sy 0 Ns B Pq u64 635Max size of ARC in bytes. 636If 637.Sy 0 , 638then the max size of ARC is determined by the amount of system memory installed. 639Under Linux, half of system memory will be used as the limit. 640Under 641.Fx , 642the larger of 643.Sy all_system_memory No \- Sy 1 GiB 644and 645.Sy 5/8 No \(mu Sy all_system_memory 646will be used as the limit. 647This value must be at least 648.Sy 67108864 Ns B Pq 64 MiB . 649.Pp 650This value can be changed dynamically, with some caveats. 651It cannot be set back to 652.Sy 0 653while running, and reducing it below the current ARC size will not cause 654the ARC to shrink without memory pressure to induce shrinking. 655. 656.It Sy zfs_arc_meta_balance Ns = Ns Sy 500 Pq uint 657Balance between metadata and data on ghost hits. 658Values above 100 increase metadata caching by proportionally reducing effect 659of ghost data hits on target data/metadata rate. 660. 661.It Sy zfs_arc_min Ns = Ns Sy 0 Ns B Pq u64 662Min size of ARC in bytes. 663.No If set to Sy 0 , arc_c_min 664will default to consuming the larger of 665.Sy 32 MiB 666and 667.Sy all_system_memory No / Sy 32 . 668. 669.It Sy zfs_arc_min_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 1s Pc Pq uint 670Minimum time prefetched blocks are locked in the ARC. 671. 672.It Sy zfs_arc_min_prescient_prefetch_ms Ns = Ns Sy 0 Ns ms Ns Po Ns ≡ Ns 6s Pc Pq uint 673Minimum time "prescient prefetched" blocks are locked in the ARC. 674These blocks are meant to be prefetched fairly aggressively ahead of 675the code that may use them. 676. 677.It Sy zfs_arc_prune_task_threads Ns = Ns Sy 1 Pq int 678Number of arc_prune threads. 679.Fx 680does not need more than one. 681Linux may theoretically use one per mount point up to number of CPUs, 682but that was not proven to be useful. 683. 684.It Sy zfs_max_missing_tvds Ns = Ns Sy 0 Pq int 685Number of missing top-level vdevs which will be allowed during 686pool import (only in read-only mode). 687. 688.It Sy zfs_max_nvlist_src_size Ns = Sy 0 Pq u64 689Maximum size in bytes allowed to be passed as 690.Sy zc_nvlist_src_size 691for ioctls on 692.Pa /dev/zfs . 693This prevents a user from causing the kernel to allocate 694an excessive amount of memory. 695When the limit is exceeded, the ioctl fails with 696.Sy EINVAL 697and a description of the error is sent to the 698.Pa zfs-dbgmsg 699log. 700This parameter should not need to be touched under normal circumstances. 701If 702.Sy 0 , 703equivalent to a quarter of the user-wired memory limit under 704.Fx 705and to 706.Sy 134217728 Ns B Pq 128 MiB 707under Linux. 708. 709.It Sy zfs_multilist_num_sublists Ns = Ns Sy 0 Pq uint 710To allow more fine-grained locking, each ARC state contains a series 711of lists for both data and metadata objects. 712Locking is performed at the level of these "sub-lists". 713This parameters controls the number of sub-lists per ARC state, 714and also applies to other uses of the multilist data structure. 715.Pp 716If 717.Sy 0 , 718equivalent to the greater of the number of online CPUs and 719.Sy 4 . 720. 721.It Sy zfs_arc_overflow_shift Ns = Ns Sy 8 Pq int 722The ARC size is considered to be overflowing if it exceeds the current 723ARC target size 724.Pq Sy arc_c 725by thresholds determined by this parameter. 726Exceeding by 727.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No / Sy 2 728starts ARC reclamation process. 729If that appears insufficient, exceeding by 730.Sy ( arc_c No >> Sy zfs_arc_overflow_shift ) No \(mu Sy 1.5 731blocks new buffer allocation until the reclaim thread catches up. 732Started reclamation process continues till ARC size returns below the 733target size. 734.Pp 735The default value of 736.Sy 8 737causes the ARC to start reclamation if it exceeds the target size by 738.Em 0.2% 739of the target size, and block allocations by 740.Em 0.6% . 741. 742.It Sy zfs_arc_shrink_shift Ns = Ns Sy 0 Pq uint 743If nonzero, this will update 744.Sy arc_shrink_shift Pq default Sy 7 745with the new value. 746. 747.It Sy zfs_arc_pc_percent Ns = Ns Sy 0 Ns % Po off Pc Pq uint 748Percent of pagecache to reclaim ARC to. 749.Pp 750This tunable allows the ZFS ARC to play more nicely 751with the kernel's LRU pagecache. 752It can guarantee that the ARC size won't collapse under scanning 753pressure on the pagecache, yet still allows the ARC to be reclaimed down to 754.Sy zfs_arc_min 755if necessary. 756This value is specified as percent of pagecache size (as measured by 757.Sy NR_FILE_PAGES ) , 758where that percent may exceed 759.Sy 100 . 760This 761only operates during memory pressure/reclaim. 762. 763.It Sy zfs_arc_shrinker_limit Ns = Ns Sy 10000 Pq int 764This is a limit on how many pages the ARC shrinker makes available for 765eviction in response to one page allocation attempt. 766Note that in practice, the kernel's shrinker can ask us to evict 767up to about four times this for one allocation attempt. 768.Pp 769The default limit of 770.Sy 10000 Pq in practice, Em 160 MiB No per allocation attempt with 4 KiB pages 771limits the amount of time spent attempting to reclaim ARC memory to 772less than 100 ms per allocation attempt, 773even with a small average compressed block size of ~8 KiB. 774.Pp 775The parameter can be set to 0 (zero) to disable the limit, 776and only applies on Linux. 777. 778.It Sy zfs_arc_sys_free Ns = Ns Sy 0 Ns B Pq u64 779The target number of bytes the ARC should leave as free memory on the system. 780If zero, equivalent to the bigger of 781.Sy 512 KiB No and Sy all_system_memory/64 . 782. 783.It Sy zfs_autoimport_disable Ns = Ns Sy 1 Ns | Ns 0 Pq int 784Disable pool import at module load by ignoring the cache file 785.Pq Sy spa_config_path . 786. 787.It Sy zfs_checksum_events_per_second Ns = Ns Sy 20 Ns /s Pq uint 788Rate limit checksum events to this many per second. 789Note that this should not be set below the ZED thresholds 790(currently 10 checksums over 10 seconds) 791or else the daemon may not trigger any action. 792. 793.It Sy zfs_commit_timeout_pct Ns = Ns Sy 5 Ns % Pq uint 794This controls the amount of time that a ZIL block (lwb) will remain "open" 795when it isn't "full", and it has a thread waiting for it to be committed to 796stable storage. 797The timeout is scaled based on a percentage of the last lwb 798latency to avoid significantly impacting the latency of each individual 799transaction record (itx). 800. 801.It Sy zfs_condense_indirect_commit_entry_delay_ms Ns = Ns Sy 0 Ns ms Pq int 802Vdev indirection layer (used for device removal) sleeps for this many 803milliseconds during mapping generation. 804Intended for use with the test suite to throttle vdev removal speed. 805. 806.It Sy zfs_condense_indirect_obsolete_pct Ns = Ns Sy 25 Ns % Pq uint 807Minimum percent of obsolete bytes in vdev mapping required to attempt to 808condense 809.Pq see Sy zfs_condense_indirect_vdevs_enable . 810Intended for use with the test suite 811to facilitate triggering condensing as needed. 812. 813.It Sy zfs_condense_indirect_vdevs_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int 814Enable condensing indirect vdev mappings. 815When set, attempt to condense indirect vdev mappings 816if the mapping uses more than 817.Sy zfs_condense_min_mapping_bytes 818bytes of memory and if the obsolete space map object uses more than 819.Sy zfs_condense_max_obsolete_bytes 820bytes on-disk. 821The condensing process is an attempt to save memory by removing obsolete 822mappings. 823. 824.It Sy zfs_condense_max_obsolete_bytes Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 825Only attempt to condense indirect vdev mappings if the on-disk size 826of the obsolete space map object is greater than this number of bytes 827.Pq see Sy zfs_condense_indirect_vdevs_enable . 828. 829.It Sy zfs_condense_min_mapping_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq u64 830Minimum size vdev mapping to attempt to condense 831.Pq see Sy zfs_condense_indirect_vdevs_enable . 832. 833.It Sy zfs_dbgmsg_enable Ns = Ns Sy 1 Ns | Ns 0 Pq int 834Internally ZFS keeps a small log to facilitate debugging. 835The log is enabled by default, and can be disabled by unsetting this option. 836The contents of the log can be accessed by reading 837.Pa /proc/spl/kstat/zfs/dbgmsg . 838Writing 839.Sy 0 840to the file clears the log. 841.Pp 842This setting does not influence debug prints due to 843.Sy zfs_flags . 844. 845.It Sy zfs_dbgmsg_maxsize Ns = Ns Sy 4194304 Ns B Po 4 MiB Pc Pq uint 846Maximum size of the internal ZFS debug log. 847. 848.It Sy zfs_dbuf_state_index Ns = Ns Sy 0 Pq int 849Historically used for controlling what reporting was available under 850.Pa /proc/spl/kstat/zfs . 851No effect. 852. 853.It Sy zfs_deadman_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 854When a pool sync operation takes longer than 855.Sy zfs_deadman_synctime_ms , 856or when an individual I/O operation takes longer than 857.Sy zfs_deadman_ziotime_ms , 858then the operation is considered to be "hung". 859If 860.Sy zfs_deadman_enabled 861is set, then the deadman behavior is invoked as described by 862.Sy zfs_deadman_failmode . 863By default, the deadman is enabled and set to 864.Sy wait 865which results in "hung" I/O operations only being logged. 866The deadman is automatically disabled when a pool gets suspended. 867. 868.It Sy zfs_deadman_failmode Ns = Ns Sy wait Pq charp 869Controls the failure behavior when the deadman detects a "hung" I/O operation. 870Valid values are: 871.Bl -tag -compact -offset 4n -width "continue" 872.It Sy wait 873Wait for a "hung" operation to complete. 874For each "hung" operation a "deadman" event will be posted 875describing that operation. 876.It Sy continue 877Attempt to recover from a "hung" operation by re-dispatching it 878to the I/O pipeline if possible. 879.It Sy panic 880Panic the system. 881This can be used to facilitate automatic fail-over 882to a properly configured fail-over partner. 883.El 884. 885.It Sy zfs_deadman_checktime_ms Ns = Ns Sy 60000 Ns ms Po 1 min Pc Pq u64 886Check time in milliseconds. 887This defines the frequency at which we check for hung I/O requests 888and potentially invoke the 889.Sy zfs_deadman_failmode 890behavior. 891. 892.It Sy zfs_deadman_synctime_ms Ns = Ns Sy 600000 Ns ms Po 10 min Pc Pq u64 893Interval in milliseconds after which the deadman is triggered and also 894the interval after which a pool sync operation is considered to be "hung". 895Once this limit is exceeded the deadman will be invoked every 896.Sy zfs_deadman_checktime_ms 897milliseconds until the pool sync completes. 898. 899.It Sy zfs_deadman_ziotime_ms Ns = Ns Sy 300000 Ns ms Po 5 min Pc Pq u64 900Interval in milliseconds after which the deadman is triggered and an 901individual I/O operation is considered to be "hung". 902As long as the operation remains "hung", 903the deadman will be invoked every 904.Sy zfs_deadman_checktime_ms 905milliseconds until the operation completes. 906. 907.It Sy zfs_dedup_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int 908Enable prefetching dedup-ed blocks which are going to be freed. 909. 910.It Sy zfs_delay_min_dirty_percent Ns = Ns Sy 60 Ns % Pq uint 911Start to delay each transaction once there is this amount of dirty data, 912expressed as a percentage of 913.Sy zfs_dirty_data_max . 914This value should be at least 915.Sy zfs_vdev_async_write_active_max_dirty_percent . 916.No See Sx ZFS TRANSACTION DELAY . 917. 918.It Sy zfs_delay_scale Ns = Ns Sy 500000 Pq int 919This controls how quickly the transaction delay approaches infinity. 920Larger values cause longer delays for a given amount of dirty data. 921.Pp 922For the smoothest delay, this value should be about 1 billion divided 923by the maximum number of operations per second. 924This will smoothly handle between ten times and a tenth of this number. 925.No See Sx ZFS TRANSACTION DELAY . 926.Pp 927.Sy zfs_delay_scale No \(mu Sy zfs_dirty_data_max Em must No be smaller than Sy 2^64 . 928. 929.It Sy zfs_disable_ivset_guid_check Ns = Ns Sy 0 Ns | Ns 1 Pq int 930Disables requirement for IVset GUIDs to be present and match when doing a raw 931receive of encrypted datasets. 932Intended for users whose pools were created with 933OpenZFS pre-release versions and now have compatibility issues. 934. 935.It Sy zfs_key_max_salt_uses Ns = Ns Sy 400000000 Po 4*10^8 Pc Pq ulong 936Maximum number of uses of a single salt value before generating a new one for 937encrypted datasets. 938The default value is also the maximum. 939. 940.It Sy zfs_object_mutex_size Ns = Ns Sy 64 Pq uint 941Size of the znode hashtable used for holds. 942.Pp 943Due to the need to hold locks on objects that may not exist yet, kernel mutexes 944are not created per-object and instead a hashtable is used where collisions 945will result in objects waiting when there is not actually contention on the 946same object. 947. 948.It Sy zfs_slow_io_events_per_second Ns = Ns Sy 20 Ns /s Pq int 949Rate limit delay and deadman zevents (which report slow I/O operations) to this 950many per 951second. 952. 953.It Sy zfs_unflushed_max_mem_amt Ns = Ns Sy 1073741824 Ns B Po 1 GiB Pc Pq u64 954Upper-bound limit for unflushed metadata changes to be held by the 955log spacemap in memory, in bytes. 956. 957.It Sy zfs_unflushed_max_mem_ppm Ns = Ns Sy 1000 Ns ppm Po 0.1% Pc Pq u64 958Part of overall system memory that ZFS allows to be used 959for unflushed metadata changes by the log spacemap, in millionths. 960. 961.It Sy zfs_unflushed_log_block_max Ns = Ns Sy 131072 Po 128k Pc Pq u64 962Describes the maximum number of log spacemap blocks allowed for each pool. 963The default value means that the space in all the log spacemaps 964can add up to no more than 965.Sy 131072 966blocks (which means 967.Em 16 GiB 968of logical space before compression and ditto blocks, 969assuming that blocksize is 970.Em 128 KiB ) . 971.Pp 972This tunable is important because it involves a trade-off between import 973time after an unclean export and the frequency of flushing metaslabs. 974The higher this number is, the more log blocks we allow when the pool is 975active which means that we flush metaslabs less often and thus decrease 976the number of I/O operations for spacemap updates per TXG. 977At the same time though, that means that in the event of an unclean export, 978there will be more log spacemap blocks for us to read, inducing overhead 979in the import time of the pool. 980The lower the number, the amount of flushing increases, destroying log 981blocks quicker as they become obsolete faster, which leaves less blocks 982to be read during import time after a crash. 983.Pp 984Each log spacemap block existing during pool import leads to approximately 985one extra logical I/O issued. 986This is the reason why this tunable is exposed in terms of blocks rather 987than space used. 988. 989.It Sy zfs_unflushed_log_block_min Ns = Ns Sy 1000 Pq u64 990If the number of metaslabs is small and our incoming rate is high, 991we could get into a situation that we are flushing all our metaslabs every TXG. 992Thus we always allow at least this many log blocks. 993. 994.It Sy zfs_unflushed_log_block_pct Ns = Ns Sy 400 Ns % Pq u64 995Tunable used to determine the number of blocks that can be used for 996the spacemap log, expressed as a percentage of the total number of 997unflushed metaslabs in the pool. 998. 999.It Sy zfs_unflushed_log_txg_max Ns = Ns Sy 1000 Pq u64 1000Tunable limiting maximum time in TXGs any metaslab may remain unflushed. 1001It effectively limits maximum number of unflushed per-TXG spacemap logs 1002that need to be read after unclean pool export. 1003. 1004.It Sy zfs_unlink_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint 1005When enabled, files will not be asynchronously removed from the list of pending 1006unlinks and the space they consume will be leaked. 1007Once this option has been disabled and the dataset is remounted, 1008the pending unlinks will be processed and the freed space returned to the pool. 1009This option is used by the test suite. 1010. 1011.It Sy zfs_delete_blocks Ns = Ns Sy 20480 Pq ulong 1012This is the used to define a large file for the purposes of deletion. 1013Files containing more than 1014.Sy zfs_delete_blocks 1015will be deleted asynchronously, while smaller files are deleted synchronously. 1016Decreasing this value will reduce the time spent in an 1017.Xr unlink 2 1018system call, at the expense of a longer delay before the freed space is 1019available. 1020This only applies on Linux. 1021. 1022.It Sy zfs_dirty_data_max Ns = Pq int 1023Determines the dirty space limit in bytes. 1024Once this limit is exceeded, new writes are halted until space frees up. 1025This parameter takes precedence over 1026.Sy zfs_dirty_data_max_percent . 1027.No See Sx ZFS TRANSACTION DELAY . 1028.Pp 1029Defaults to 1030.Sy physical_ram/10 , 1031capped at 1032.Sy zfs_dirty_data_max_max . 1033. 1034.It Sy zfs_dirty_data_max_max Ns = Pq int 1035Maximum allowable value of 1036.Sy zfs_dirty_data_max , 1037expressed in bytes. 1038This limit is only enforced at module load time, and will be ignored if 1039.Sy zfs_dirty_data_max 1040is later changed. 1041This parameter takes precedence over 1042.Sy zfs_dirty_data_max_max_percent . 1043.No See Sx ZFS TRANSACTION DELAY . 1044.Pp 1045Defaults to 1046.Sy min(physical_ram/4, 4GiB) , 1047or 1048.Sy min(physical_ram/4, 1GiB) 1049for 32-bit systems. 1050. 1051.It Sy zfs_dirty_data_max_max_percent Ns = Ns Sy 25 Ns % Pq uint 1052Maximum allowable value of 1053.Sy zfs_dirty_data_max , 1054expressed as a percentage of physical RAM. 1055This limit is only enforced at module load time, and will be ignored if 1056.Sy zfs_dirty_data_max 1057is later changed. 1058The parameter 1059.Sy zfs_dirty_data_max_max 1060takes precedence over this one. 1061.No See Sx ZFS TRANSACTION DELAY . 1062. 1063.It Sy zfs_dirty_data_max_percent Ns = Ns Sy 10 Ns % Pq uint 1064Determines the dirty space limit, expressed as a percentage of all memory. 1065Once this limit is exceeded, new writes are halted until space frees up. 1066The parameter 1067.Sy zfs_dirty_data_max 1068takes precedence over this one. 1069.No See Sx ZFS TRANSACTION DELAY . 1070.Pp 1071Subject to 1072.Sy zfs_dirty_data_max_max . 1073. 1074.It Sy zfs_dirty_data_sync_percent Ns = Ns Sy 20 Ns % Pq uint 1075Start syncing out a transaction group if there's at least this much dirty data 1076.Pq as a percentage of Sy zfs_dirty_data_max . 1077This should be less than 1078.Sy zfs_vdev_async_write_active_min_dirty_percent . 1079. 1080.It Sy zfs_wrlog_data_max Ns = Pq int 1081The upper limit of write-transaction zil log data size in bytes. 1082Write operations are throttled when approaching the limit until log data is 1083cleared out after transaction group sync. 1084Because of some overhead, it should be set at least 2 times the size of 1085.Sy zfs_dirty_data_max 1086.No to prevent harming normal write throughput . 1087It also should be smaller than the size of the slog device if slog is present. 1088.Pp 1089Defaults to 1090.Sy zfs_dirty_data_max*2 1091. 1092.It Sy zfs_fallocate_reserve_percent Ns = Ns Sy 110 Ns % Pq uint 1093Since ZFS is a copy-on-write filesystem with snapshots, blocks cannot be 1094preallocated for a file in order to guarantee that later writes will not 1095run out of space. 1096Instead, 1097.Xr fallocate 2 1098space preallocation only checks that sufficient space is currently available 1099in the pool or the user's project quota allocation, 1100and then creates a sparse file of the requested size. 1101The requested space is multiplied by 1102.Sy zfs_fallocate_reserve_percent 1103to allow additional space for indirect blocks and other internal metadata. 1104Setting this to 1105.Sy 0 1106disables support for 1107.Xr fallocate 2 1108and causes it to return 1109.Sy EOPNOTSUPP . 1110. 1111.It Sy zfs_fletcher_4_impl Ns = Ns Sy fastest Pq string 1112Select a fletcher 4 implementation. 1113.Pp 1114Supported selectors are: 1115.Sy fastest , scalar , sse2 , ssse3 , avx2 , avx512f , avx512bw , 1116.No and Sy aarch64_neon . 1117All except 1118.Sy fastest No and Sy scalar 1119require instruction set extensions to be available, 1120and will only appear if ZFS detects that they are present at runtime. 1121If multiple implementations of fletcher 4 are available, the 1122.Sy fastest 1123will be chosen using a micro benchmark. 1124Selecting 1125.Sy scalar 1126results in the original CPU-based calculation being used. 1127Selecting any option other than 1128.Sy fastest No or Sy scalar 1129results in vector instructions 1130from the respective CPU instruction set being used. 1131. 1132.It Sy zfs_blake3_impl Ns = Ns Sy fastest Pq string 1133Select a BLAKE3 implementation. 1134.Pp 1135Supported selectors are: 1136.Sy cycle , fastest , generic , sse2 , sse41 , avx2 , avx512 . 1137All except 1138.Sy cycle , fastest No and Sy generic 1139require instruction set extensions to be available, 1140and will only appear if ZFS detects that they are present at runtime. 1141If multiple implementations of BLAKE3 are available, the 1142.Sy fastest will be chosen using a micro benchmark. You can see the 1143benchmark results by reading this kstat file: 1144.Pa /proc/spl/kstat/zfs/chksum_bench . 1145. 1146.It Sy zfs_free_bpobj_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 1147Enable/disable the processing of the free_bpobj object. 1148. 1149.It Sy zfs_async_block_max_blocks Ns = Ns Sy UINT64_MAX Po unlimited Pc Pq u64 1150Maximum number of blocks freed in a single TXG. 1151. 1152.It Sy zfs_max_async_dedup_frees Ns = Ns Sy 100000 Po 10^5 Pc Pq u64 1153Maximum number of dedup blocks freed in a single TXG. 1154. 1155.It Sy zfs_vdev_async_read_max_active Ns = Ns Sy 3 Pq uint 1156Maximum asynchronous read I/O operations active to each device. 1157.No See Sx ZFS I/O SCHEDULER . 1158. 1159.It Sy zfs_vdev_async_read_min_active Ns = Ns Sy 1 Pq uint 1160Minimum asynchronous read I/O operation active to each device. 1161.No See Sx ZFS I/O SCHEDULER . 1162. 1163.It Sy zfs_vdev_async_write_active_max_dirty_percent Ns = Ns Sy 60 Ns % Pq uint 1164When the pool has more than this much dirty data, use 1165.Sy zfs_vdev_async_write_max_active 1166to limit active async writes. 1167If the dirty data is between the minimum and maximum, 1168the active I/O limit is linearly interpolated. 1169.No See Sx ZFS I/O SCHEDULER . 1170. 1171.It Sy zfs_vdev_async_write_active_min_dirty_percent Ns = Ns Sy 30 Ns % Pq uint 1172When the pool has less than this much dirty data, use 1173.Sy zfs_vdev_async_write_min_active 1174to limit active async writes. 1175If the dirty data is between the minimum and maximum, 1176the active I/O limit is linearly 1177interpolated. 1178.No See Sx ZFS I/O SCHEDULER . 1179. 1180.It Sy zfs_vdev_async_write_max_active Ns = Ns Sy 10 Pq uint 1181Maximum asynchronous write I/O operations active to each device. 1182.No See Sx ZFS I/O SCHEDULER . 1183. 1184.It Sy zfs_vdev_async_write_min_active Ns = Ns Sy 2 Pq uint 1185Minimum asynchronous write I/O operations active to each device. 1186.No See Sx ZFS I/O SCHEDULER . 1187.Pp 1188Lower values are associated with better latency on rotational media but poorer 1189resilver performance. 1190The default value of 1191.Sy 2 1192was chosen as a compromise. 1193A value of 1194.Sy 3 1195has been shown to improve resilver performance further at a cost of 1196further increasing latency. 1197. 1198.It Sy zfs_vdev_initializing_max_active Ns = Ns Sy 1 Pq uint 1199Maximum initializing I/O operations active to each device. 1200.No See Sx ZFS I/O SCHEDULER . 1201. 1202.It Sy zfs_vdev_initializing_min_active Ns = Ns Sy 1 Pq uint 1203Minimum initializing I/O operations active to each device. 1204.No See Sx ZFS I/O SCHEDULER . 1205. 1206.It Sy zfs_vdev_max_active Ns = Ns Sy 1000 Pq uint 1207The maximum number of I/O operations active to each device. 1208Ideally, this will be at least the sum of each queue's 1209.Sy max_active . 1210.No See Sx ZFS I/O SCHEDULER . 1211. 1212.It Sy zfs_vdev_open_timeout_ms Ns = Ns Sy 1000 Pq uint 1213Timeout value to wait before determining a device is missing 1214during import. 1215This is helpful for transient missing paths due 1216to links being briefly removed and recreated in response to 1217udev events. 1218. 1219.It Sy zfs_vdev_rebuild_max_active Ns = Ns Sy 3 Pq uint 1220Maximum sequential resilver I/O operations active to each device. 1221.No See Sx ZFS I/O SCHEDULER . 1222. 1223.It Sy zfs_vdev_rebuild_min_active Ns = Ns Sy 1 Pq uint 1224Minimum sequential resilver I/O operations active to each device. 1225.No See Sx ZFS I/O SCHEDULER . 1226. 1227.It Sy zfs_vdev_removal_max_active Ns = Ns Sy 2 Pq uint 1228Maximum removal I/O operations active to each device. 1229.No See Sx ZFS I/O SCHEDULER . 1230. 1231.It Sy zfs_vdev_removal_min_active Ns = Ns Sy 1 Pq uint 1232Minimum removal I/O operations active to each device. 1233.No See Sx ZFS I/O SCHEDULER . 1234. 1235.It Sy zfs_vdev_scrub_max_active Ns = Ns Sy 2 Pq uint 1236Maximum scrub I/O operations active to each device. 1237.No See Sx ZFS I/O SCHEDULER . 1238. 1239.It Sy zfs_vdev_scrub_min_active Ns = Ns Sy 1 Pq uint 1240Minimum scrub I/O operations active to each device. 1241.No See Sx ZFS I/O SCHEDULER . 1242. 1243.It Sy zfs_vdev_sync_read_max_active Ns = Ns Sy 10 Pq uint 1244Maximum synchronous read I/O operations active to each device. 1245.No See Sx ZFS I/O SCHEDULER . 1246. 1247.It Sy zfs_vdev_sync_read_min_active Ns = Ns Sy 10 Pq uint 1248Minimum synchronous read I/O operations active to each device. 1249.No See Sx ZFS I/O SCHEDULER . 1250. 1251.It Sy zfs_vdev_sync_write_max_active Ns = Ns Sy 10 Pq uint 1252Maximum synchronous write I/O operations active to each device. 1253.No See Sx ZFS I/O SCHEDULER . 1254. 1255.It Sy zfs_vdev_sync_write_min_active Ns = Ns Sy 10 Pq uint 1256Minimum synchronous write I/O operations active to each device. 1257.No See Sx ZFS I/O SCHEDULER . 1258. 1259.It Sy zfs_vdev_trim_max_active Ns = Ns Sy 2 Pq uint 1260Maximum trim/discard I/O operations active to each device. 1261.No See Sx ZFS I/O SCHEDULER . 1262. 1263.It Sy zfs_vdev_trim_min_active Ns = Ns Sy 1 Pq uint 1264Minimum trim/discard I/O operations active to each device. 1265.No See Sx ZFS I/O SCHEDULER . 1266. 1267.It Sy zfs_vdev_nia_delay Ns = Ns Sy 5 Pq uint 1268For non-interactive I/O (scrub, resilver, removal, initialize and rebuild), 1269the number of concurrently-active I/O operations is limited to 1270.Sy zfs_*_min_active , 1271unless the vdev is "idle". 1272When there are no interactive I/O operations active (synchronous or otherwise), 1273and 1274.Sy zfs_vdev_nia_delay 1275operations have completed since the last interactive operation, 1276then the vdev is considered to be "idle", 1277and the number of concurrently-active non-interactive operations is increased to 1278.Sy zfs_*_max_active . 1279.No See Sx ZFS I/O SCHEDULER . 1280. 1281.It Sy zfs_vdev_nia_credit Ns = Ns Sy 5 Pq uint 1282Some HDDs tend to prioritize sequential I/O so strongly, that concurrent 1283random I/O latency reaches several seconds. 1284On some HDDs this happens even if sequential I/O operations 1285are submitted one at a time, and so setting 1286.Sy zfs_*_max_active Ns = Sy 1 1287does not help. 1288To prevent non-interactive I/O, like scrub, 1289from monopolizing the device, no more than 1290.Sy zfs_vdev_nia_credit operations can be sent 1291while there are outstanding incomplete interactive operations. 1292This enforced wait ensures the HDD services the interactive I/O 1293within a reasonable amount of time. 1294.No See Sx ZFS I/O SCHEDULER . 1295. 1296.It Sy zfs_vdev_queue_depth_pct Ns = Ns Sy 1000 Ns % Pq uint 1297Maximum number of queued allocations per top-level vdev expressed as 1298a percentage of 1299.Sy zfs_vdev_async_write_max_active , 1300which allows the system to detect devices that are more capable 1301of handling allocations and to allocate more blocks to those devices. 1302This allows for dynamic allocation distribution when devices are imbalanced, 1303as fuller devices will tend to be slower than empty devices. 1304.Pp 1305Also see 1306.Sy zio_dva_throttle_enabled . 1307. 1308.It Sy zfs_vdev_def_queue_depth Ns = Ns Sy 32 Pq uint 1309Default queue depth for each vdev IO allocator. 1310Higher values allow for better coalescing of sequential writes before sending 1311them to the disk, but can increase transaction commit times. 1312. 1313.It Sy zfs_vdev_failfast_mask Ns = Ns Sy 1 Pq uint 1314Defines if the driver should retire on a given error type. 1315The following options may be bitwise-ored together: 1316.TS 1317box; 1318lbz r l l . 1319 Value Name Description 1320_ 1321 1 Device No driver retries on device errors 1322 2 Transport No driver retries on transport errors. 1323 4 Driver No driver retries on driver errors. 1324.TE 1325. 1326.It Sy zfs_expire_snapshot Ns = Ns Sy 300 Ns s Pq int 1327Time before expiring 1328.Pa .zfs/snapshot . 1329. 1330.It Sy zfs_admin_snapshot Ns = Ns Sy 0 Ns | Ns 1 Pq int 1331Allow the creation, removal, or renaming of entries in the 1332.Sy .zfs/snapshot 1333directory to cause the creation, destruction, or renaming of snapshots. 1334When enabled, this functionality works both locally and over NFS exports 1335which have the 1336.Em no_root_squash 1337option set. 1338. 1339.It Sy zfs_flags Ns = Ns Sy 0 Pq int 1340Set additional debugging flags. 1341The following flags may be bitwise-ored together: 1342.TS 1343box; 1344lbz r l l . 1345 Value Name Description 1346_ 1347 1 ZFS_DEBUG_DPRINTF Enable dprintf entries in the debug log. 1348* 2 ZFS_DEBUG_DBUF_VERIFY Enable extra dbuf verifications. 1349* 4 ZFS_DEBUG_DNODE_VERIFY Enable extra dnode verifications. 1350 8 ZFS_DEBUG_SNAPNAMES Enable snapshot name verification. 1351* 16 ZFS_DEBUG_MODIFY Check for illegally modified ARC buffers. 1352 64 ZFS_DEBUG_ZIO_FREE Enable verification of block frees. 1353 128 ZFS_DEBUG_HISTOGRAM_VERIFY Enable extra spacemap histogram verifications. 1354 256 ZFS_DEBUG_METASLAB_VERIFY Verify space accounting on disk matches in-memory \fBrange_trees\fP. 1355 512 ZFS_DEBUG_SET_ERROR Enable \fBSET_ERROR\fP and dprintf entries in the debug log. 1356 1024 ZFS_DEBUG_INDIRECT_REMAP Verify split blocks created by device removal. 1357 2048 ZFS_DEBUG_TRIM Verify TRIM ranges are always within the allocatable range tree. 1358 4096 ZFS_DEBUG_LOG_SPACEMAP Verify that the log summary is consistent with the spacemap log 1359 and enable \fBzfs_dbgmsgs\fP for metaslab loading and flushing. 1360.TE 1361.Sy \& * No Requires debug build . 1362. 1363.It Sy zfs_btree_verify_intensity Ns = Ns Sy 0 Pq uint 1364Enables btree verification. 1365The following settings are culminative: 1366.TS 1367box; 1368lbz r l l . 1369 Value Description 1370 1371 1 Verify height. 1372 2 Verify pointers from children to parent. 1373 3 Verify element counts. 1374 4 Verify element order. (expensive) 1375* 5 Verify unused memory is poisoned. (expensive) 1376.TE 1377.Sy \& * No Requires debug build . 1378. 1379.It Sy zfs_free_leak_on_eio Ns = Ns Sy 0 Ns | Ns 1 Pq int 1380If destroy encounters an 1381.Sy EIO 1382while reading metadata (e.g. indirect blocks), 1383space referenced by the missing metadata can not be freed. 1384Normally this causes the background destroy to become "stalled", 1385as it is unable to make forward progress. 1386While in this stalled state, all remaining space to free 1387from the error-encountering filesystem is "temporarily leaked". 1388Set this flag to cause it to ignore the 1389.Sy EIO , 1390permanently leak the space from indirect blocks that can not be read, 1391and continue to free everything else that it can. 1392.Pp 1393The default "stalling" behavior is useful if the storage partially 1394fails (i.e. some but not all I/O operations fail), and then later recovers. 1395In this case, we will be able to continue pool operations while it is 1396partially failed, and when it recovers, we can continue to free the 1397space, with no leaks. 1398Note, however, that this case is actually fairly rare. 1399.Pp 1400Typically pools either 1401.Bl -enum -compact -offset 4n -width "1." 1402.It 1403fail completely (but perhaps temporarily, 1404e.g. due to a top-level vdev going offline), or 1405.It 1406have localized, permanent errors (e.g. disk returns the wrong data 1407due to bit flip or firmware bug). 1408.El 1409In the former case, this setting does not matter because the 1410pool will be suspended and the sync thread will not be able to make 1411forward progress regardless. 1412In the latter, because the error is permanent, the best we can do 1413is leak the minimum amount of space, 1414which is what setting this flag will do. 1415It is therefore reasonable for this flag to normally be set, 1416but we chose the more conservative approach of not setting it, 1417so that there is no possibility of 1418leaking space in the "partial temporary" failure case. 1419. 1420.It Sy zfs_free_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1s Pc Pq uint 1421During a 1422.Nm zfs Cm destroy 1423operation using the 1424.Sy async_destroy 1425feature, 1426a minimum of this much time will be spent working on freeing blocks per TXG. 1427. 1428.It Sy zfs_obsolete_min_time_ms Ns = Ns Sy 500 Ns ms Pq uint 1429Similar to 1430.Sy zfs_free_min_time_ms , 1431but for cleanup of old indirection records for removed vdevs. 1432. 1433.It Sy zfs_immediate_write_sz Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq s64 1434Largest data block to write to the ZIL. 1435Larger blocks will be treated as if the dataset being written to had the 1436.Sy logbias Ns = Ns Sy throughput 1437property set. 1438. 1439.It Sy zfs_initialize_value Ns = Ns Sy 16045690984833335022 Po 0xDEADBEEFDEADBEEE Pc Pq u64 1440Pattern written to vdev free space by 1441.Xr zpool-initialize 8 . 1442. 1443.It Sy zfs_initialize_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 1444Size of writes used by 1445.Xr zpool-initialize 8 . 1446This option is used by the test suite. 1447. 1448.It Sy zfs_livelist_max_entries Ns = Ns Sy 500000 Po 5*10^5 Pc Pq u64 1449The threshold size (in block pointers) at which we create a new sub-livelist. 1450Larger sublists are more costly from a memory perspective but the fewer 1451sublists there are, the lower the cost of insertion. 1452. 1453.It Sy zfs_livelist_min_percent_shared Ns = Ns Sy 75 Ns % Pq int 1454If the amount of shared space between a snapshot and its clone drops below 1455this threshold, the clone turns off the livelist and reverts to the old 1456deletion method. 1457This is in place because livelists no long give us a benefit 1458once a clone has been overwritten enough. 1459. 1460.It Sy zfs_livelist_condense_new_alloc Ns = Ns Sy 0 Pq int 1461Incremented each time an extra ALLOC blkptr is added to a livelist entry while 1462it is being condensed. 1463This option is used by the test suite to track race conditions. 1464. 1465.It Sy zfs_livelist_condense_sync_cancel Ns = Ns Sy 0 Pq int 1466Incremented each time livelist condensing is canceled while in 1467.Fn spa_livelist_condense_sync . 1468This option is used by the test suite to track race conditions. 1469. 1470.It Sy zfs_livelist_condense_sync_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int 1471When set, the livelist condense process pauses indefinitely before 1472executing the synctask \(em 1473.Fn spa_livelist_condense_sync . 1474This option is used by the test suite to trigger race conditions. 1475. 1476.It Sy zfs_livelist_condense_zthr_cancel Ns = Ns Sy 0 Pq int 1477Incremented each time livelist condensing is canceled while in 1478.Fn spa_livelist_condense_cb . 1479This option is used by the test suite to track race conditions. 1480. 1481.It Sy zfs_livelist_condense_zthr_pause Ns = Ns Sy 0 Ns | Ns 1 Pq int 1482When set, the livelist condense process pauses indefinitely before 1483executing the open context condensing work in 1484.Fn spa_livelist_condense_cb . 1485This option is used by the test suite to trigger race conditions. 1486. 1487.It Sy zfs_lua_max_instrlimit Ns = Ns Sy 100000000 Po 10^8 Pc Pq u64 1488The maximum execution time limit that can be set for a ZFS channel program, 1489specified as a number of Lua instructions. 1490. 1491.It Sy zfs_lua_max_memlimit Ns = Ns Sy 104857600 Po 100 MiB Pc Pq u64 1492The maximum memory limit that can be set for a ZFS channel program, specified 1493in bytes. 1494. 1495.It Sy zfs_max_dataset_nesting Ns = Ns Sy 50 Pq int 1496The maximum depth of nested datasets. 1497This value can be tuned temporarily to 1498fix existing datasets that exceed the predefined limit. 1499. 1500.It Sy zfs_max_log_walking Ns = Ns Sy 5 Pq u64 1501The number of past TXGs that the flushing algorithm of the log spacemap 1502feature uses to estimate incoming log blocks. 1503. 1504.It Sy zfs_max_logsm_summary_length Ns = Ns Sy 10 Pq u64 1505Maximum number of rows allowed in the summary of the spacemap log. 1506. 1507.It Sy zfs_max_recordsize Ns = Ns Sy 16777216 Po 16 MiB Pc Pq uint 1508We currently support block sizes from 1509.Em 512 Po 512 B Pc No to Em 16777216 Po 16 MiB Pc . 1510The benefits of larger blocks, and thus larger I/O, 1511need to be weighed against the cost of COWing a giant block to modify one byte. 1512Additionally, very large blocks can have an impact on I/O latency, 1513and also potentially on the memory allocator. 1514Therefore, we formerly forbade creating blocks larger than 1M. 1515Larger blocks could be created by changing it, 1516and pools with larger blocks can always be imported and used, 1517regardless of this setting. 1518. 1519.It Sy zfs_allow_redacted_dataset_mount Ns = Ns Sy 0 Ns | Ns 1 Pq int 1520Allow datasets received with redacted send/receive to be mounted. 1521Normally disabled because these datasets may be missing key data. 1522. 1523.It Sy zfs_min_metaslabs_to_flush Ns = Ns Sy 1 Pq u64 1524Minimum number of metaslabs to flush per dirty TXG. 1525. 1526.It Sy zfs_metaslab_fragmentation_threshold Ns = Ns Sy 70 Ns % Pq uint 1527Allow metaslabs to keep their active state as long as their fragmentation 1528percentage is no more than this value. 1529An active metaslab that exceeds this threshold 1530will no longer keep its active status allowing better metaslabs to be selected. 1531. 1532.It Sy zfs_mg_fragmentation_threshold Ns = Ns Sy 95 Ns % Pq uint 1533Metaslab groups are considered eligible for allocations if their 1534fragmentation metric (measured as a percentage) is less than or equal to 1535this value. 1536If a metaslab group exceeds this threshold then it will be 1537skipped unless all metaslab groups within the metaslab class have also 1538crossed this threshold. 1539. 1540.It Sy zfs_mg_noalloc_threshold Ns = Ns Sy 0 Ns % Pq uint 1541Defines a threshold at which metaslab groups should be eligible for allocations. 1542The value is expressed as a percentage of free space 1543beyond which a metaslab group is always eligible for allocations. 1544If a metaslab group's free space is less than or equal to the 1545threshold, the allocator will avoid allocating to that group 1546unless all groups in the pool have reached the threshold. 1547Once all groups have reached the threshold, all groups are allowed to accept 1548allocations. 1549The default value of 1550.Sy 0 1551disables the feature and causes all metaslab groups to be eligible for 1552allocations. 1553.Pp 1554This parameter allows one to deal with pools having heavily imbalanced 1555vdevs such as would be the case when a new vdev has been added. 1556Setting the threshold to a non-zero percentage will stop allocations 1557from being made to vdevs that aren't filled to the specified percentage 1558and allow lesser filled vdevs to acquire more allocations than they 1559otherwise would under the old 1560.Sy zfs_mg_alloc_failures 1561facility. 1562. 1563.It Sy zfs_ddt_data_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int 1564If enabled, ZFS will place DDT data into the special allocation class. 1565. 1566.It Sy zfs_user_indirect_is_special Ns = Ns Sy 1 Ns | Ns 0 Pq int 1567If enabled, ZFS will place user data indirect blocks 1568into the special allocation class. 1569. 1570.It Sy zfs_multihost_history Ns = Ns Sy 0 Pq uint 1571Historical statistics for this many latest multihost updates will be available 1572in 1573.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /multihost . 1574. 1575.It Sy zfs_multihost_interval Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq u64 1576Used to control the frequency of multihost writes which are performed when the 1577.Sy multihost 1578pool property is on. 1579This is one of the factors used to determine the 1580length of the activity check during import. 1581.Pp 1582The multihost write period is 1583.Sy zfs_multihost_interval No / Sy leaf-vdevs . 1584On average a multihost write will be issued for each leaf vdev 1585every 1586.Sy zfs_multihost_interval 1587milliseconds. 1588In practice, the observed period can vary with the I/O load 1589and this observed value is the delay which is stored in the uberblock. 1590. 1591.It Sy zfs_multihost_import_intervals Ns = Ns Sy 20 Pq uint 1592Used to control the duration of the activity test on import. 1593Smaller values of 1594.Sy zfs_multihost_import_intervals 1595will reduce the import time but increase 1596the risk of failing to detect an active pool. 1597The total activity check time is never allowed to drop below one second. 1598.Pp 1599On import the activity check waits a minimum amount of time determined by 1600.Sy zfs_multihost_interval No \(mu Sy zfs_multihost_import_intervals , 1601or the same product computed on the host which last had the pool imported, 1602whichever is greater. 1603The activity check time may be further extended if the value of MMP 1604delay found in the best uberblock indicates actual multihost updates happened 1605at longer intervals than 1606.Sy zfs_multihost_interval . 1607A minimum of 1608.Em 100 ms 1609is enforced. 1610.Pp 1611.Sy 0 No is equivalent to Sy 1 . 1612. 1613.It Sy zfs_multihost_fail_intervals Ns = Ns Sy 10 Pq uint 1614Controls the behavior of the pool when multihost write failures or delays are 1615detected. 1616.Pp 1617When 1618.Sy 0 , 1619multihost write failures or delays are ignored. 1620The failures will still be reported to the ZED which depending on 1621its configuration may take action such as suspending the pool or offlining a 1622device. 1623.Pp 1624Otherwise, the pool will be suspended if 1625.Sy zfs_multihost_fail_intervals No \(mu Sy zfs_multihost_interval 1626milliseconds pass without a successful MMP write. 1627This guarantees the activity test will see MMP writes if the pool is imported. 1628.Sy 1 No is equivalent to Sy 2 ; 1629this is necessary to prevent the pool from being suspended 1630due to normal, small I/O latency variations. 1631. 1632.It Sy zfs_no_scrub_io Ns = Ns Sy 0 Ns | Ns 1 Pq int 1633Set to disable scrub I/O. 1634This results in scrubs not actually scrubbing data and 1635simply doing a metadata crawl of the pool instead. 1636. 1637.It Sy zfs_no_scrub_prefetch Ns = Ns Sy 0 Ns | Ns 1 Pq int 1638Set to disable block prefetching for scrubs. 1639. 1640.It Sy zfs_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int 1641Disable cache flush operations on disks when writing. 1642Setting this will cause pool corruption on power loss 1643if a volatile out-of-order write cache is enabled. 1644. 1645.It Sy zfs_nopwrite_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 1646Allow no-operation writes. 1647The occurrence of nopwrites will further depend on other pool properties 1648.Pq i.a. the checksumming and compression algorithms . 1649. 1650.It Sy zfs_dmu_offset_next_sync Ns = Ns Sy 1 Ns | Ns 0 Pq int 1651Enable forcing TXG sync to find holes. 1652When enabled forces ZFS to sync data when 1653.Sy SEEK_HOLE No or Sy SEEK_DATA 1654flags are used allowing holes in a file to be accurately reported. 1655When disabled holes will not be reported in recently dirtied files. 1656. 1657.It Sy zfs_pd_bytes_max Ns = Ns Sy 52428800 Ns B Po 50 MiB Pc Pq int 1658The number of bytes which should be prefetched during a pool traversal, like 1659.Nm zfs Cm send 1660or other data crawling operations. 1661. 1662.It Sy zfs_traverse_indirect_prefetch_limit Ns = Ns Sy 32 Pq uint 1663The number of blocks pointed by indirect (non-L0) block which should be 1664prefetched during a pool traversal, like 1665.Nm zfs Cm send 1666or other data crawling operations. 1667. 1668.It Sy zfs_per_txg_dirty_frees_percent Ns = Ns Sy 30 Ns % Pq u64 1669Control percentage of dirtied indirect blocks from frees allowed into one TXG. 1670After this threshold is crossed, additional frees will wait until the next TXG. 1671.Sy 0 No disables this throttle . 1672. 1673.It Sy zfs_prefetch_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 1674Disable predictive prefetch. 1675Note that it leaves "prescient" prefetch 1676.Pq for, e.g., Nm zfs Cm send 1677intact. 1678Unlike predictive prefetch, prescient prefetch never issues I/O 1679that ends up not being needed, so it can't hurt performance. 1680. 1681.It Sy zfs_qat_checksum_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 1682Disable QAT hardware acceleration for SHA256 checksums. 1683May be unset after the ZFS modules have been loaded to initialize the QAT 1684hardware as long as support is compiled in and the QAT driver is present. 1685. 1686.It Sy zfs_qat_compress_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 1687Disable QAT hardware acceleration for gzip compression. 1688May be unset after the ZFS modules have been loaded to initialize the QAT 1689hardware as long as support is compiled in and the QAT driver is present. 1690. 1691.It Sy zfs_qat_encrypt_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 1692Disable QAT hardware acceleration for AES-GCM encryption. 1693May be unset after the ZFS modules have been loaded to initialize the QAT 1694hardware as long as support is compiled in and the QAT driver is present. 1695. 1696.It Sy zfs_vnops_read_chunk_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 1697Bytes to read per chunk. 1698. 1699.It Sy zfs_read_history Ns = Ns Sy 0 Pq uint 1700Historical statistics for this many latest reads will be available in 1701.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /reads . 1702. 1703.It Sy zfs_read_history_hits Ns = Ns Sy 0 Ns | Ns 1 Pq int 1704Include cache hits in read history 1705. 1706.It Sy zfs_rebuild_max_segment Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq u64 1707Maximum read segment size to issue when sequentially resilvering a 1708top-level vdev. 1709. 1710.It Sy zfs_rebuild_scrub_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 1711Automatically start a pool scrub when the last active sequential resilver 1712completes in order to verify the checksums of all blocks which have been 1713resilvered. 1714This is enabled by default and strongly recommended. 1715. 1716.It Sy zfs_rebuild_vdev_limit Ns = Ns Sy 67108864 Ns B Po 64 MiB Pc Pq u64 1717Maximum amount of I/O that can be concurrently issued for a sequential 1718resilver per leaf device, given in bytes. 1719. 1720.It Sy zfs_reconstruct_indirect_combinations_max Ns = Ns Sy 4096 Pq int 1721If an indirect split block contains more than this many possible unique 1722combinations when being reconstructed, consider it too computationally 1723expensive to check them all. 1724Instead, try at most this many randomly selected 1725combinations each time the block is accessed. 1726This allows all segment copies to participate fairly 1727in the reconstruction when all combinations 1728cannot be checked and prevents repeated use of one bad copy. 1729. 1730.It Sy zfs_recover Ns = Ns Sy 0 Ns | Ns 1 Pq int 1731Set to attempt to recover from fatal errors. 1732This should only be used as a last resort, 1733as it typically results in leaked space, or worse. 1734. 1735.It Sy zfs_removal_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int 1736Ignore hard I/O errors during device removal. 1737When set, if a device encounters a hard I/O error during the removal process 1738the removal will not be cancelled. 1739This can result in a normally recoverable block becoming permanently damaged 1740and is hence not recommended. 1741This should only be used as a last resort when the 1742pool cannot be returned to a healthy state prior to removing the device. 1743. 1744.It Sy zfs_removal_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq uint 1745This is used by the test suite so that it can ensure that certain actions 1746happen while in the middle of a removal. 1747. 1748.It Sy zfs_remove_max_segment Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 1749The largest contiguous segment that we will attempt to allocate when removing 1750a device. 1751If there is a performance problem with attempting to allocate large blocks, 1752consider decreasing this. 1753The default value is also the maximum. 1754. 1755.It Sy zfs_resilver_disable_defer Ns = Ns Sy 0 Ns | Ns 1 Pq int 1756Ignore the 1757.Sy resilver_defer 1758feature, causing an operation that would start a resilver to 1759immediately restart the one in progress. 1760. 1761.It Sy zfs_resilver_min_time_ms Ns = Ns Sy 3000 Ns ms Po 3 s Pc Pq uint 1762Resilvers are processed by the sync thread. 1763While resilvering, it will spend at least this much time 1764working on a resilver between TXG flushes. 1765. 1766.It Sy zfs_scan_ignore_errors Ns = Ns Sy 0 Ns | Ns 1 Pq int 1767If set, remove the DTL (dirty time list) upon completion of a pool scan (scrub), 1768even if there were unrepairable errors. 1769Intended to be used during pool repair or recovery to 1770stop resilvering when the pool is next imported. 1771. 1772.It Sy zfs_scrub_min_time_ms Ns = Ns Sy 1000 Ns ms Po 1 s Pc Pq uint 1773Scrubs are processed by the sync thread. 1774While scrubbing, it will spend at least this much time 1775working on a scrub between TXG flushes. 1776. 1777.It Sy zfs_scrub_error_blocks_per_txg Ns = Ns Sy 4096 Pq uint 1778Error blocks to be scrubbed in one txg. 1779. 1780.It Sy zfs_scan_checkpoint_intval Ns = Ns Sy 7200 Ns s Po 2 hour Pc Pq uint 1781To preserve progress across reboots, the sequential scan algorithm periodically 1782needs to stop metadata scanning and issue all the verification I/O to disk. 1783The frequency of this flushing is determined by this tunable. 1784. 1785.It Sy zfs_scan_fill_weight Ns = Ns Sy 3 Pq uint 1786This tunable affects how scrub and resilver I/O segments are ordered. 1787A higher number indicates that we care more about how filled in a segment is, 1788while a lower number indicates we care more about the size of the extent without 1789considering the gaps within a segment. 1790This value is only tunable upon module insertion. 1791Changing the value afterwards will have no effect on scrub or resilver 1792performance. 1793. 1794.It Sy zfs_scan_issue_strategy Ns = Ns Sy 0 Pq uint 1795Determines the order that data will be verified while scrubbing or resilvering: 1796.Bl -tag -compact -offset 4n -width "a" 1797.It Sy 1 1798Data will be verified as sequentially as possible, given the 1799amount of memory reserved for scrubbing 1800.Pq see Sy zfs_scan_mem_lim_fact . 1801This may improve scrub performance if the pool's data is very fragmented. 1802.It Sy 2 1803The largest mostly-contiguous chunk of found data will be verified first. 1804By deferring scrubbing of small segments, we may later find adjacent data 1805to coalesce and increase the segment size. 1806.It Sy 0 1807.No Use strategy Sy 1 No during normal verification 1808.No and strategy Sy 2 No while taking a checkpoint . 1809.El 1810. 1811.It Sy zfs_scan_legacy Ns = Ns Sy 0 Ns | Ns 1 Pq int 1812If unset, indicates that scrubs and resilvers will gather metadata in 1813memory before issuing sequential I/O. 1814Otherwise indicates that the legacy algorithm will be used, 1815where I/O is initiated as soon as it is discovered. 1816Unsetting will not affect scrubs or resilvers that are already in progress. 1817. 1818.It Sy zfs_scan_max_ext_gap Ns = Ns Sy 2097152 Ns B Po 2 MiB Pc Pq int 1819Sets the largest gap in bytes between scrub/resilver I/O operations 1820that will still be considered sequential for sorting purposes. 1821Changing this value will not 1822affect scrubs or resilvers that are already in progress. 1823. 1824.It Sy zfs_scan_mem_lim_fact Ns = Ns Sy 20 Ns ^-1 Pq uint 1825Maximum fraction of RAM used for I/O sorting by sequential scan algorithm. 1826This tunable determines the hard limit for I/O sorting memory usage. 1827When the hard limit is reached we stop scanning metadata and start issuing 1828data verification I/O. 1829This is done until we get below the soft limit. 1830. 1831.It Sy zfs_scan_mem_lim_soft_fact Ns = Ns Sy 20 Ns ^-1 Pq uint 1832The fraction of the hard limit used to determined the soft limit for I/O sorting 1833by the sequential scan algorithm. 1834When we cross this limit from below no action is taken. 1835When we cross this limit from above it is because we are issuing verification 1836I/O. 1837In this case (unless the metadata scan is done) we stop issuing verification I/O 1838and start scanning metadata again until we get to the hard limit. 1839. 1840.It Sy zfs_scan_report_txgs Ns = Ns Sy 0 Ns | Ns 1 Pq uint 1841When reporting resilver throughput and estimated completion time use the 1842performance observed over roughly the last 1843.Sy zfs_scan_report_txgs 1844TXGs. 1845When set to zero performance is calculated over the time between checkpoints. 1846. 1847.It Sy zfs_scan_strict_mem_lim Ns = Ns Sy 0 Ns | Ns 1 Pq int 1848Enforce tight memory limits on pool scans when a sequential scan is in progress. 1849When disabled, the memory limit may be exceeded by fast disks. 1850. 1851.It Sy zfs_scan_suspend_progress Ns = Ns Sy 0 Ns | Ns 1 Pq int 1852Freezes a scrub/resilver in progress without actually pausing it. 1853Intended for testing/debugging. 1854. 1855.It Sy zfs_scan_vdev_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int 1856Maximum amount of data that can be concurrently issued at once for scrubs and 1857resilvers per leaf device, given in bytes. 1858. 1859.It Sy zfs_send_corrupt_data Ns = Ns Sy 0 Ns | Ns 1 Pq int 1860Allow sending of corrupt data (ignore read/checksum errors when sending). 1861. 1862.It Sy zfs_send_unmodified_spill_blocks Ns = Ns Sy 1 Ns | Ns 0 Pq int 1863Include unmodified spill blocks in the send stream. 1864Under certain circumstances, previous versions of ZFS could incorrectly 1865remove the spill block from an existing object. 1866Including unmodified copies of the spill blocks creates a backwards-compatible 1867stream which will recreate a spill block if it was incorrectly removed. 1868. 1869.It Sy zfs_send_no_prefetch_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint 1870The fill fraction of the 1871.Nm zfs Cm send 1872internal queues. 1873The fill fraction controls the timing with which internal threads are woken up. 1874. 1875.It Sy zfs_send_no_prefetch_queue_length Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint 1876The maximum number of bytes allowed in 1877.Nm zfs Cm send Ns 's 1878internal queues. 1879. 1880.It Sy zfs_send_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint 1881The fill fraction of the 1882.Nm zfs Cm send 1883prefetch queue. 1884The fill fraction controls the timing with which internal threads are woken up. 1885. 1886.It Sy zfs_send_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 1887The maximum number of bytes allowed that will be prefetched by 1888.Nm zfs Cm send . 1889This value must be at least twice the maximum block size in use. 1890. 1891.It Sy zfs_recv_queue_ff Ns = Ns Sy 20 Ns ^\-1 Pq uint 1892The fill fraction of the 1893.Nm zfs Cm receive 1894queue. 1895The fill fraction controls the timing with which internal threads are woken up. 1896. 1897.It Sy zfs_recv_queue_length Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq uint 1898The maximum number of bytes allowed in the 1899.Nm zfs Cm receive 1900queue. 1901This value must be at least twice the maximum block size in use. 1902. 1903.It Sy zfs_recv_write_batch_size Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint 1904The maximum amount of data, in bytes, that 1905.Nm zfs Cm receive 1906will write in one DMU transaction. 1907This is the uncompressed size, even when receiving a compressed send stream. 1908This setting will not reduce the write size below a single block. 1909Capped at a maximum of 1910.Sy 32 MiB . 1911. 1912.It Sy zfs_recv_best_effort_corrective Ns = Ns Sy 0 Pq int 1913When this variable is set to non-zero a corrective receive: 1914.Bl -enum -compact -offset 4n -width "1." 1915.It 1916Does not enforce the restriction of source & destination snapshot GUIDs 1917matching. 1918.It 1919If there is an error during healing, the healing receive is not 1920terminated instead it moves on to the next record. 1921.El 1922. 1923.It Sy zfs_override_estimate_recordsize Ns = Ns Sy 0 Ns | Ns 1 Pq uint 1924Setting this variable overrides the default logic for estimating block 1925sizes when doing a 1926.Nm zfs Cm send . 1927The default heuristic is that the average block size 1928will be the current recordsize. 1929Override this value if most data in your dataset is not of that size 1930and you require accurate zfs send size estimates. 1931. 1932.It Sy zfs_sync_pass_deferred_free Ns = Ns Sy 2 Pq uint 1933Flushing of data to disk is done in passes. 1934Defer frees starting in this pass. 1935. 1936.It Sy zfs_spa_discard_memory_limit Ns = Ns Sy 16777216 Ns B Po 16 MiB Pc Pq int 1937Maximum memory used for prefetching a checkpoint's space map on each 1938vdev while discarding the checkpoint. 1939. 1940.It Sy zfs_special_class_metadata_reserve_pct Ns = Ns Sy 25 Ns % Pq uint 1941Only allow small data blocks to be allocated on the special and dedup vdev 1942types when the available free space percentage on these vdevs exceeds this 1943value. 1944This ensures reserved space is available for pool metadata as the 1945special vdevs approach capacity. 1946. 1947.It Sy zfs_sync_pass_dont_compress Ns = Ns Sy 8 Pq uint 1948Starting in this sync pass, disable compression (including of metadata). 1949With the default setting, in practice, we don't have this many sync passes, 1950so this has no effect. 1951.Pp 1952The original intent was that disabling compression would help the sync passes 1953to converge. 1954However, in practice, disabling compression increases 1955the average number of sync passes; because when we turn compression off, 1956many blocks' size will change, and thus we have to re-allocate 1957(not overwrite) them. 1958It also increases the number of 1959.Em 128 KiB 1960allocations (e.g. for indirect blocks and spacemaps) 1961because these will not be compressed. 1962The 1963.Em 128 KiB 1964allocations are especially detrimental to performance 1965on highly fragmented systems, which may have very few free segments of this 1966size, 1967and may need to load new metaslabs to satisfy these allocations. 1968. 1969.It Sy zfs_sync_pass_rewrite Ns = Ns Sy 2 Pq uint 1970Rewrite new block pointers starting in this pass. 1971. 1972.It Sy zfs_sync_taskq_batch_pct Ns = Ns Sy 75 Ns % Pq int 1973This controls the number of threads used by 1974.Sy dp_sync_taskq . 1975The default value of 1976.Sy 75% 1977will create a maximum of one thread per CPU. 1978. 1979.It Sy zfs_trim_extent_bytes_max Ns = Ns Sy 134217728 Ns B Po 128 MiB Pc Pq uint 1980Maximum size of TRIM command. 1981Larger ranges will be split into chunks no larger than this value before 1982issuing. 1983. 1984.It Sy zfs_trim_extent_bytes_min Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint 1985Minimum size of TRIM commands. 1986TRIM ranges smaller than this will be skipped, 1987unless they're part of a larger range which was chunked. 1988This is done because it's common for these small TRIMs 1989to negatively impact overall performance. 1990. 1991.It Sy zfs_trim_metaslab_skip Ns = Ns Sy 0 Ns | Ns 1 Pq uint 1992Skip uninitialized metaslabs during the TRIM process. 1993This option is useful for pools constructed from large thinly-provisioned 1994devices 1995where TRIM operations are slow. 1996As a pool ages, an increasing fraction of the pool's metaslabs 1997will be initialized, progressively degrading the usefulness of this option. 1998This setting is stored when starting a manual TRIM and will 1999persist for the duration of the requested TRIM. 2000. 2001.It Sy zfs_trim_queue_limit Ns = Ns Sy 10 Pq uint 2002Maximum number of queued TRIMs outstanding per leaf vdev. 2003The number of concurrent TRIM commands issued to the device is controlled by 2004.Sy zfs_vdev_trim_min_active No and Sy zfs_vdev_trim_max_active . 2005. 2006.It Sy zfs_trim_txg_batch Ns = Ns Sy 32 Pq uint 2007The number of transaction groups' worth of frees which should be aggregated 2008before TRIM operations are issued to the device. 2009This setting represents a trade-off between issuing larger, 2010more efficient TRIM operations and the delay 2011before the recently trimmed space is available for use by the device. 2012.Pp 2013Increasing this value will allow frees to be aggregated for a longer time. 2014This will result is larger TRIM operations and potentially increased memory 2015usage. 2016Decreasing this value will have the opposite effect. 2017The default of 2018.Sy 32 2019was determined to be a reasonable compromise. 2020. 2021.It Sy zfs_txg_history Ns = Ns Sy 0 Pq uint 2022Historical statistics for this many latest TXGs will be available in 2023.Pa /proc/spl/kstat/zfs/ Ns Ao Ar pool Ac Ns Pa /TXGs . 2024. 2025.It Sy zfs_txg_timeout Ns = Ns Sy 5 Ns s Pq uint 2026Flush dirty data to disk at least every this many seconds (maximum TXG 2027duration). 2028. 2029.It Sy zfs_vdev_aggregation_limit Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq uint 2030Max vdev I/O aggregation size. 2031. 2032.It Sy zfs_vdev_aggregation_limit_non_rotating Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint 2033Max vdev I/O aggregation size for non-rotating media. 2034. 2035.It Sy zfs_vdev_mirror_rotating_inc Ns = Ns Sy 0 Pq int 2036A number by which the balancing algorithm increments the load calculation for 2037the purpose of selecting the least busy mirror member when an I/O operation 2038immediately follows its predecessor on rotational vdevs 2039for the purpose of making decisions based on load. 2040. 2041.It Sy zfs_vdev_mirror_rotating_seek_inc Ns = Ns Sy 5 Pq int 2042A number by which the balancing algorithm increments the load calculation for 2043the purpose of selecting the least busy mirror member when an I/O operation 2044lacks locality as defined by 2045.Sy zfs_vdev_mirror_rotating_seek_offset . 2046Operations within this that are not immediately following the previous operation 2047are incremented by half. 2048. 2049.It Sy zfs_vdev_mirror_rotating_seek_offset Ns = Ns Sy 1048576 Ns B Po 1 MiB Pc Pq int 2050The maximum distance for the last queued I/O operation in which 2051the balancing algorithm considers an operation to have locality. 2052.No See Sx ZFS I/O SCHEDULER . 2053. 2054.It Sy zfs_vdev_mirror_non_rotating_inc Ns = Ns Sy 0 Pq int 2055A number by which the balancing algorithm increments the load calculation for 2056the purpose of selecting the least busy mirror member on non-rotational vdevs 2057when I/O operations do not immediately follow one another. 2058. 2059.It Sy zfs_vdev_mirror_non_rotating_seek_inc Ns = Ns Sy 1 Pq int 2060A number by which the balancing algorithm increments the load calculation for 2061the purpose of selecting the least busy mirror member when an I/O operation 2062lacks 2063locality as defined by the 2064.Sy zfs_vdev_mirror_rotating_seek_offset . 2065Operations within this that are not immediately following the previous operation 2066are incremented by half. 2067. 2068.It Sy zfs_vdev_read_gap_limit Ns = Ns Sy 32768 Ns B Po 32 KiB Pc Pq uint 2069Aggregate read I/O operations if the on-disk gap between them is within this 2070threshold. 2071. 2072.It Sy zfs_vdev_write_gap_limit Ns = Ns Sy 4096 Ns B Po 4 KiB Pc Pq uint 2073Aggregate write I/O operations if the on-disk gap between them is within this 2074threshold. 2075. 2076.It Sy zfs_vdev_raidz_impl Ns = Ns Sy fastest Pq string 2077Select the raidz parity implementation to use. 2078.Pp 2079Variants that don't depend on CPU-specific features 2080may be selected on module load, as they are supported on all systems. 2081The remaining options may only be set after the module is loaded, 2082as they are available only if the implementations are compiled in 2083and supported on the running system. 2084.Pp 2085Once the module is loaded, 2086.Pa /sys/module/zfs/parameters/zfs_vdev_raidz_impl 2087will show the available options, 2088with the currently selected one enclosed in square brackets. 2089.Pp 2090.TS 2091lb l l . 2092fastest selected by built-in benchmark 2093original original implementation 2094scalar scalar implementation 2095sse2 SSE2 instruction set 64-bit x86 2096ssse3 SSSE3 instruction set 64-bit x86 2097avx2 AVX2 instruction set 64-bit x86 2098avx512f AVX512F instruction set 64-bit x86 2099avx512bw AVX512F & AVX512BW instruction sets 64-bit x86 2100aarch64_neon NEON Aarch64/64-bit ARMv8 2101aarch64_neonx2 NEON with more unrolling Aarch64/64-bit ARMv8 2102powerpc_altivec Altivec PowerPC 2103.TE 2104. 2105.It Sy zfs_vdev_scheduler Pq charp 2106.Sy DEPRECATED . 2107Prints warning to kernel log for compatibility. 2108. 2109.It Sy zfs_zevent_len_max Ns = Ns Sy 512 Pq uint 2110Max event queue length. 2111Events in the queue can be viewed with 2112.Xr zpool-events 8 . 2113. 2114.It Sy zfs_zevent_retain_max Ns = Ns Sy 2000 Pq int 2115Maximum recent zevent records to retain for duplicate checking. 2116Setting this to 2117.Sy 0 2118disables duplicate detection. 2119. 2120.It Sy zfs_zevent_retain_expire_secs Ns = Ns Sy 900 Ns s Po 15 min Pc Pq int 2121Lifespan for a recent ereport that was retained for duplicate checking. 2122. 2123.It Sy zfs_zil_clean_taskq_maxalloc Ns = Ns Sy 1048576 Pq int 2124The maximum number of taskq entries that are allowed to be cached. 2125When this limit is exceeded transaction records (itxs) 2126will be cleaned synchronously. 2127. 2128.It Sy zfs_zil_clean_taskq_minalloc Ns = Ns Sy 1024 Pq int 2129The number of taskq entries that are pre-populated when the taskq is first 2130created and are immediately available for use. 2131. 2132.It Sy zfs_zil_clean_taskq_nthr_pct Ns = Ns Sy 100 Ns % Pq int 2133This controls the number of threads used by 2134.Sy dp_zil_clean_taskq . 2135The default value of 2136.Sy 100% 2137will create a maximum of one thread per cpu. 2138. 2139.It Sy zil_maxblocksize Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint 2140This sets the maximum block size used by the ZIL. 2141On very fragmented pools, lowering this 2142.Pq typically to Sy 36 KiB 2143can improve performance. 2144. 2145.It Sy zil_min_commit_timeout Ns = Ns Sy 5000 Pq u64 2146This sets the minimum delay in nanoseconds ZIL care to delay block commit, 2147waiting for more records. 2148If ZIL writes are too fast, kernel may not be able sleep for so short interval, 2149increasing log latency above allowed by 2150.Sy zfs_commit_timeout_pct . 2151. 2152.It Sy zil_nocacheflush Ns = Ns Sy 0 Ns | Ns 1 Pq int 2153Disable the cache flush commands that are normally sent to disk by 2154the ZIL after an LWB write has completed. 2155Setting this will cause ZIL corruption on power loss 2156if a volatile out-of-order write cache is enabled. 2157. 2158.It Sy zil_replay_disable Ns = Ns Sy 0 Ns | Ns 1 Pq int 2159Disable intent logging replay. 2160Can be disabled for recovery from corrupted ZIL. 2161. 2162.It Sy zil_slog_bulk Ns = Ns Sy 786432 Ns B Po 768 KiB Pc Pq u64 2163Limit SLOG write size per commit executed with synchronous priority. 2164Any writes above that will be executed with lower (asynchronous) priority 2165to limit potential SLOG device abuse by single active ZIL writer. 2166. 2167.It Sy zfs_zil_saxattr Ns = Ns Sy 1 Ns | Ns 0 Pq int 2168Setting this tunable to zero disables ZIL logging of new 2169.Sy xattr Ns = Ns Sy sa 2170records if the 2171.Sy org.openzfs:zilsaxattr 2172feature is enabled on the pool. 2173This would only be necessary to work around bugs in the ZIL logging or replay 2174code for this record type. 2175The tunable has no effect if the feature is disabled. 2176. 2177.It Sy zfs_embedded_slog_min_ms Ns = Ns Sy 64 Pq uint 2178Usually, one metaslab from each normal-class vdev is dedicated for use by 2179the ZIL to log synchronous writes. 2180However, if there are fewer than 2181.Sy zfs_embedded_slog_min_ms 2182metaslabs in the vdev, this functionality is disabled. 2183This ensures that we don't set aside an unreasonable amount of space for the 2184ZIL. 2185. 2186.It Sy zstd_earlyabort_pass Ns = Ns Sy 1 Pq uint 2187Whether heuristic for detection of incompressible data with zstd levels >= 3 2188using LZ4 and zstd-1 passes is enabled. 2189. 2190.It Sy zstd_abort_size Ns = Ns Sy 131072 Pq uint 2191Minimal uncompressed size (inclusive) of a record before the early abort 2192heuristic will be attempted. 2193. 2194.It Sy zio_deadman_log_all Ns = Ns Sy 0 Ns | Ns 1 Pq int 2195If non-zero, the zio deadman will produce debugging messages 2196.Pq see Sy zfs_dbgmsg_enable 2197for all zios, rather than only for leaf zios possessing a vdev. 2198This is meant to be used by developers to gain 2199diagnostic information for hang conditions which don't involve a mutex 2200or other locking primitive: typically conditions in which a thread in 2201the zio pipeline is looping indefinitely. 2202. 2203.It Sy zio_slow_io_ms Ns = Ns Sy 30000 Ns ms Po 30 s Pc Pq int 2204When an I/O operation takes more than this much time to complete, 2205it's marked as slow. 2206Each slow operation causes a delay zevent. 2207Slow I/O counters can be seen with 2208.Nm zpool Cm status Fl s . 2209. 2210.It Sy zio_dva_throttle_enabled Ns = Ns Sy 1 Ns | Ns 0 Pq int 2211Throttle block allocations in the I/O pipeline. 2212This allows for dynamic allocation distribution when devices are imbalanced. 2213When enabled, the maximum number of pending allocations per top-level vdev 2214is limited by 2215.Sy zfs_vdev_queue_depth_pct . 2216. 2217.It Sy zfs_xattr_compat Ns = Ns 0 Ns | Ns 1 Pq int 2218Control the naming scheme used when setting new xattrs in the user namespace. 2219If 2220.Sy 0 2221.Pq the default on Linux , 2222user namespace xattr names are prefixed with the namespace, to be backwards 2223compatible with previous versions of ZFS on Linux. 2224If 2225.Sy 1 2226.Pq the default on Fx , 2227user namespace xattr names are not prefixed, to be backwards compatible with 2228previous versions of ZFS on illumos and 2229.Fx . 2230.Pp 2231Either naming scheme can be read on this and future versions of ZFS, regardless 2232of this tunable, but legacy ZFS on illumos or 2233.Fx 2234are unable to read user namespace xattrs written in the Linux format, and 2235legacy versions of ZFS on Linux are unable to read user namespace xattrs written 2236in the legacy ZFS format. 2237.Pp 2238An existing xattr with the alternate naming scheme is removed when overwriting 2239the xattr so as to not accumulate duplicates. 2240. 2241.It Sy zio_requeue_io_start_cut_in_line Ns = Ns Sy 0 Ns | Ns 1 Pq int 2242Prioritize requeued I/O. 2243. 2244.It Sy zio_taskq_batch_pct Ns = Ns Sy 80 Ns % Pq uint 2245Percentage of online CPUs which will run a worker thread for I/O. 2246These workers are responsible for I/O work such as compression and 2247checksum calculations. 2248Fractional number of CPUs will be rounded down. 2249.Pp 2250The default value of 2251.Sy 80% 2252was chosen to avoid using all CPUs which can result in 2253latency issues and inconsistent application performance, 2254especially when slower compression and/or checksumming is enabled. 2255. 2256.It Sy zio_taskq_batch_tpq Ns = Ns Sy 0 Pq uint 2257Number of worker threads per taskq. 2258Lower values improve I/O ordering and CPU utilization, 2259while higher reduces lock contention. 2260.Pp 2261If 2262.Sy 0 , 2263generate a system-dependent value close to 6 threads per taskq. 2264. 2265.It Sy zvol_inhibit_dev Ns = Ns Sy 0 Ns | Ns 1 Pq uint 2266Do not create zvol device nodes. 2267This may slightly improve startup time on 2268systems with a very large number of zvols. 2269. 2270.It Sy zvol_major Ns = Ns Sy 230 Pq uint 2271Major number for zvol block devices. 2272. 2273.It Sy zvol_max_discard_blocks Ns = Ns Sy 16384 Pq long 2274Discard (TRIM) operations done on zvols will be done in batches of this 2275many blocks, where block size is determined by the 2276.Sy volblocksize 2277property of a zvol. 2278. 2279.It Sy zvol_prefetch_bytes Ns = Ns Sy 131072 Ns B Po 128 KiB Pc Pq uint 2280When adding a zvol to the system, prefetch this many bytes 2281from the start and end of the volume. 2282Prefetching these regions of the volume is desirable, 2283because they are likely to be accessed immediately by 2284.Xr blkid 8 2285or the kernel partitioner. 2286. 2287.It Sy zvol_request_sync Ns = Ns Sy 0 Ns | Ns 1 Pq uint 2288When processing I/O requests for a zvol, submit them synchronously. 2289This effectively limits the queue depth to 2290.Em 1 2291for each I/O submitter. 2292When unset, requests are handled asynchronously by a thread pool. 2293The number of requests which can be handled concurrently is controlled by 2294.Sy zvol_threads . 2295.Sy zvol_request_sync 2296is ignored when running on a kernel that supports block multiqueue 2297.Pq Li blk-mq . 2298. 2299.It Sy zvol_threads Ns = Ns Sy 0 Pq uint 2300The number of system wide threads to use for processing zvol block IOs. 2301If 2302.Sy 0 2303(the default) then internally set 2304.Sy zvol_threads 2305to the number of CPUs present or 32 (whichever is greater). 2306. 2307.It Sy zvol_blk_mq_threads Ns = Ns Sy 0 Pq uint 2308The number of threads per zvol to use for queuing IO requests. 2309This parameter will only appear if your kernel supports 2310.Li blk-mq 2311and is only read and assigned to a zvol at zvol load time. 2312If 2313.Sy 0 2314(the default) then internally set 2315.Sy zvol_blk_mq_threads 2316to the number of CPUs present. 2317. 2318.It Sy zvol_use_blk_mq Ns = Ns Sy 0 Ns | Ns 1 Pq uint 2319Set to 2320.Sy 1 2321to use the 2322.Li blk-mq 2323API for zvols. 2324Set to 2325.Sy 0 2326(the default) to use the legacy zvol APIs. 2327This setting can give better or worse zvol performance depending on 2328the workload. 2329This parameter will only appear if your kernel supports 2330.Li blk-mq 2331and is only read and assigned to a zvol at zvol load time. 2332. 2333.It Sy zvol_blk_mq_blocks_per_thread Ns = Ns Sy 8 Pq uint 2334If 2335.Sy zvol_use_blk_mq 2336is enabled, then process this number of 2337.Sy volblocksize Ns -sized blocks per zvol thread. 2338This tunable can be use to favor better performance for zvol reads (lower 2339values) or writes (higher values). 2340If set to 2341.Sy 0 , 2342then the zvol layer will process the maximum number of blocks 2343per thread that it can. 2344This parameter will only appear if your kernel supports 2345.Li blk-mq 2346and is only applied at each zvol's load time. 2347. 2348.It Sy zvol_blk_mq_queue_depth Ns = Ns Sy 0 Pq uint 2349The queue_depth value for the zvol 2350.Li blk-mq 2351interface. 2352This parameter will only appear if your kernel supports 2353.Li blk-mq 2354and is only applied at each zvol's load time. 2355If 2356.Sy 0 2357(the default) then use the kernel's default queue depth. 2358Values are clamped to the kernel's 2359.Dv BLKDEV_MIN_RQ 2360and 2361.Dv BLKDEV_MAX_RQ Ns / Ns Dv BLKDEV_DEFAULT_RQ 2362limits. 2363. 2364.It Sy zvol_volmode Ns = Ns Sy 1 Pq uint 2365Defines zvol block devices behaviour when 2366.Sy volmode Ns = Ns Sy default : 2367.Bl -tag -compact -offset 4n -width "a" 2368.It Sy 1 2369.No equivalent to Sy full 2370.It Sy 2 2371.No equivalent to Sy dev 2372.It Sy 3 2373.No equivalent to Sy none 2374.El 2375. 2376.It Sy zvol_enforce_quotas Ns = Ns Sy 0 Ns | Ns 1 Pq uint 2377Enable strict ZVOL quota enforcement. 2378The strict quota enforcement may have a performance impact. 2379.El 2380. 2381.Sh ZFS I/O SCHEDULER 2382ZFS issues I/O operations to leaf vdevs to satisfy and complete I/O operations. 2383The scheduler determines when and in what order those operations are issued. 2384The scheduler divides operations into five I/O classes, 2385prioritized in the following order: sync read, sync write, async read, 2386async write, and scrub/resilver. 2387Each queue defines the minimum and maximum number of concurrent operations 2388that may be issued to the device. 2389In addition, the device has an aggregate maximum, 2390.Sy zfs_vdev_max_active . 2391Note that the sum of the per-queue minima must not exceed the aggregate maximum. 2392If the sum of the per-queue maxima exceeds the aggregate maximum, 2393then the number of active operations may reach 2394.Sy zfs_vdev_max_active , 2395in which case no further operations will be issued, 2396regardless of whether all per-queue minima have been met. 2397.Pp 2398For many physical devices, throughput increases with the number of 2399concurrent operations, but latency typically suffers. 2400Furthermore, physical devices typically have a limit 2401at which more concurrent operations have no 2402effect on throughput or can actually cause it to decrease. 2403.Pp 2404The scheduler selects the next operation to issue by first looking for an 2405I/O class whose minimum has not been satisfied. 2406Once all are satisfied and the aggregate maximum has not been hit, 2407the scheduler looks for classes whose maximum has not been satisfied. 2408Iteration through the I/O classes is done in the order specified above. 2409No further operations are issued 2410if the aggregate maximum number of concurrent operations has been hit, 2411or if there are no operations queued for an I/O class that has not hit its 2412maximum. 2413Every time an I/O operation is queued or an operation completes, 2414the scheduler looks for new operations to issue. 2415.Pp 2416In general, smaller 2417.Sy max_active Ns s 2418will lead to lower latency of synchronous operations. 2419Larger 2420.Sy max_active Ns s 2421may lead to higher overall throughput, depending on underlying storage. 2422.Pp 2423The ratio of the queues' 2424.Sy max_active Ns s 2425determines the balance of performance between reads, writes, and scrubs. 2426For example, increasing 2427.Sy zfs_vdev_scrub_max_active 2428will cause the scrub or resilver to complete more quickly, 2429but reads and writes to have higher latency and lower throughput. 2430.Pp 2431All I/O classes have a fixed maximum number of outstanding operations, 2432except for the async write class. 2433Asynchronous writes represent the data that is committed to stable storage 2434during the syncing stage for transaction groups. 2435Transaction groups enter the syncing state periodically, 2436so the number of queued async writes will quickly burst up 2437and then bleed down to zero. 2438Rather than servicing them as quickly as possible, 2439the I/O scheduler changes the maximum number of active async write operations 2440according to the amount of dirty data in the pool. 2441Since both throughput and latency typically increase with the number of 2442concurrent operations issued to physical devices, reducing the 2443burstiness in the number of simultaneous operations also stabilizes the 2444response time of operations from other queues, in particular synchronous ones. 2445In broad strokes, the I/O scheduler will issue more concurrent operations 2446from the async write queue as there is more dirty data in the pool. 2447. 2448.Ss Async Writes 2449The number of concurrent operations issued for the async write I/O class 2450follows a piece-wise linear function defined by a few adjustable points: 2451.Bd -literal 2452 | o---------| <-- \fBzfs_vdev_async_write_max_active\fP 2453 ^ | /^ | 2454 | | / | | 2455active | / | | 2456 I/O | / | | 2457count | / | | 2458 | / | | 2459 |-------o | | <-- \fBzfs_vdev_async_write_min_active\fP 2460 0|_______^______|_________| 2461 0% | | 100% of \fBzfs_dirty_data_max\fP 2462 | | 2463 | `-- \fBzfs_vdev_async_write_active_max_dirty_percent\fP 2464 `--------- \fBzfs_vdev_async_write_active_min_dirty_percent\fP 2465.Ed 2466.Pp 2467Until the amount of dirty data exceeds a minimum percentage of the dirty 2468data allowed in the pool, the I/O scheduler will limit the number of 2469concurrent operations to the minimum. 2470As that threshold is crossed, the number of concurrent operations issued 2471increases linearly to the maximum at the specified maximum percentage 2472of the dirty data allowed in the pool. 2473.Pp 2474Ideally, the amount of dirty data on a busy pool will stay in the sloped 2475part of the function between 2476.Sy zfs_vdev_async_write_active_min_dirty_percent 2477and 2478.Sy zfs_vdev_async_write_active_max_dirty_percent . 2479If it exceeds the maximum percentage, 2480this indicates that the rate of incoming data is 2481greater than the rate that the backend storage can handle. 2482In this case, we must further throttle incoming writes, 2483as described in the next section. 2484. 2485.Sh ZFS TRANSACTION DELAY 2486We delay transactions when we've determined that the backend storage 2487isn't able to accommodate the rate of incoming writes. 2488.Pp 2489If there is already a transaction waiting, we delay relative to when 2490that transaction will finish waiting. 2491This way the calculated delay time 2492is independent of the number of threads concurrently executing transactions. 2493.Pp 2494If we are the only waiter, wait relative to when the transaction started, 2495rather than the current time. 2496This credits the transaction for "time already served", 2497e.g. reading indirect blocks. 2498.Pp 2499The minimum time for a transaction to take is calculated as 2500.D1 min_time = min( Ns Sy zfs_delay_scale No \(mu Po Sy dirty No \- Sy min Pc / Po Sy max No \- Sy dirty Pc , 100ms) 2501.Pp 2502The delay has two degrees of freedom that can be adjusted via tunables. 2503The percentage of dirty data at which we start to delay is defined by 2504.Sy zfs_delay_min_dirty_percent . 2505This should typically be at or above 2506.Sy zfs_vdev_async_write_active_max_dirty_percent , 2507so that we only start to delay after writing at full speed 2508has failed to keep up with the incoming write rate. 2509The scale of the curve is defined by 2510.Sy zfs_delay_scale . 2511Roughly speaking, this variable determines the amount of delay at the midpoint 2512of the curve. 2513.Bd -literal 2514delay 2515 10ms +-------------------------------------------------------------*+ 2516 | *| 2517 9ms + *+ 2518 | *| 2519 8ms + *+ 2520 | * | 2521 7ms + * + 2522 | * | 2523 6ms + * + 2524 | * | 2525 5ms + * + 2526 | * | 2527 4ms + * + 2528 | * | 2529 3ms + * + 2530 | * | 2531 2ms + (midpoint) * + 2532 | | ** | 2533 1ms + v *** + 2534 | \fBzfs_delay_scale\fP ----------> ******** | 2535 0 +-------------------------------------*********----------------+ 2536 0% <- \fBzfs_dirty_data_max\fP -> 100% 2537.Ed 2538.Pp 2539Note, that since the delay is added to the outstanding time remaining on the 2540most recent transaction it's effectively the inverse of IOPS. 2541Here, the midpoint of 2542.Em 500 us 2543translates to 2544.Em 2000 IOPS . 2545The shape of the curve 2546was chosen such that small changes in the amount of accumulated dirty data 2547in the first three quarters of the curve yield relatively small differences 2548in the amount of delay. 2549.Pp 2550The effects can be easier to understand when the amount of delay is 2551represented on a logarithmic scale: 2552.Bd -literal 2553delay 2554100ms +-------------------------------------------------------------++ 2555 + + 2556 | | 2557 + *+ 2558 10ms + *+ 2559 + ** + 2560 | (midpoint) ** | 2561 + | ** + 2562 1ms + v **** + 2563 + \fBzfs_delay_scale\fP ----------> ***** + 2564 | **** | 2565 + **** + 2566100us + ** + 2567 + * + 2568 | * | 2569 + * + 2570 10us + * + 2571 + + 2572 | | 2573 + + 2574 +--------------------------------------------------------------+ 2575 0% <- \fBzfs_dirty_data_max\fP -> 100% 2576.Ed 2577.Pp 2578Note here that only as the amount of dirty data approaches its limit does 2579the delay start to increase rapidly. 2580The goal of a properly tuned system should be to keep the amount of dirty data 2581out of that range by first ensuring that the appropriate limits are set 2582for the I/O scheduler to reach optimal throughput on the back-end storage, 2583and then by changing the value of 2584.Sy zfs_delay_scale 2585to increase the steepness of the curve. 2586